Re: [PATCHv2 0/3] Find mirrored memory, use for boot time allocations

[Tony Luck]

On Mon, May 18, 2015 at 8:01 PM, Xishi Qiu <qiuxishi@xxxxxxxxxx> wrote:
> In part2, does it means the memory allocated from kernel should use mirrored memory?

Yes. I want to use mirrored memory for all (or as many as
possible) kernel allocations.

> I have heard of this feature(address range mirroring) before, and I changed some
> code to test it(implement memory allocations in specific physical areas).
>
> In my opinion, add a new zone(ZONE_MIRROR) to fill the mirrored memory is not a good
> idea. If there are XX discontiguous mirrored areas in one numa node, there should be
> XX ZONE_MIRROR zones in one pgdat, it is impossible, right?

With current h/w implementations XX is at most 2, and is possibly only 1
on most nodes.  But we shouldn't depend on that.

> I think add a new migrate type(MIGRATE_MIRROR) will be better, the following print
> is from my changed kernel.

This sounds interesting.

> [root@localhost ~]# cat /proc/pagetypeinfo
> Page block order: 9
> Pages per block:  512
>
> Free pages count per migrate type at order       0      1      2      3      4      5      6      7      8      9     10
...
> Node    0, zone      DMA, type       Mirror      0      0      0      0      0      0      0      0      0      0      0
...
> Node    0, zone    DMA32, type       Mirror      0      0      0      0      0      0      0      0      0      0      0

I see all zero counts here ... which is fine.  I expect that systems
will mirror all memory below 4GB ... but we should probably
ignore the attribute for this range because we want to make
sure that the memory is still available for users that depend
on getting memory that legacy devices can access. On systems
that support address range mirror the <4GB area is <2% of even
a small system (128GB seems to be the minimum rational configuration
for a 4 socket machine ... you end up with that much if you populate
every channel with just one 4GB DIMM). On a big system (in the TB
range) <4GB area is a trivial rounding error.

> Also I add a new flag(GFP_MIRROR), then we can use the mirrored form both
> kernel-space and user-space. If there is no mirrored memory, we will allocate
> other types memory.

But I *think* I want all kernel and no users to allocate mirror
memory.  I'd like to not have to touch every place that allocates
memory to add/clear this flag.

> 1) kernel-space(pcp, page buddy, slab/slub ...):
>         -> use mirrored memory(e.g. /proc/sys/vm/mirrorable)
>                 -> __alloc_pages_nodemask()
>                         ->gfpflags_to_migratetype()
>                                 -> use MIGRATE_MIRROR list

I think you are telling me that we can do this, but I don't understand
how the code would look.

> 2) user-space(syscall, madvise, mmap ...):
>         -> add VM_MIRROR flag in the vma
>                 -> add GFP_MIRROR when page fault in the vma
>                         -> __alloc_pages_nodemask()
>                                 -> use MIGRATE_MIRROR list

If we do let users have access to mirrored memory, then
madvise/mmap seem a plausible way to allow it.  Not sure
what access privileges are appropriate to allow it. I expect
mirrored memory to be in short supply (the whole point of
address range mirror is to make do with a minimal amount
of mirrored memory ... if you expect to want/have lots of
mirrored memory, then just take the 50% hit in capacity
and mirror everything and ignore all the s/w complexity).

Are your patches ready to be shared?

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/