Re: pcpu allocator on large NUMA machines

[Tejun Heo]

Hello,

On Mon, Jul 24, 2017 at 03:42:40PM +0200, Michal Hocko wrote:
> we are seeing a strange pcpu allocation failures on a large ppc machine
> on our older distribution kernel. Please note that I am not yet sure
> this is the case with the current up-to-date kernel and I do not have
> direct access to the machine but it seems there were not many changes in
> the pcpu area since 4.4 that would make any difference.
> 
> The machine has 32TB of memory and 192 cores.
> 
> Warnings are as follows:
> WARNING: at ../mm/vmalloc.c:2423
> [...]
> NIP [c00000000028e048] pcpu_get_vm_areas+0x698/0x6d0
> LR [c000000000268be0] pcpu_create_chunk+0xb0/0x160
> Call Trace:
> [c000106c3948f900] [c000000000268684] pcpu_mem_zalloc+0x54/0xd0 (unreliable)
> [c000106c3948f9d0] [c000000000268be0] pcpu_create_chunk+0xb0/0x160
> [c000106c3948fa00] [c000000000269dc4] pcpu_alloc+0x284/0x740
> [c000106c3948faf0] [c00000000017ac90] hotplug_cfd+0x100/0x150
> [c000106c3948fb30] [c0000000000eabf8] notifier_call_chain+0x98/0x110
> [c000106c3948fb80] [c0000000000bdae0] _cpu_up+0x150/0x210
> [c000106c3948fc30] [c0000000000bdcbc] cpu_up+0x11c/0x140
> [c000106c3948fcb0] [c000000000ac47dc] smp_init+0x110/0x118
> [c000106c3948fd00] [c000000000aa4228] kernel_init_freeable+0x19c/0x364
> [c000106c3948fdc0] [c00000000000bf58] kernel_init+0x28/0x150
> [c000106c3948fe30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4

That looks like the spread between NUMA addresses is larger than the
size of vmalloc area.

> And the kernel log complains about the max_distance.
> PERCPU: max_distance=0x1d452f940000 too large for vmalloc space 0x80000000000

Yeah, that triggers when the distance becomes larger than 75%.

> The boot dies eventually...
> 
> Reducing the number of cores doesn't help but reducing the size of
> memory does.  Increasing the vmalloc space (to 56TB) helps as well. Our
> older kernels (based on 4.4) booted just fine and it seems that
> ba4a648f12f4 ("powerpc/numa: Fix percpu allocations to be NUMA aware")
> (which went to stable) changed the picture. Previously the same machine
> consumed ~400MB vmalloc area per NUMA node.
> 0xd00007ffb8000000-0xd00007ffd0000000 402653184 pcpu_get_vm_areas+0x0/0x6d0 vmalloc
> 0xd00007ffd0000000-0xd00007ffe8000000 402653184 pcpu_get_vm_areas+0x0/0x6d0 vmalloc
> 0xd00007ffe8000000-0xd000080000000000 402653184 pcpu_get_vm_areas+0x0/0x6d0 vmalloc
> 
> My understanding of the pcpu allocator is basically close to zero but it
> seems weird to me that we would need many TB of vmalloc address space
> just to allocate vmalloc areas that are in range of hundreds of MB. So I
> am wondering whether this is an expected behavior of the allocator or
> there is a problem somwehere else.

It's not actually using the entire region but the area allocations try
to follow the same topology as kernel linear address layouts.  ie. if
kernel address for different NUMA nodes are apart by certain amount,
the percpu allocator tries to replicate that for dynamic allocations
which allows leaving the static and first dynamic area in the kernel
linear address which helps reducing TLB pressure.

This optimization can be turned off when vmalloc area isn't spacious
enough by using pcpu_page_first_chunk() instead of
pcpu_embed_first_chunk() while initializing percpu allocator.  Can you
see whether replacing that in arch/powerpc/kernel/setup_64.c fixes the
issue?  If so, all it needs to do is figuring out what conditions we
need to check to opt out of embedding the first chunk.  Note that x86
32bit does about the same thing.

Thanks.

-- 
tejun