Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

[Mel Gorman]

On Wed, Nov 15, 2017 at 11:49:19AM -0800, Andrew Morton wrote:
> On Wed, 15 Nov 2017 08:55:56 +0000 Mel Gorman <mgorman@xxxxxxxxxxxxxxxxxxx> wrote:
> 
> > Yasuaki Ishimatsu reported a premature OOM when trace_buf_size=100m was
> > specified on a machine with many CPUs. The kernel tried to allocate 38.4GB
> > but only 16GB was available due to deferred memory initialisation.
> > 
> > The allocation context is within smp_init() so there are no opportunities
> > to do the deferred meminit earlier. Furthermore, the partial initialisation
> > of memory occurs before the size of the trace buffers is set so there is
> > no opportunity to adjust the amount of memory that is pre-initialised. We
> > could potentially catch when memory is low during system boot and adjust the
> > amount that is initialised serially but it's a little clumsy as it would
> > require a check in the failure path of the page allocator.  Given that
> > deferred meminit is basically a minor optimisation that only benefits very
> > large machines and trace_buf_size is somewhat specialised, it follows that
> > the most straight-forward option is to go back to serialised meminit if
> > trace_buf_size is specified.
> 
> Patch is rather messy.
> 
> I went cross-eyed trying to work out how tracing allocates that buffer,
> but I assume it ends up somewhere in the page allocator. 

Basic path is

[ ]  __alloc_pages_slowpath+0x9a6/0xba7
[ ]  __alloc_pages_nodemask+0x26a/0x290
[ ]  new_slab+0x297/0x500
[ ]  ___slab_alloc+0x335/0x4a0
[ ]  __slab_alloc+0x40/0x66
[ ]  __kmalloc_node+0xbd/0x270
[ ]  __rb_allocate_pages+0xae/0x180
[ ]  rb_allocate_cpu_buffer+0x204/0x2f0
[ ]  trace_rb_cpu_prepare+0x7e/0xc5
[ ]  cpuhp_invoke_callback+0x3ea/0x5c0
[ ]  _cpu_up+0xbc/0x190
[ ]  do_cpu_up+0x87/0xb0
[ ]  cpu_up+0x13/0x20
[ ]  smp_init+0x69/0xca
[ ]  kernel_init_freeable+0x115/0x244

Note that it's during smp_init and part of the CPU onlining which is before
deferred meminit can start.

> If the page
> allocator is about to fail an allocation request and sees that memory
> initialization is still ongoing, surely the page allocator should just
> wait?  That seems to be the most general fix?
> 

In other contexts yes, but as deferred meminit has not started, there is
nothing to wait for yet.

-- 
Mel Gorman
SUSE Labs