RE: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

[Koki.Sanagi@xxxxxxxxxxxxxx]

Pavel,

I assume you are working on the fix.
Do you have any progress ?

Koki

>>-----Original Message-----
>>From: Mel Gorman [mailto:mgorman@xxxxxxxxxxxxxxxxxxx]
>>Sent: Wednesday, December 06, 2017 5:50 AM
>>To: Pavel Tatashin <pasha.tatashin@xxxxxxxxxx>
>>Cc: Michal Hocko <mhocko@xxxxxxxxxx>; YASUAKI ISHIMATSU
>><yasu.isimatu@xxxxxxxxx>; Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>;
>>Linux Memory Management List <linux-mm@xxxxxxxxx>; linux-
>>kernel@xxxxxxxxxxxxxxx; Sanagi, Koki <Koki.Sanagi@xxxxxxxxxxxxxx>; Steve
>>Sistare <steven.sistare@xxxxxxxxxx>
>>Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if
>>trace_buf_size is specified
>>
>>On Wed, Nov 29, 2017 at 10:41:59PM -0500, Pavel Tatashin wrote:
>>> Hi Mel,
>>>
>>> Thank you very much for your feedback, my replies below:
>>>
>>> > A lack of involvement from admins is indeed desirable. For example,
>>> > while I might concede on using a disable-everything-switch, I would
>>> > not be happy to introduce a switch that specified how much memory
>>> > per node to initialise.
>>> >
>>> > For the forth approach, I really would be only thinking of a blunt
>>> > "initialise everything instead of going OOM". I was wary of making
>>> > things too complicated and I worried about some side-effects I'll cover later.
>>>
>>> I see, I misunderstood your suggestion. Switching to serial
>>> initialization when OOM works, however, boot time becomes
>>> unpredictable, with some configurations boot is fast with others it is
>>> slow. All of that depends on whether predictions in
>>> reset_deferred_meminit() were good or not which is not easy to debug
>>> for users. Also, overtime predictions in reset_deferred_meminit() can
>>> become very off, and I do not think that we want to continuously
>>> adjust this function.
>>>
>>
>>You could increase the probabilty of a report by doing a WARN_ON_ONCE if the
>>serialised meminit is used.
>>
>>> >> With this approach we could always init a very small amount of
>>> >> struct pages, and allow the rest to be initialized on demand as
>>> >> boot requires until deferred struct pages are initialized. Since,
>>> >> having deferred pages feature assumes that the machine is large,
>>> >> there is no drawback of having some extra byte of dead code,
>>> >> especially that all the checks can be permanently switched of via
>>> >> static branches once deferred init is complete.
>>> >>
>>> >
>>> > This is where I fear there may be dragons. If we minimse the number
>>> > of struct pages and initialise serially as necessary, there is a
>>> > danger that we'll allocate remote memory in cases where local memory
>>> > would have done because a remote node had enough memory.
>>>
>>> True, but is not what we have now has the same issue as well? If one
>>> node is gets out of memory we start using memory from another node,
>>> before deferred pages are initialized?
>>>
>>
>>It's possible but I'm not aware of it happening currently.
>>
>>>  To offset that risk, it would be
>>> > necessary at boot-time to force allocations from local node where
>>> > possible and initialise more memory as necessary. That starts
>>> > getting complicated because we'd need to adjust gfp-flags in the
>>> > fast path with init-and-retry logic in the slow path and that could
>>> > be a constant penalty. We could offset that in the fast path by
>>> > using static branches
>>>
>>> I will try to implement this, and see how complicated the patch will
>>> be, if it gets too complicated for the problem I am trying to solve we
>>> can return to one of your suggestions.
>>>
>>> I was thinking to do something like this:
>>>
>>> Start with every small amount of initialized pages in every node.
>>> If allocation fails, initialize enough struct pages to cover this
>>> particular allocation with struct pages rounded up to section size but
>>> in every single node.
>>>
>>
>>Ok, just make sure it's all in the slow paths of the allocator when the alternative
>>is to fail the allocation.
>>
>>> > but it's getting more and
>>> > more complex for what is a minor optimisation -- shorter boot times
>>> > on large machines where userspace itself could take a *long* time to
>>> > get up and running (think database reading in 1TB of data from disk as it
>>warms up).
>>>
>>> On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time,
>>> and this is on SPARC with 8K pages, on x86 it would be around of 8
>>> minutes because of twice as many pages. This feature improves
>>> availability for larger machines quite a bit. Overtime, systems are
>>> growing, so I expect this feature to become a default configuration in
>>> the next several years on server configs.
>>>
>>
>>Ok, when developing the series originally, I had no machine even close to 32T of
>>memory.
>>
>>--
>>Mel Gorman
>>SUSE Labs