Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

[Pavel Tatashin]

Thank you Masayoshi for verifying this work. I will submit it as you suggested.

Pavel

On Mon, Feb 5, 2018 at 9:14 AM, Masayoshi Mizuma <msys.mizuma@xxxxxxxxx> wrote:
> Hello Pavel,
>
>> Yes, the patch is here:
>> https://lkml.org/lkml/2018/1/12/600
>
> I tested your patch in my box and it worked well.
> Please feel free to add the following.
>
> Tested-by: Masayoshi Mizuma <m.mizuma@xxxxxxxxxxxxxx>
>
> You may repost the patch after adding your reply for
> Andrew's comment as [PATCH 0/1]...
>
> - Masayoshi
>
> Wed, 31 Jan 2018 13:24:55 -0500 Pavel Tatashin wrote:
>> Hi Koki,
>>
>> Yes, the patch is here:
>> https://lkml.org/lkml/2018/1/12/600
>>
>> It has not been reviewed yet.
>>
>> Pavel
>>
>> On Wed, Jan 31, 2018 at 12:28 PM, Koki.Sanagi@xxxxxxxxxxxxxx
>> <Koki.Sanagi@xxxxxxxxxxxxxx> wrote:
>>> Pavel,
>>>
>>> I assume you are working on the fix.
>>> Do you have any progress ?
>>>
>>> Koki
>>>
>>>>> -----Original Message-----
>>>>> From: Mel Gorman [mailto:mgorman@xxxxxxxxxxxxxxxxxxx]
>>>>> Sent: Wednesday, December 06, 2017 5:50 AM
>>>>> To: Pavel Tatashin <pasha.tatashin@xxxxxxxxxx>
>>>>> Cc: Michal Hocko <mhocko@xxxxxxxxxx>; YASUAKI ISHIMATSU
>>>>> <yasu.isimatu@xxxxxxxxx>; Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>;
>>>>> Linux Memory Management List <linux-mm@xxxxxxxxx>; linux-
>>>>> kernel@xxxxxxxxxxxxxxx; Sanagi, Koki <Koki.Sanagi@xxxxxxxxxxxxxx>; Steve
>>>>> Sistare <steven.sistare@xxxxxxxxxx>
>>>>> Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if
>>>>> trace_buf_size is specified
>>>>>
>>>>> On Wed, Nov 29, 2017 at 10:41:59PM -0500, Pavel Tatashin wrote:
>>>>>> Hi Mel,
>>>>>>
>>>>>> Thank you very much for your feedback, my replies below:
>>>>>>
>>>>>>> A lack of involvement from admins is indeed desirable. For example,
>>>>>>> while I might concede on using a disable-everything-switch, I would
>>>>>>> not be happy to introduce a switch that specified how much memory
>>>>>>> per node to initialise.
>>>>>>>
>>>>>>> For the forth approach, I really would be only thinking of a blunt
>>>>>>> "initialise everything instead of going OOM". I was wary of making
>>>>>>> things too complicated and I worried about some side-effects I'll cover later.
>>>>>>
>>>>>> I see, I misunderstood your suggestion. Switching to serial
>>>>>> initialization when OOM works, however, boot time becomes
>>>>>> unpredictable, with some configurations boot is fast with others it is
>>>>>> slow. All of that depends on whether predictions in
>>>>>> reset_deferred_meminit() were good or not which is not easy to debug
>>>>>> for users. Also, overtime predictions in reset_deferred_meminit() can
>>>>>> become very off, and I do not think that we want to continuously
>>>>>> adjust this function.
>>>>>>
>>>>>
>>>>> You could increase the probabilty of a report by doing a WARN_ON_ONCE if the
>>>>> serialised meminit is used.
>>>>>
>>>>>>>> With this approach we could always init a very small amount of
>>>>>>>> struct pages, and allow the rest to be initialized on demand as
>>>>>>>> boot requires until deferred struct pages are initialized. Since,
>>>>>>>> having deferred pages feature assumes that the machine is large,
>>>>>>>> there is no drawback of having some extra byte of dead code,
>>>>>>>> especially that all the checks can be permanently switched of via
>>>>>>>> static branches once deferred init is complete.
>>>>>>>>
>>>>>>>
>>>>>>> This is where I fear there may be dragons. If we minimse the number
>>>>>>> of struct pages and initialise serially as necessary, there is a
>>>>>>> danger that we'll allocate remote memory in cases where local memory
>>>>>>> would have done because a remote node had enough memory.
>>>>>>
>>>>>> True, but is not what we have now has the same issue as well? If one
>>>>>> node is gets out of memory we start using memory from another node,
>>>>>> before deferred pages are initialized?
>>>>>>
>>>>>
>>>>> It's possible but I'm not aware of it happening currently.
>>>>>
>>>>>>  To offset that risk, it would be
>>>>>>> necessary at boot-time to force allocations from local node where
>>>>>>> possible and initialise more memory as necessary. That starts
>>>>>>> getting complicated because we'd need to adjust gfp-flags in the
>>>>>>> fast path with init-and-retry logic in the slow path and that could
>>>>>>> be a constant penalty. We could offset that in the fast path by
>>>>>>> using static branches
>>>>>>
>>>>>> I will try to implement this, and see how complicated the patch will
>>>>>> be, if it gets too complicated for the problem I am trying to solve we
>>>>>> can return to one of your suggestions.
>>>>>>
>>>>>> I was thinking to do something like this:
>>>>>>
>>>>>> Start with every small amount of initialized pages in every node.
>>>>>> If allocation fails, initialize enough struct pages to cover this
>>>>>> particular allocation with struct pages rounded up to section size but
>>>>>> in every single node.
>>>>>>
>>>>>
>>>>> Ok, just make sure it's all in the slow paths of the allocator when the alternative
>>>>> is to fail the allocation.
>>>>>
>>>>>>> but it's getting more and
>>>>>>> more complex for what is a minor optimisation -- shorter boot times
>>>>>>> on large machines where userspace itself could take a *long* time to
>>>>>>> get up and running (think database reading in 1TB of data from disk as it
>>>>> warms up).
>>>>>>
>>>>>> On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time,
>>>>>> and this is on SPARC with 8K pages, on x86 it would be around of 8
>>>>>> minutes because of twice as many pages. This feature improves
>>>>>> availability for larger machines quite a bit. Overtime, systems are
>>>>>> growing, so I expect this feature to become a default configuration in
>>>>>> the next several years on server configs.
>>>>>>
>>>>>
>>>>> Ok, when developing the series originally, I had no machine even close to 32T of
>>>>> memory.
>>>>>
>>>>> --
>>>>> Mel Gorman
>>>>> SUSE Labs
>>>
>>> --
>>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>> the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
>>> see: http://www.linux-mm.org/ .
>>> Don't email: <a hrefmailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>