Re: [RFC 0/4] Transparent on-demand struct page initializationembedded in the buddy allocator

[Robin Holt]

On Fri, Jul 19, 2013 at 04:51:49PM -0700, Yinghai Lu wrote:
> On Wed, Jul 17, 2013 at 2:30 AM, Robin Holt <holt@xxxxxxx> wrote:
> > On Wed, Jul 17, 2013 at 01:17:44PM +0800, Sam Ben wrote:
> >> >With this patch, we did boot a 16TiB machine.  Without the patches,
> >> >the v3.10 kernel with the same configuration took 407 seconds for
> >> >free_all_bootmem.  With the patches and operating on 2MiB pages instead
> >> >of 1GiB, it took 26 seconds so performance was improved.  I have no feel
> >> >for how the 1GiB chunk size will perform.
> >>
> >> How to test how much time spend on free_all_bootmem?
> >
> > We had put a pr_emerg at the beginning and end of free_all_bootmem and
> > then used a modified version of script which record the time in uSecs
> > at the beginning of each line of output.
> 
> used two patches, found 3TiB system will take 100s before slub is ready.
> 
> about three portions:
> 1. sparse vmemap buf allocation, it is with bootmem wrapper, so clear those
> struct page area take about 30s.
> 2. memmap_init_zone: take about 25s
> 3. mem_init/free_all_bootmem about 30s.
> 
> so still wonder why 16TiB will need hours.

I don't know where you got the figure of hours for memory initialization.
That is likely for a 32TiB boot and includes the entire boot, not just
getting the memory allocator initialized.

For a 16 TiB boot:
1) 344
2) 1151
3) 407

I hope that illustrates why we chose to address the memmap_init_zone first
which had the nice side effect of also impacting the free_all_bootmem
slowdown.

With these patches, those numbers are currently:
1) 344
2) 49
3) 26

> also your patches looks like only address 2 and 3.

Right, but I thought that was the normal way to do things.  Address
one thing at a time and work toward a better kernel.  I don't see a
relationship between the work we are doing here and the sparse vmemmap
buffer allocation.  Have I missed something?

Did you happen to time a boot with these patches applied to see how
long it took and how much impact they had on a smaller config?

Robin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/