Re: Performance regressions in "boot_time" tests in Linux 5.8 Kernel

[Rahul Gopakumar]

Hi Baoquan,

Attached collected dmesg logs for with and without
commit after adding memblock=debug to kernel cmdline.

________________________________________
From: bhe@xxxxxxxxxx <bhe@xxxxxxxxxx>
Sent: 10 October 2020 11:41 AM
To: Rahul Gopakumar
Cc: linux-mm@xxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; akpm@xxxxxxxxxxxxxxxxxxxx; natechancellor@xxxxxxxxx; ndesaulniers@xxxxxxxxxx; clang-built-linux@xxxxxxxxxxxxxxxx; rostedt@xxxxxxxxxxx; Rajender M; Yiu Cho Lau; Peter Jonasson; Venkatesh Rajaram
Subject: Re: Performance regressions in "boot_time" tests in Linux 5.8 Kernel

On 10/09/20 at 01:15pm, Rahul Gopakumar wrote:
> As part of VMware's performance regression testing for Linux Kernel
> upstream releases, we identified boot time increase when comparing
> Linux 5.8 kernel against Linux 5.7 kernel. Increase in boot time is
> noticeable on VM with a **large amount of memory**.
>
> In our test cases, it's noticeable with memory 1TB and more, whereas
> there was no major difference noticed in testcases with <1TB.
>
> On bisecting between 5.7 and 5.8, we found the following commit from
> “Baoquan He” to be the cause of boot time increase in big VM test cases.
>
> -------------------------------------
>
> commit 73a6e474cb376921a311786652782155eac2fdf0
> Author: Baoquan He <bhe@xxxxxxxxxx>
> Date: Wed Jun 3 15:57:55 2020 -0700
>
> mm: memmap_init: iterate over memblock regions rather that check each PFN
>
> When called during boot the memmap_init_zone() function checks if each PFN
> is valid and actually belongs to the node being initialized using
> early_pfn_valid() and early_pfn_in_nid().
>
> Each such check may cost up to O(log(n)) where n is the number of memory
> banks, so for large amount of memory overall time spent in early_pfn*()
> becomes substantial.
>
> -------------------------------------
>
> For boot time test, we used RHEL 8.1 as the guest OS.
> VM config is 84 vcpu and 1TB vRAM.
>
> Here are the actual performance numbers.
>
> 5.7 GA - 18.17 secs
> Baoquan's commit - 21.6 secs (-16% increase in time)
>
> From dmesg logs, we can see significant time delay around memmap.
>
> Refer below logs.
>
> Good commit
>
> [0.033176] Normal zone: 1445888 pages used for memmap
> [0.033176] Normal zone: 89391104 pages, LIFO batch:63
> [0.035851] ACPI: PM-Timer IO Port: 0x448
>
> Problem commit
>
> [0.026874] Normal zone: 1445888 pages used for memmap
> [0.026875] Normal zone: 89391104 pages, LIFO batch:63
> [2.028450] ACPI: PM-Timer IO Port: 0x448

Could you add memblock=debug to kernel cmdline and paste the boot logs of
system w and w/o the commit?

>
> We did some analysis, and it looks like with the problem commit it's
> not deferring the memory initialization to a later stage and it's
> initializing the huge chunk of memory in serial - during the boot-up
> time.  Whereas with the good commit, it was able to defer the
> initialization of the memory when it could be done in parallel.
>
>
> Rahul Gopakumar
> Performance Engineering
> VMware, Inc.
>

Attachment: with_commit_dmesg.log
Description: with_commit_dmesg.log

Attachment: without_commit_dmesg.log
Description: without_commit_dmesg.log