Re: [PATCH 00/23] kbuild: significantly speed up kernel builds

[Nathan Chancellor]

> A typical kernel build consists of a frustratingly large amount of time
> spent stuck in single-threaded bottlenecks.
> 
> It turns out that there's a lot we can do about this and doing so
> significantly impacts kernel build times.
> 
> This series makes allmodconfig builds up to 36% faster, incremental builds
> up to ~70% faster, and noop builds up to ~90% faster.
> 
> Builds are faster across the board on every device I tested.
> 
> Machines used for perf testing:
> 
> * Threadripper -   x86, AMD Threadripper 9980X, 64 cores, 128 threads
> * EPYC         -   x86, 2 socket EPYC 9754, 256 cores, 512 threads
> * M2           - arm64, 2022 M2 macbook pro, 8 cores, 8 threads
>                         (4 perf, 4 efficiency)
> 
> Cutting to the chase:
> 
> 		== allmodconfig FULL build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc       345.3s   275.2s    -70.1s (-20%)
> Threadripper, clang     345.5s   265.7s    -79.8s (-23%)
> EPYC, gcc               188.6s   121.1s    -67.5s (-36%)
> EPYC, clang             261.7s   184.6s    -77.1s (-29%)
> 
> 		== allmodconfig INCREMENTAL build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc        46.4s    15.3s    -31.1s (-67%)
> Threadripper, clang      44.4s    15.2s    -29.1s (-66%)
> EPYC, gcc                82.0s    24.3s    -57.7s (-70%)
> EPYC, clang              77.6s    24.3s    -53.3s (-69%)
> 
> 		== allmodconfig NO-OP build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc       12.97s    1.18s   -11.79s (-91%)
> Threadripper, clang     13.84s    1.64s   -12.20s (-88%)
> EPYC, gcc               25.92s    1.52s   -24.40s (-94%)
> EPYC, clang             27.58s    2.17s   -25.41s (-92%)
> 
> 		== defconfig FULL build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc        35.3s    28.6s     -6.7s (-19%)
> Threadripper, clang      33.5s    26.1s     -7.4s (-22%)
> EPYC, gcc                31.2s    20.6s    -10.6s (-34%)
> EPYC, clang              38.6s    32.2s     -6.4s (-17%)
> M2, gcc                 564.0s   512.4s    -51.6s (-9%)
> M2, clang               616.1s   569.4s    -46.7s (-8%)
> 
> 		== defconfig INCREMENTAL build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc        11.6s     5.5s     -6.1s (-53%)
> Threadripper, clang      11.6s     5.0s     -6.6s (-57%)
> EPYC, gcc                19.3s     9.2s    -10.1s (-52%)
> EPYC, clang              20.1s     8.6s    -11.5s (-57%)
> M2, gcc                  18.6s     9.9s     -8.7s (-47%)
> M2, clang                18.6s     8.2s    -10.4s (-56%)
> 
> 		== defconfig NO-OP build ==
> 
>                         before    after     delta
>                       ----------------------------------
> Threadripper, gcc        0.96s    0.47s    -0.49s (-51%)
> Threadripper, clang      1.21s    0.55s    -0.66s (-55%)
> EPYC, gcc                1.47s    0.66s    -0.81s (-55%)
> EPYC, clang              1.87s    0.80s    -1.07s (-57%)
> M2, gcc                  5.59s    1.69s    -3.90s (-70%)
> M2, clang                6.62s    1.74s    -4.88s (-74%)
> 
> Further performance numbers are provided for each commit giving a sense of
> what each contributes to the final result.
> 
> 			== Testing ==
> 
> Beyond x86, allmodconfig was built with the series for arm64, arm, riscv,
> powerpc64, s390 and loongarch, and for arm64, arm, s390 and loongarch the
> System.map is identical to what the previous shell mksysmap produces for
> the same vmlinux.
> 
> Also tested were parisc64 and m68k build as far as mainline lets them
> (a driver's static assertion and an undefined-symbol check on parisc, gcc
> 16 internal compiler errors on m68k, none of it from this series).
> 
> Kernels for x86, arm64, arm, riscv, loongarch, powerpc64, s390, m68k and
> parisc64 all boot under qemu, every text symbol of System.map is in
> /proc/kallsyms at the relocated address, and a module loads and unloads.
> The s390, arm and loongarch kernels also pass the kallsyms selftest.
> 
> An x86 kernel with CONFIG_MODVERSIONS, CONFIG_EXTENDED_MODVERSIONS and
> CONFIG_MODULE_SRCVERSION_ALL boots, loads and unloads modules. External
> modules build against both in-tree and O= builds, and every commit builds
> on x86 defconfig.
> 
> Build times are the best of several runs, no unexpected errors or
> warnings were seen.
> 
> While some aspects of the build process have been changed, all tooling
> should function identically to before.

I appreciate all of the testing that you have done! I will test this
side by side on a couple of my own machines to see what results I get
but I do like those results.

> 			== LLM usage ==
> 
> An LLM was used to first determine where the bottlenecks were then to
> figure out how to improve them.
> 
> It generated a lot of code, much of it hideous.
> 
> I extensively audited and rewrote a lot of it, and heavily edited commit
> messages, the cover letter and comments.x
> 
> The LLM has also orchestrated build runs, testing, debugging and analysis.
> 
> I have manually checked for correctness in both build and running kernels
> generated with this series applied.
> 
> Performance improvements were also verified manually.
> 
> Since an LLM was used extensively, each commit carries an Assisted-by tag.

Thank you for calling all of this out as well.

> 			== What was changed? ==
> 
> Fundamentally the series improves build times by parallelising
> single-threaded tasks as much as possible and improving the efficiency of
> code used in the build process.
> 
> kbuild, kallsyms, modpost, objtool, mksysmap and the rust build system were
> all updated as part of this change.
> 
> Nothing too controversial was included. There are further improvements that
> could be made, but they would either by very invasive (large scale C header
> changes) or generate diminishing returns.
> 
> 			== Patches ==
> 
>    mksysmap (1, 2):  A couple of bugfixes the rest of the series relies
>                      upon.

As I note in other threads of this review, I think we should take these
to Linus now, they seem Obviously CorrectTM and it helps chunk out the
series.

I have just reviewed a few of the low hanging fruit patches. I will try
to take a look at the rest over the next couple of weeks but I might not
get to them until after Plumbers. I would greatly appreciate if there
are others who are more knowledgeable in these areas who could review
these changes, as I don't know too much about some of the corners of
Kbuild yet.

-- 
Cheers,
Nathan