Re: [PATCH 16/23] kbuild: batch module finalisation

[Lorenzo Stoakes (ARM)]

On Thu, Sep 10, 2026 at 05:48:53PM +0200, Nicolas Schier wrote:
> On Tue, Sep 08, 2026 at 09:55:16PM +0100, Lorenzo Stoakes (ARM) wrote:
> > Module finalisation on allmodconfig builds consists of a large number of
> > very short-lived jobs, and the make job dispatcher cannot possibly dispatch
> > jobs fast enough.
> >
> > For allmodconfig x86-64 this can be on the order of ~22,000 jobs of a few
> > milliseconds in duration each.
> >
> > However per-job cost grows with the variables the instance holds, here the
> > savedcmd_* of every .mod.o and .ko read back from the .cmd files, since it
> > walks them all to build each child's environment.
> >
> > This makes module finalisation very inefficient when large numbers of
> > modules are being built.
> >
> > Fix this by splitting modules.order into chunks of 128 at a time, run in
> > parallel.
> >
> > Each instance holds only its own modules' variables and the top-level one
> > reads no per-module .cmd files at all, the same rules serve both levels,
> > and an instance is told its chunk with modfinal-first=<index>.
> >
> > "make modules" with every *.mod.o and *.ko deleted goes from 28.9s to 15.9s
> > with clang 22. No-op "make modules" goes from 5.6s to 4.8s, as checking the
> > 22,000 targets is spread over the chunks too.
> >
> > Whole build, 128-thread Threadripper 9980X, best of N runs:
> >
> >                                          before   after     delta
> >                                          -------------------------------
> >   x86 allmodconfig, no-op make, gcc         1.9s     1.2s    -0.74s (-39%)
> >   x86 allmodconfig, no-op make, clang       2.4s     1.6s    -0.77s (-32%)
> >   x86 allmodconfig, clean, gcc            304.5s   291.4s     -13.1s (-4%)
> >   x86 allmodconfig, clean, clang          301.7s   297.0s     -4.7s (-2%)
> >
> > Assisted-by: LLM
> > Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
> > ---
> >  scripts/Makefile.modfinal | 26 +++++++++++++++++++++++++-
> >  scripts/mod/sumversion.c  |  4 ++--
> >  2 files changed, 27 insertions(+), 3 deletions(-)
> >
> > diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> > index 75e9effdf02c..858fa798090b 100644
> > --- a/scripts/Makefile.modfinal
> > +++ b/scripts/Makefile.modfinal
>
> [...]
> > diff --git a/scripts/mod/sumversion.c b/scripts/mod/sumversion.c
> > index 2cbadd3cd97d..5501d6aa0bea 100644
> > --- a/scripts/mod/sumversion.c
> > +++ b/scripts/mod/sumversion.c
> > @@ -249,7 +249,7 @@ static int parse_comment(const char *file, unsigned long len)
> >  /* FIXME: Handle .s files differently (eg. # starts comments) --RR */
> >  static bool stop_char[256];
> >
> > -static void init_stop_chars(void)
> > +static void sumversion_init(void)
> >  {
> >  	static bool done;
> >  	int chr;
> > @@ -274,7 +274,7 @@ static int parse_file(const char *fname, struct md4_ctx *md)
> >  	len = strlen(file);
> >  	if (!len)
> >  		goto out_file;
> > -	init_stop_chars();
> > +	sumversion_init();
> >  	buf = xmalloc(len); /* File output buffer. */
> >
> >  	for (i = 0; i < len; i++) {
> >
> > --
> > 2.55.0
> >
>
> (I have only skimmed through, not yet reviewed this patch.)
>
> The change in sumversion.c seems to be wrong here.

Ugh yeah this is a merge conflict artifact that I thought I had fixed but
clearly not :) Will fix up on respin.

>
> --
> Nicolas



--
Cheers, Lorenzo