Re: [PATCH v2 21/21] kbuild: use pigz for gzip compression if available

[H. Peter Anvin]

On September 14, 2026 9:39:07 AM PDT, Kees Cook <kees@xxxxxxxxxx> wrote:
>On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote:
>> On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
>> 1.6s, and with pigz it takes 0.09s, so the performance increase is
>> significant.
>
>Neat! I wanna go learn how pigz accomplishes this -- I thought the
>problem with gzip was a common lookup table. Anyway...
>
>> --- a/Documentation/kbuild/reproducible-builds.rst
>> +++ b/Documentation/kbuild/reproducible-builds.rst
>> @@ -76,6 +76,22 @@ include generated files.  You should ensure the source tree is
>>  pristine by running ``make mrproper`` or ``git clean -d -f -x`` before
>>  building a source package.
>>  
>> +Compression tools
>> +-----------------
>> +
>> +The compressed kernel image, compressed modules and packages are produced
>> +using the program named by the make variable ``KGZIP`` (described in
>> +Documentation/kbuild/kbuild.rst).
>> +
>> +It defaults to ``pigz`` if installed (a parallel implementation of gzip),
>> +or ``gzip`` otherwise.
>> +
>> +The generated output between two invocations of identical builds with
>> +either of the default tools will be byte-for-byte equivalent.
>> +
>> +However, for reproducible builds, ensure the same tool is used on all build
>> +hosts, as different tools may generate different output from one another.
>> +
>>  Module signing
>>  --------------
>>  
>
>This doc update seems totally unneeded? Having the same build tools for RB
>is already a known requirement. I don't think anything new is added here?
>
>> diff --git a/Makefile b/Makefile
>> index 790ef23c5e8a..38c0cdc9f591 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -561,7 +561,7 @@ PERL		= perl
>>  PYTHON3		= python3
>>  CHECK		= sparse
>>  BASH		= bash
>> -KGZIP		= gzip
>> +KGZIP		:= $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)
>
>I think the more idiomatic way to do this is:
>
>KGZIP := $(call try-run,command -v pigz,pigz,gzip)
>
>However, parallelism needs to be set. We can't let it eat all CPUs: it
>needs to respect the -j make option (and make its CPU reservation known
>to "make"), which we already have a solution for in
>scripts/jobserver-exec.
>
>However, I would actually argue that given such an improvement we should just
>make pigz explicitly required and not optional. It is packaged everywhere:
>
>  │ Debian / Ubuntu              │ ✅          │ pigz (main)
>  │ Fedora                       │ ✅          │ pigz 2.8 (current)
>  │ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream
>  │ openSUSE / SLE               │ ✅          │ pigz
>  │ Arch Linux                   │ ✅          │ pigz (extra)
>  │ Alpine                       │ ✅          │ pigz
>  │ Gentoo                       │ ✅          │ app-arch/pigz 2.8
>
>Only RHEL appears a little glitchy, but likely they would trivially move
>it to base since it's already packaged, but off in EPEL.
>
>So, I would say that pigz would be best run as something like:
>
>  KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec $(abs_srctree)/scripts/parallel-pigz
>
>with scripts/parallel-pigz being something like:
>
>	#!/bin/sh
>	exec pigz -p ${PARALLELISM:-1} "$@"
>
>> --- a/scripts/Makefile.modinst
>> +++ b/scripts/Makefile.modinst
>> @@ -145,8 +145,10 @@ endif
>>  #
>>  # Compression
>>  #
>> +# Modules are compressed in parallel by make itself, so keep the compressor
>> +# single-threaded when it is pigz.
>>  quiet_cmd_gzip = GZIP    $@
>> -      cmd_gzip = $(KGZIP) -n -f $<
>> +      cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword $(KGZIP)))),-p 1) -n -f $<
>>  quiet_cmd_xz = XZ      $@
>
>Then this could be:
>
>	cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $<
>
>and we patch scripts/jobserver-exec:
>
>diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
>index 21b319e6c9a5..8b953148ee9f 100755
>--- a/scripts/jobserver-exec
>+++ b/scripts/jobserver-exec
>@@ -5,6 +5,7 @@
> Determines how many parallel tasks "make" is expecting, as it is
> not exposed via any special variables, reserves them all, runs a subprocess
> with PARALLELISM environment variable set, and releases the jobs back again.
>+If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1.
> 
> See:
>     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
>diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
>index 0b1ffdf9f7a3..fc38c5020b22 100755
>--- a/tools/lib/python/jobserver.py
>+++ b/tools/lib/python/jobserver.py
>@@ -29,6 +29,11 @@ $claim child to do the actual work.
> The end goal here is to keep the total number of build tasks under the
> limit established by the initial ``make -j$n_proc`` call.
> 
>+Setting the ``SINGLE_THREADED`` environment variable skips the reservation
>+entirely and runs the command with ``PARALLELISM=1``. This is meant for callers
>+that run many short commands in parallel themselves, where each one should use
>+only the job slot it already holds.
>+
> See:
>     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
> """
>@@ -68,6 +73,13 @@ class JobserverExec:
>         self.is_open = True  # We only try once
>         self.claim = None
>         #
>+        # SINGLE_THREADED asks for no reservation at all: the command runs
>+        # with PARALLELISM=1, using only the slot its caller already holds.
>+        #
>+        if os.environ.get('SINGLE_THREADED'):
>+            self.claim = 1
>+            return
>+        #
>         # Check the make flags for "--jobserver=R,W"
>         # Note that GNU Make has used --jobserver-fds and --jobserver-auth
>         # so this handles all of them.
>

If we are going to add dependencies, can we add *any* scripting language that is less stone tablet than POSIX awk?