Re: [PATCH v2 21/21] kbuild: use pigz for gzip compression if available

[Kees Cook]

On Mon, Sep 14, 2026 at 10:22:20AM +0100, Lorenzo Stoakes (ARM) wrote:
> On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
> 1.6s, and with pigz it takes 0.09s, so the performance increase is
> significant.

Neat! I wanna go learn how pigz accomplishes this -- I thought the
problem with gzip was a common lookup table. Anyway...

> --- a/Documentation/kbuild/reproducible-builds.rst
> +++ b/Documentation/kbuild/reproducible-builds.rst
> @@ -76,6 +76,22 @@ include generated files.  You should ensure the source tree is
>  pristine by running ``make mrproper`` or ``git clean -d -f -x`` before
>  building a source package.
>  
> +Compression tools
> +-----------------
> +
> +The compressed kernel image, compressed modules and packages are produced
> +using the program named by the make variable ``KGZIP`` (described in
> +Documentation/kbuild/kbuild.rst).
> +
> +It defaults to ``pigz`` if installed (a parallel implementation of gzip),
> +or ``gzip`` otherwise.
> +
> +The generated output between two invocations of identical builds with
> +either of the default tools will be byte-for-byte equivalent.
> +
> +However, for reproducible builds, ensure the same tool is used on all build
> +hosts, as different tools may generate different output from one another.
> +
>  Module signing
>  --------------
>  

This doc update seems totally unneeded? Having the same build tools for RB
is already a known requirement. I don't think anything new is added here?

> diff --git a/Makefile b/Makefile
> index 790ef23c5e8a..38c0cdc9f591 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -561,7 +561,7 @@ PERL		= perl
>  PYTHON3		= python3
>  CHECK		= sparse
>  BASH		= bash
> -KGZIP		= gzip
> +KGZIP		:= $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)

I think the more idiomatic way to do this is:

KGZIP := $(call try-run,command -v pigz,pigz,gzip)

However, parallelism needs to be set. We can't let it eat all CPUs: it
needs to respect the -j make option (and make its CPU reservation known
to "make"), which we already have a solution for in
scripts/jobserver-exec.

However, I would actually argue that given such an improvement we should just
make pigz explicitly required and not optional. It is packaged everywhere:

  │ Debian / Ubuntu              │ ✅          │ pigz (main)
  │ Fedora                       │ ✅          │ pigz 2.8 (current)
  │ RHEL / CentOS / Rocky / Alma │ ✅ via EPEL │ pigz — not in base/AppStream
  │ openSUSE / SLE               │ ✅          │ pigz
  │ Arch Linux                   │ ✅          │ pigz (extra)
  │ Alpine                       │ ✅          │ pigz
  │ Gentoo                       │ ✅          │ app-arch/pigz 2.8

Only RHEL appears a little glitchy, but likely they would trivially move
it to base since it's already packaged, but off in EPEL.

So, I would say that pigz would be best run as something like:

  KGZIP := $(PYTHON3) $(abs_srctree)/scripts/jobserver-exec $(abs_srctree)/scripts/parallel-pigz

with scripts/parallel-pigz being something like:

	#!/bin/sh
	exec pigz -p ${PARALLELISM:-1} "$@"

> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -145,8 +145,10 @@ endif
>  #
>  # Compression
>  #
> +# Modules are compressed in parallel by make itself, so keep the compressor
> +# single-threaded when it is pigz.
>  quiet_cmd_gzip = GZIP    $@
> -      cmd_gzip = $(KGZIP) -n -f $<
> +      cmd_gzip = $(KGZIP) $(if $(filter pigz,$(notdir $(firstword $(KGZIP)))),-p 1) -n -f $<
>  quiet_cmd_xz = XZ      $@

Then this could be:

	cmd_gzip = SINGLE_THREADED=1 $(KGZIP) -n -f $<

and we patch scripts/jobserver-exec:

diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec
index 21b319e6c9a5..8b953148ee9f 100755
--- a/scripts/jobserver-exec
+++ b/scripts/jobserver-exec
@@ -5,6 +5,7 @@
 Determines how many parallel tasks "make" is expecting, as it is
 not exposed via any special variables, reserves them all, runs a subprocess
 with PARALLELISM environment variable set, and releases the jobs back again.
+If SINGLE_THREADED is set, nothing is reserved and PARALLELISM is 1.
 
 See:
     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
diff --git a/tools/lib/python/jobserver.py b/tools/lib/python/jobserver.py
index 0b1ffdf9f7a3..fc38c5020b22 100755
--- a/tools/lib/python/jobserver.py
+++ b/tools/lib/python/jobserver.py
@@ -29,6 +29,11 @@ $claim child to do the actual work.
 The end goal here is to keep the total number of build tasks under the
 limit established by the initial ``make -j$n_proc`` call.
 
+Setting the ``SINGLE_THREADED`` environment variable skips the reservation
+entirely and runs the command with ``PARALLELISM=1``. This is meant for callers
+that run many short commands in parallel themselves, where each one should use
+only the job slot it already holds.
+
 See:
     https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver
 """
@@ -68,6 +73,13 @@ class JobserverExec:
         self.is_open = True  # We only try once
         self.claim = None
         #
+        # SINGLE_THREADED asks for no reservation at all: the command runs
+        # with PARALLELISM=1, using only the slot its caller already holds.
+        #
+        if os.environ.get('SINGLE_THREADED'):
+            self.claim = 1
+            return
+        #
         # Check the make flags for "--jobserver=R,W"
         # Note that GNU Make has used --jobserver-fds and --jobserver-auth
         # so this handles all of them.

-- 
Kees Cook