Re: [PATCH 23/23] kbuild: use pigz for gzip compression if available

[Nathan Chancellor]

> The gzip step of a kernel build is very lengthily, especially for larger
> builds such as allmodconfig.
> 
> gzip itself cannot be run in parallel, however an alternative tool exists
> that can, providing the same feature set as gzip itself - pigz - which
> works as a drop-in replacement.
> 
> On a 128-core Threadripper, gzip -9 of a 36 MiB x86-64 vmlinux.bin takes
> 1.6s, and with pigz it takes 0.09s, so the performance increase is
> significant.
> 
> It is already possible to specify the KGZIP environmental variable to make
> use of pigz, however it seems sensible to make use of pigz if it is
> available.
> 
> Therefore default to using pigz if it is available on the system upon which
> the kernel is being built, otherwise fall back to gzip.
> 
> The output is byte-for-byte identical between pigz/gzip invocations, but
> gzip and pigz do not produce the same stream as one another.
> 
> Therefore, for reproducible builds, the same set of tools should be used.
> 
> This seems to already be an implicit requirement in any case, but update
> the reproducible build documentation to make this clear.
> 
> Also update the kbuild documentation to reflect the change.
> 
> Every x86 build ends with the compression of vmlinux.bin, 36MB for
> defconfig and over 200MB for allmodconfig, no-op builds are unchanged.
> 
> Whole build, 128-thread Threadripper 9980X, best of N runs:
> 
>                                          before   after     delta
>                                          -------------------------------
>   x86 defconfig, touch mm/vma.c, gcc        7.4s     5.4s     -2.0s (-27%)
>   x86 defconfig, touch mm/vma.c, clang      6.6s     4.9s     -1.7s (-26%)
>   x86 defconfig, clean, gcc                26.2s    24.4s     -1.8s (-7%)
>   x86 defconfig, clean, clang              25.8s    24.3s     -1.5s (-6%)
>   x86 allmodconfig, touch mm/vma.c, gcc    23.7s    15.3s     -8.4s (-35%)
>   x86 allmodconfig, touch mm/vma.c, clang  22.1s    15.2s     -6.9s (-31%)
>   x86 allmodconfig, clean, gcc            291.4s   275.2s     -16.2s (-6%)
>   x86 allmodconfig, clean, clang          278.3s   265.7s     -12.6s (-5%)
> 
> Link: https://zlib.net/pigz/
> Assisted-by: LLM
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@xxxxxxxxxx>
...
> diff --git a/Makefile b/Makefile
> index a0d29dee81d7..efa6a2d77e6b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -561,7 +561,7 @@ PERL		= perl
>  PYTHON3		= python3
>  CHECK		= sparse
>  BASH		= bash
> -KGZIP		= gzip
> +KGZIP		:= $(if $(shell command -v pigz 2>/dev/null),pigz,gzip)

This should be '?=', right? Sashiko also notes a concern around parallel
module installation with gzip module compression since there is no
thread limitation, which could bring a little risk to doing this by
default? Might not be that big of a deal though?

-- 
Cheers,
Nathan