Re: [PATCH 3/5] kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}

[Eugeniu Rosca]

Hello Yamada-san,

Appreciate your generous support, which allowed finding the root-cause.

On Wed, Aug 02, 2023 at 06:21:14PM +0900, Masahiro Yamada wrote:

[..]

> Indeed, reverting 7ce7e984ab2b218d6e92d5165629022fe2daf9ee
> makes qcom's external module build successfully
> (but rebuilding is super slow).

Same observation.

> 
> Interestingly, revert 7ce7e984ab2b218d6e92d5165629022fe2daf9ee
> then apply the attached patch, then
> 'Argument list too long' will come back.
> 
> So, this is unrelated to the actual build commands.
> 
> I suspect bare 'export', which expands all variables
> while apparently most of them are not meant exported.

Indeed, that seems to be the case and more evidence supports this:

 * Several pre-existing commits point out to the same root-cause:

    https://git.codelinaro.org/clo/la/platform/vendor/qcom/opensource/audio-kernel-ar/-/commit/4025a25a2479fc34
    ("makefile: kona: remove make export <all variables> instances")

    https://github.com/sonyxperiadev/kernel-techpack-audio/commit/02f00754120df2
    ("audio-kernel: Fix build time issue")

 * The more recent GNU Make 4.4.1 (called with -ddd) reveals more
    details just before hitting 'Argument list too long':

    scripts/Makefile.lib:431: not recursively expanding size_append to export to shell function
    scripts/Makefile.lib:447: not recursively expanding cmd_file_size to export to shell function
    scripts/Makefile.lib:453: not recursively expanding cmd_bzip2_with_size to export to shell function
    scripts/Makefile.lib:462: not recursively expanding cmd_lzma_with_size to export to shell function
    scripts/Makefile.lib:468: not recursively expanding cmd_lzo_with_size to export to shell function
    scripts/Makefile.lib:474: not recursively expanding cmd_lz4_with_size to export to shell function
    scripts/Makefile.lib:520: not recursively expanding cmd_xzkern_with_size to export to shell function
    scripts/Makefile.lib:549: not recursively expanding cmd_zstd22_with_size to export to shell function

    The "not recursively expanding" messages above come from GNU
    make commit https://gnu.googlesource.com/make/+/7d484017077089a
    ("[SV 63016] Don't fail exporting to $(shell ...)"), which appears
    to be saying that our issue is caused by exporting a makefile
    variable which contains the 'shell' directive in its definition.

    I think it would be too much of a burden for Kbuild not to make use
    of any variables using the 'shell' keyword, so I tend to agree that
    defending against bare 'export' statements in Kbuild is rather not
    feasible and it should be fixed in the makefiles of out-of-tree *ko.

> Insert the following in your reproducer, then it will work.

Thanks for the demo code snippet.

-- 
Best regards,
Eugeniu Rosca