Re: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk

[Andreas Larsson]

On 2026-03-10 08:50, Mikko Rapeli wrote:
> Hi,
> 
> On Mon, Mar 09, 2026 at 10:09:04AM -0700, Nathan Chancellor wrote:
>> On Mon, Mar 09, 2026 at 05:38:58PM +0100, Andreas Larsson wrote:
>>> On 2026-01-22 11:57, Mikko Rapeli wrote:
>>>> From: Anders Roxell <anders.roxell@xxxxxxxxxx>
>>>>
>>>> merge_config.sh shell/sed/grep loop scales poorly and is slow.
>>>> With Yocto genericarm64 kernel and around 190 config fragments
>>>> the script takes more than 20 minutes to run on a fast build machine.
>>>> Re-implementation with awk does the same job in 10 seconds.
>>>> Using awk since it is likely available in the build environments
>>>> and using perl, python etc would introduce more complex runtime
>>>> dependencies. awk is good enough and lot better than shell/sed/grep.
>>>>
>>>> Output stays the same but changed execution time means that
>>>> parallel job output may be ordered differently.
>>>>
>>>> Signed-off-by: Anders Roxell <anders.roxell@xxxxxxxxxx>
>>>> Signed-off-by: Mikko Rapeli <mikko.rapeli@xxxxxxxxxx>
>>>> ---
>>>>  scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++--------
>>>>  1 file changed, 128 insertions(+), 40 deletions(-)
>>>
>>> Hi,
>>>
>>> Commit 5fa9b82cbcfc ("scripts: kconfig: merge_config.sh: refactor from
>>> shell/sed/grep to awk") breaks merge_config.sh for me:
>>>
>>> ---------->%----------
>>> $ make tiny.config
>>>   HOSTCC  scripts/basic/fixdep
>>>   HOSTCC  scripts/kconfig/conf.o
>>>   HOSTCC  scripts/kconfig/confdata.o
>>>   HOSTCC  scripts/kconfig/expr.o
>>>   LEX     scripts/kconfig/lexer.lex.c
>>>   YACC    scripts/kconfig/parser.tab.[ch]
>>>   HOSTCC  scripts/kconfig/lexer.lex.o
>>>   HOSTCC  scripts/kconfig/menu.o
>>>   HOSTCC  scripts/kconfig/parser.tab.o
>>>   HOSTCC  scripts/kconfig/preprocess.o
>>>   HOSTCC  scripts/kconfig/symbol.o
>>>   HOSTCC  scripts/kconfig/util.o
>>>   HOSTLD  scripts/kconfig/conf
>>> The base file '.config' does not exist. Creating one...
>>> Using .config as base
>>> Merging ./kernel/configs/tiny.config
>>> awk: cannot open ./.tmp.config.U9SROCKTBj.new (No such file or directory)
>>> mv: cannot stat './.tmp.config.U9SROCKTBj.new': No such file or directory
>>> make[2]: *** [scripts/kconfig/Makefile:112: tiny.config] Error 1
>>> make[1]: *** [<kernel-source-top-dir>/Makefile:744: tiny.config] Error 2
>>> make: *** [Makefile:248: __sub-make] Error 2
>>> ---------->%----------
>>>
>>> with this underlying call to merge_config.sh, an empty .config and where
>>> my awk is GNU Awk 5.1.0:
>>>
>>> ---------->%----------
>>> $ make tiny.config V=1
>>> make --no-print-directory -C <kernel-source-top-dir> \
>>> -f <kernel-source-top-dir>/Makefile tiny.config
>>> make -f ./scripts/Makefile.build obj=scripts/basic
>>> make -f ./scripts/Makefile.build obj=scripts/kconfig tiny.config
>>> # cmd_merge_fragments tiny.config
>>>   ./scripts/kconfig/merge_config.sh -m .config ./kernel/configs/tiny.config ./arch/x86/configs/tiny.config
>>> ...
>>>
>>> $ awk --version
>>> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
>>> ---------->%----------
>>>
>>> At 5fa9b82cbcfc~ things works as expected.
> 
> Hmm, all these steps work for me with GNU awk 5.1.0 so something else
> is different here. On line 267 awk is given input file args
> "$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new" and it fails to
> create the output file "$TMP_FILE.new" with >> and plain print statement.
> 
>> Hmmm, not sure how I have not seen this error myself since I test
>> tiny.config but I can reproduce with a clean output directory. Something
>> like this seems like a simple fix but the only instances of ARGV[3] in
>> the awk script that I can see use '>>', which should create the file if
>> it does not exist. Mikko, any ideas?
>>
>> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
>> index 735e1de450c6..070ecae87a1c 100755
>> --- a/scripts/kconfig/merge_config.sh
>> +++ b/scripts/kconfig/merge_config.sh
>> @@ -123,6 +123,7 @@ fi
>>  MERGE_LIST=$*
>>  
>>  TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
>> +touch "$TMP_FILE.new"
>>  
>>  echo "Using $INITFILE as base"
> 
> This could help but I fail to understand why this would be needed. Why is awk
> not able to create this file on line 256?
> 
> awk manual says:
> 
> https://www.gnu.org/software/gawk/manual/html_node/Redirection.html
> 
>     print items >> output-file
> 
>     This redirection prints the items into the preexisting output file named
>     output-file. The difference between this and the single-‘>’ redirection
>     is that the old contents (if any) of output-file are not erased. Instead,
>     the awk output is appended to the file. If output-file does not exist,
>     then it is created.

Awk is opening the file for reading and fails before any prints with >>
to it, much like this one does:

$ awk 'FILENAME == "non-existent" {nextfile}' non-existent
awk: fatal: cannot open file `non-existent' for reading: No such file or directory

$ strace awk 'FILENAME == "non-existent" {nextfile}' non-existent |& grep non-existent
execve("/usr/bin/awk", ["awk", "FILENAME == \"non-existent\" {next"..., "non-existent"], 0x7ffd32bda2e8 /* 66 vars */) = 0
openat(AT_FDCWD, "non-existent", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "cannot open file `non-existent' "..., 70cannot open file `non-existent' for reading: No such file or directory) = 70


> And in all my testing this works. Which distro is this? Is something else like
> file system setup in some way unusual?
Ubuntu 22.04.5 and a regular ext4 file system.

Cheers,
Andreas