Re: [PATCH v3 1/3] scripts: kconfig: merge_config.sh: refactor from shell/sed/grep to awk

[Mikko Rapeli]

Hi,

On Tue, Mar 10, 2026 at 10:55:46AM +0100, Andreas Larsson wrote:
> On 2026-03-10 08:50, Mikko Rapeli wrote:
> > On Mon, Mar 09, 2026 at 10:09:04AM -0700, Nathan Chancellor wrote:
> >> On Mon, Mar 09, 2026 at 05:38:58PM +0100, Andreas Larsson wrote:
> >>> On 2026-01-22 11:57, Mikko Rapeli wrote:
> >>>> From: Anders Roxell <anders.roxell@xxxxxxxxxx>
> >>>>
> >>>> merge_config.sh shell/sed/grep loop scales poorly and is slow.
> >>>> With Yocto genericarm64 kernel and around 190 config fragments
> >>>> the script takes more than 20 minutes to run on a fast build machine.
> >>>> Re-implementation with awk does the same job in 10 seconds.
> >>>> Using awk since it is likely available in the build environments
> >>>> and using perl, python etc would introduce more complex runtime
> >>>> dependencies. awk is good enough and lot better than shell/sed/grep.
> >>>>
> >>>> Output stays the same but changed execution time means that
> >>>> parallel job output may be ordered differently.
> >>>>
> >>>> Signed-off-by: Anders Roxell <anders.roxell@xxxxxxxxxx>
> >>>> Signed-off-by: Mikko Rapeli <mikko.rapeli@xxxxxxxxxx>
> >>>> ---
> >>>>  scripts/kconfig/merge_config.sh | 168 ++++++++++++++++++++++++--------
> >>>>  1 file changed, 128 insertions(+), 40 deletions(-)
> >>>
> >>> Hi,
> >>>
> >>> Commit 5fa9b82cbcfc ("scripts: kconfig: merge_config.sh: refactor from
> >>> shell/sed/grep to awk") breaks merge_config.sh for me:
> >>>
> >>> ---------->%----------
> >>> $ make tiny.config
> >>>   HOSTCC  scripts/basic/fixdep
> >>>   HOSTCC  scripts/kconfig/conf.o
> >>>   HOSTCC  scripts/kconfig/confdata.o
> >>>   HOSTCC  scripts/kconfig/expr.o
> >>>   LEX     scripts/kconfig/lexer.lex.c
> >>>   YACC    scripts/kconfig/parser.tab.[ch]
> >>>   HOSTCC  scripts/kconfig/lexer.lex.o
> >>>   HOSTCC  scripts/kconfig/menu.o
> >>>   HOSTCC  scripts/kconfig/parser.tab.o
> >>>   HOSTCC  scripts/kconfig/preprocess.o
> >>>   HOSTCC  scripts/kconfig/symbol.o
> >>>   HOSTCC  scripts/kconfig/util.o
> >>>   HOSTLD  scripts/kconfig/conf
> >>> The base file '.config' does not exist. Creating one...
> >>> Using .config as base
> >>> Merging ./kernel/configs/tiny.config
> >>> awk: cannot open ./.tmp.config.U9SROCKTBj.new (No such file or directory)
> >>> mv: cannot stat './.tmp.config.U9SROCKTBj.new': No such file or directory
> >>> make[2]: *** [scripts/kconfig/Makefile:112: tiny.config] Error 1
> >>> make[1]: *** [<kernel-source-top-dir>/Makefile:744: tiny.config] Error 2
> >>> make: *** [Makefile:248: __sub-make] Error 2
> >>> ---------->%----------
> >>>
> >>> with this underlying call to merge_config.sh, an empty .config and where
> >>> my awk is GNU Awk 5.1.0:
> >>>
> >>> ---------->%----------
> >>> $ make tiny.config V=1
> >>> make --no-print-directory -C <kernel-source-top-dir> \
> >>> -f <kernel-source-top-dir>/Makefile tiny.config
> >>> make -f ./scripts/Makefile.build obj=scripts/basic
> >>> make -f ./scripts/Makefile.build obj=scripts/kconfig tiny.config
> >>> # cmd_merge_fragments tiny.config
> >>>   ./scripts/kconfig/merge_config.sh -m .config ./kernel/configs/tiny.config ./arch/x86/configs/tiny.config
> >>> ...
> >>>
> >>> $ awk --version
> >>> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)
> >>> ---------->%----------
> >>>
> >>> At 5fa9b82cbcfc~ things works as expected.
> > 
> > Hmm, all these steps work for me with GNU awk 5.1.0 so something else
> > is different here. On line 267 awk is given input file args
> > "$ORIG_MERGE_FILE" "$TMP_FILE" "$TMP_FILE.new" and it fails to
> > create the output file "$TMP_FILE.new" with >> and plain print statement.
> > 
> >> Hmmm, not sure how I have not seen this error myself since I test
> >> tiny.config but I can reproduce with a clean output directory. Something
> >> like this seems like a simple fix but the only instances of ARGV[3] in
> >> the awk script that I can see use '>>', which should create the file if
> >> it does not exist. Mikko, any ideas?
> >>
> >> diff --git a/scripts/kconfig/merge_config.sh b/scripts/kconfig/merge_config.sh
> >> index 735e1de450c6..070ecae87a1c 100755
> >> --- a/scripts/kconfig/merge_config.sh
> >> +++ b/scripts/kconfig/merge_config.sh
> >> @@ -123,6 +123,7 @@ fi
> >>  MERGE_LIST=$*
> >>  
> >>  TMP_FILE=$(mktemp ./.tmp.config.XXXXXXXXXX)
> >> +touch "$TMP_FILE.new"
> >>  
> >>  echo "Using $INITFILE as base"
> > 
> > This could help but I fail to understand why this would be needed. Why is awk
> > not able to create this file on line 256?
> > 
> > awk manual says:
> > 
> > https://www.gnu.org/software/gawk/manual/html_node/Redirection.html
> > 
> >     print items >> output-file
> > 
> >     This redirection prints the items into the preexisting output file named
> >     output-file. The difference between this and the single-‘>’ redirection
> >     is that the old contents (if any) of output-file are not erased. Instead,
> >     the awk output is appended to the file. If output-file does not exist,
> >     then it is created.
> 
> Awk is opening the file for reading and fails before any prints with >>
> to it, much like this one does:
> 
> $ awk 'FILENAME == "non-existent" {nextfile}' non-existent
> awk: fatal: cannot open file `non-existent' for reading: No such file or directory
> 
> $ strace awk 'FILENAME == "non-existent" {nextfile}' non-existent |& grep non-existent
> execve("/usr/bin/awk", ["awk", "FILENAME == \"non-existent\" {next"..., "non-existent"], 0x7ffd32bda2e8 /* 66 vars */) = 0
> openat(AT_FDCWD, "non-existent", O_RDONLY) = -1 ENOENT (No such file or directory)
> write(2, "cannot open file `non-existent' "..., 70cannot open file `non-existent' for reading: No such file or directory) = 70
> 
> 
> > And in all my testing this works. Which distro is this? Is something else like
> > file system setup in some way unusual?
> Ubuntu 22.04.5 and a regular ext4 file system.

Right, so some awk versions behave differently. For the record in my testing on
Ubuntu 22.04.5 LTS (Jammy Jellyfish) this script works :/

Can you try if the touch fix from Nathan works?

Cheers,

-Mikko