Re: [PATCH v1] kheaders: prevent `find` from seeing perl temp files

From: Hong, Yifan
Date: Thu Dec 05 2024 - 01:16:56 EST

Next message: Ilkka Koskinen: "[PATCH 0/2] perf/dwc_pcie: Fix the swapped event numbers as well as the documentation"
Previous message: Anup Patel: "Re: [PATCH v2 0/4] RISC-V: KVM: Allow Svvptc/Zabha/Ziccrse exts for guests"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Sun, Nov 10, 2024 at 3:18 AM Masahiro Yamada <masahiroy@xxxxxxxxxx> wrote:
>
> On Thu, Nov 7, 2024 at 9:58 AM HONG Yifan <elsk@xxxxxxxxxx> wrote:
> >
> > Symptom:
> >
> > The command
> >
> > find ... | xargs ... perl -i
> >
> > occasionally triggers error messages like the following, with the build
> > still succeeding:
> >
> > Can't open <redacted>/kernel/.tmp_cpio_dir/include/dt-bindings/clock/XXNX4nW9: No such file or directory.
>
>
> I tested this patch on Alpine Linux
> with "for i in $(seq 100); do" loop.
>
>
> [Without this patch]
>
> CHK kernel/kheaders_data.tar.xz
> GEN kernel/kheaders_data.tar.xz
> AR kernel/module/built-in.a
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXKOEPLf:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXFEJdaa:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXdcljBm:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlLaJJF:
> No such file or directory.
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXELBDIO:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXoHheAk:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXeMGhpj:
> No such file or directory.
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXbJmIOD:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiHiAAd:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXjmiien:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXHDjNNM:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlOHclm:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXbOGokp:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXfiMojG:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXhjfhgd:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiHbNEg:
> No such file or directory.
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKiEgne:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXLNlene:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmKpBbk:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXdgokkN:
> No such file or directory.
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDpfMGd:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXhAMmbc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXpaKiKk:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXJNPgAH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXlkCjbB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXmkfeAA:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXFFbADB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXEJlBBd:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXPHMiNP:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXeiMaiG:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXpfopfE:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXLOHHKc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXiaBiIc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXJainoG:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXJFhLEI:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXhPACEf:
> No such file or directory.
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXpMhaAc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXNdoMNL:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXgKMLkg:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXaKkEKh:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXmdOFEo:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmjKioE:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXPeEhNj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXJCnpLp:
> No such file or directory
> Can't open /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXHNOiKd:
> No such file or directory.
> CC kernel/kheaders.o
>
>
>
>
> [With this patch]
>
>
> CHK kernel/kheaders_data.tar.xz
> GEN kernel/kheaders_data.tar.xz
> AR kernel/module/built-in.a
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXeDFldH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXLJMCah:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXckkiOH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXNggIFH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXnjOibB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXkHOnGO:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXMogCin:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXogNJeJ:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXBAgJfo:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXfgFjJH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKiiEfj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXkOEknj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXjbddCE:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDHGOlH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXMGAdak:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXhfAChH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXmGoEnP:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXGJEpaH:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXkKODpJ:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDEfBDk:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXmgjJkj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXBjhcbB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXHoLKEE:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXOgcfAB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXdjoEEc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXFdEDFO:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXiNpHAN:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXopbKfc:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXlJnGkN:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXKfPIJB:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXDfbPkb:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXEiOgDb:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXdCdCIK:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXdoCePE:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXnDjKlh:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXEEcAMM:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/uapi/linux/XXjlDBLj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/include/linux/XXEFPFLj:
> No such file or directory
> find: /home/masahiro/ref/linux-next/kernel/.tmp_cpio_dir/arch/x86/include/asm/XXpoFkCA:
> No such file or directory
> CC kernel/kheaders.o
>
>
>
>
> Indeed, the error messages from perl were fixed, but
> I still see a bunch of error messages,
> which were presumably emitted by 'find'.
>
>
> Do you know how to fix them?

Sorry for the late reply; I just got back from my vacation.

Thanks for taking a look at the patch. The error would suggest that
the patch is not robust enough against different `find`
implementations. I'll upload a separate version of the patch that
properly streams `find` into a temporary file before feeding it into
xargs perl. That way, find shouldn't see the temporary files created
by perl.

>
>
>
>
>
>
>
>
>
> > Analysis:
> >
> > With strace, the root cause has been identified to be `perl -i` creating
> > temporary files inside $cpio_dir, which causes `find` to see the
> > temporary files and emit the names. `find` is likely implemented with
> > readdir. POSIX `readdir` says:
> >
> > If a file is removed from or added to the directory after the most
> > recent call to opendir() or rewinddir(), whether a subsequent call
> > to readdir() returns an entry for that file is unspecified.
> >
> > So if the libc that `find` links against choose to return that entry
> > in readdir(), a possible sequence of events is the following:
> >
> > 1. find emits foo.h
> > 2. xargs executes `perl -i foo.h`
> > 3. perl (pid=100) creates temporary file `XXXXXXXX`
> > 4. find sees file `XXXXXXXX` and emit it
> > 5. PID 100 exits, cleaning up the temporary file `XXXXXXXX`
> > 6. xargs executes `perl -i XXXXXXXX`
> > 7. perl (pid=200) tries to read the file, but it doesn't exist any more.
> >
> > ... triggering the error message.
> >
> > One can reproduce the bug with the following command (assuming PWD
> > contains the list of headers in kheaders.tar.xz)
> >
> > for i in $(seq 100); do
> > find -type f -print0 |
> > xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;';
> > done
> >
> > With a `find` linking against musl libc, the error message is emitted
> > 6/100 times.
> >
> > The fix:
> >
> > This change modifies the command so `find` only emits .h names, thereby
> > skipping the temporary files. Another possible fix would be to store
> > the results of `find` before feeding them into xargs; I didn't take this
> > approach because the current approach appears marginally more optimized
> > and involves a smaller change.
> >
> > Signed-off-by: HONG Yifan <elsk@xxxxxxxxxx>
> > ---
> > kernel/gen_kheaders.sh | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> > index 383fd43ac612..271e0145f406 100755
> > --- a/kernel/gen_kheaders.sh
> > +++ b/kernel/gen_kheaders.sh
> > @@ -84,7 +84,7 @@ for f in $dir_list;
> > done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1
> >
> > # Remove comments except SDPX lines
> > -find $cpio_dir -type f -print0 |
> > +find $cpio_dir -type f -name '*.h' -print0 |
> > xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
> >
> > # Create archive and try to normalize metadata for reproducibility.
> > --
> > 2.47.0.199.ga7371fff76-goog
> >
>
>
> --
> Best Regards
> Masahiro Yamada

Next message: Ilkka Koskinen: "[PATCH 0/2] perf/dwc_pcie: Fix the swapped event numbers as well as the documentation"
Previous message: Anup Patel: "Re: [PATCH v2 0/4] RISC-V: KVM: Allow Svvptc/Zabha/Ziccrse exts for guests"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]