Re: [PATCH] scripts: Fix incremental build header re-generation

[Masahiro Yamada]

On Thu, Mar 18, 2021 at 11:55 AM Jeevan Shriram <jshriram@xxxxxxxxxxxxxx> wrote:
>
>
> On 3/1/2021 7:36 PM, Masahiro Yamada wrote:
> > On Mon, Mar 1, 2021 at 11:23 PM Jeevan Shriram <jshriram@xxxxxxxxxxxxxx> wrote:
> >> compile.h and autoconf.h are ignored when checking headers sha as they
> >> are always re-generated for every kernel compilation. However,
> >> these two headers are packaged into kheaders tar. During incremental
> >> compilation of kernel, kheaders tar file is always generated and re-packaged
> >> irrespective of the changes in headers.
> >
> > I do not see this problem.
> > Could you describe the steps to reproduce it, please?
> >
> Without making any changes in the kernel or it's headers, re-compile the kernel. i.e.,incremental kernel build without any changes.
> I have added following log in gen_kheaders.sh script for confirming the hash differences.


Of course they are different because you are comparing
the hashes of two different files.



>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index b7425a0..ee542a0 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -40,6 +40,10 @@ obj_files_md5="$(find $dir_list -name "*.h"                     |
>   # Any changes to this script will also cause a rebuild of the archive.
>   this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> +
> +echo "Old Tar file $tarfile_md5"


This is the hash of kernel/kheaders_data.tar.xz


> +echo "New Tar file hash $this_file_md5"


This is the hash of kernel/gen_kheaders.sh




> +
>   if [ -f kernel/kheaders.md5 ] &&
>          [ "$(cat kernel/kheaders.md5|head -1)" == "$src_files_md5" ] &&
>          [ "$(cat kernel/kheaders.md5|head -2|tail -1)" == "$obj_files_md5" ] &&
>
> log output :
> 89306 19:29:02.109961   CHK     kernel/kheaders_data.tar.xz
> 89307 19:29:02.109971 Old Tar file 2aa6990e4183c31a862951f4bcac037e
> 89308 19:29:02.109982 New Tar file hash ecf84e700c7cacfe8b35a0905859582d


I do not understand what your claim is.



CHK     kernel/kheaders_data.tar.xz

is displayed when it is checking the hash in order to
determine if the tarball should be updated.


GEN     kernel/kheaders_data.tar.xz

is displayed when the tarball is really updated.





[Incremental build with no change]


masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
Kernel: arch/x86/boot/bzImage is ready  (#2)


[Touch one header and rebuild]

masahiro@oscar:~/ref/linux$ touch include/uapi/drm/exynos_drm.h
masahiro@oscar:~/ref/linux$ make -j24
  DESCEND  objtool
  CALL    scripts/atomic/check-atomics.sh
  CALL    scripts/checksyscalls.sh
  CHK     include/generated/compile.h
  CHK     kernel/kheaders_data.tar.xz
  GEN     kernel/kheaders_data.tar.xz
  CC      kernel/kheaders.o
  AR      kernel/built-in.a
  GEN     .version
  CHK     include/generated/compile.h
  UPD     include/generated/compile.h
  CC      init/version.o
  AR      init/built-in.a
  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  SORTTAB vmlinux
  SYSMAP  System.map
  MODPOST Module.symvers
  CC      arch/x86/boot/version.o
  VOFFSET arch/x86/boot/compressed/../voffset.h
  OBJCOPY arch/x86/boot/compressed/vmlinux.bin
  RELOCS  arch/x86/boot/compressed/vmlinux.relocs
  CC      arch/x86/boot/compressed/kaslr.o
  CC      arch/x86/boot/compressed/misc.o
  GZIP    arch/x86/boot/compressed/vmlinux.bin.gz
  MKPIGGY arch/x86/boot/compressed/piggy.S
  AS      arch/x86/boot/compressed/piggy.o
  LD      arch/x86/boot/compressed/vmlinux
  ZOFFSET arch/x86/boot/zoffset.h
  OBJCOPY arch/x86/boot/vmlinux.bin
  AS      arch/x86/boot/header.o
  LD      arch/x86/boot/setup.elf
  OBJCOPY arch/x86/boot/setup.bin
  BUILD   arch/x86/boot/bzImage
Kernel: arch/x86/boot/bzImage is ready  (#3)







> >> Change-Id: I7a64faebb81df44c32230b0fea1d6df09d7ce66f
> >> Signed-off-by: Jeevan Shriram <jshriram@xxxxxxxxxxxxxx>
> >> ---
> >>   kernel/gen_kheaders.sh | 3 +--
> >>   1 file changed, 1 insertion(+), 2 deletions(-)
> >>
> >> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> >> index c1510f0..5499f72 100755
> >> --- a/kernel/gen_kheaders.sh
> >> +++ b/kernel/gen_kheaders.sh
> >> @@ -51,8 +51,7 @@ this_file_md5="$(ls -l $sfile | md5sum | cut -d ' ' -f1)"
> >>   if [ -f $tarfile ]; then tarfile_md5="$(md5sum $tarfile | cut -d ' ' -f1)"; fi
> >>   if [ -f kernel/kheaders.md5 ] &&
> >>          [ "$(head -n 1 kernel/kheaders.md5)" = "$headers_md5" ] &&
> >> -       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ] &&
> >> -       [ "$(tail -n 1 kernel/kheaders.md5)" = "$tarfile_md5" ]; then
> >> +       [ "$(head -n 2 kernel/kheaders.md5 | tail -n 1)" = "$this_file_md5" ]; then
> >>                  exit
> >>   fi
> >>
> >> --
> >> The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
> >> a Linux Foundation Collaborative Project
> >>
> >
--
Best Regards
Masahiro Yamada