Re: [PATCH] kbuild: pass jobserver to cmd_ld_vmlinux.o

[Sedat Dilek]

On Fri, Jun 17, 2022 at 12:53 AM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
>
> On Thu, Jun 16, 2022 at 4:09 PM Sedat Dilek <sedat.dilek@xxxxxxxxx> wrote:
> >
> > On Thu, Jun 16, 2022 at 12:45 PM Jiri Slaby <jslaby@xxxxxxx> wrote:
> > >
> > > Until the link-vmlinux.sh split (cf. the commit below), the linker was
> > > run with jobserver set in MAKEFLAGS. After the split, the command in
> > > Makefile.vmlinux_o is not prefixed by "+" anymore, so this information
> > > is lost.
> > >
> > > Restore it as linkers working in parallel (esp. the LTO ones) make a use
> > > of it.
> > >
> > > Cc: Sedat Dilek <sedat.dilek@xxxxxxxxx>
> > > Cc: Masahiro Yamada <masahiroy@xxxxxxxxxx>
> > > Cc: Michal Marek <michal.lkml@xxxxxxxxxxx>
> > > Cc: Nick Desaulniers <ndesaulniers@xxxxxxxxxx>
> > > Fixes: 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
> > > Signed-off-by: Jiri Slaby <jslaby@xxxxxxx>
> >
> > Nice catch.
> > ( I have seen some slowdown in my build-time. Will test and report. )
> >
> > Reviewed-by: Sedat Dilek <sedat.dilek@xxxxxxxxx>
> >
>
> No measurable difference in build-time.
>
> Tested-by: Sedat Dilek <sedat.dilek@xxxxxxxxx> # LLVM-14 (x86-64)
>

[ CC Nathan and Sami ]

I re-checked as my build-time takes approx. 15mins more than usual
with Linux v5.17.

[ Perf - seconds time elapsed ]
v5.17: approx. 12500
v5.19: approx. 13500

+1.000 secs = +16,67mins

I am still investigating.

I use LLVM v14.0.4 and CONFIG_LTO_CLANG_THIN=y.

When I check for...

$ egrep 'Makefile.vmlinux_o|jobserver-exec|link-vmlinux.sh'
build-log_5.19.0-rc2-2-amd64-clang14-lto.txt
61414:  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64
--thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5"
"--emit-relocs --discard-none -z max-page-size=0x200000
--build-id=sha1  -X --orphan-handling=warn";  true
61426:+ make -f ./scripts/Makefile.vmlinux_o
61427:  python3 ./scripts/jobserver-exec perl
scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o
arch/x86/kernel/head64.o arch/x86/kernel/ebda.o
arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a
arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a
fs/built-in.a ipc/built-in.a security/built-i
n.a crypto/built-in.a block/built-in.a lib/built-in.a
arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a
drivers/built-in.a sound/built-in.a net/built-in.a virt/buil
t-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a
arch/x86/video/built-in.a > .tmp_initcalls.lds
61693:+ echo vmlinux: scripts/link-vmlinux.sh

Plus vmlinux.o

61428:  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm
-import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds
--whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o
arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o
init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a
certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a
security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a
arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a
drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a
arch/x86/pci/built-in.a arch/x86/power/built-in.a
arch/x86/video/built-in.a --no-whole-archive --start-group
--end-group  ; ./tools/objtool/objtool  --hacks=jump_label
--hacks=noinstr   --mcount  --orc  --retpoline    --static-call
--uaccess  --link     --
link vmlinux.o

Pooh, truncated via Gmail? Attached as a file.

Checking via ps...

$ ps -ef | grep l[l]d > lld-jobserver.txt

...and top util shows me only one lld link-job is running (file attached).

Unsure, if in my setup lld linker benefits from jobserver feature or not.

Am I missing something?

My kernel-config is attached.

Regards,
-Sedat-
dileks    310886   10507  0 10:33 pts/1    00:00:00 /bin/sh -c set -e;  echo '  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5" "--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1  -X --orphan-handling=warn";  true';  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5" "--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1  -X --orphan-handling=warn";  true; scripts/basic/fixdep ./.vmlinux.d vmlinux 'sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5" "--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1  -X --orphan-handling=warn";  true' > ./.vmlinux.cmd; rm -f ./.vmlinux.d
dileks    310887  310886  0 10:33 pts/1    00:00:00 sh scripts/link-vmlinux.sh ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1  -X --orphan-handling=warn
dileks    311020  310935  0 10:34 pts/1    00:00:00 /bin/sh -c set -e;  echo '  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group  --end-group  ; ./tools/objtool/objtool  --hacks=jump_label  --hacks=noinstr   --mcount  --orc  --retpoline    --static-call  --uaccess  --link     --link vmlinux.o';  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group  --end-group  ; ./tools/objtool/objtool  --hacks=jump_label  --hacks=noinstr   --mcount  --orc  --retpoline    --static-call  --uaccess  --link     --link vmlinux.o; printf '%s\n' 'cmd_vmlinux.o := ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group  --end-group  ; ./tools/objtool/objtool  --hacks=jump_label  --hacks=noinstr   --mcount  --orc  --retpoline    --static-call  --uaccess  --link     --link vmlinux.o' > ./.vmlinux.o.cmd
dileks    311021  311020 99 10:34 pts/1    00:11:30 ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group

Attachment: config-5.19.0-rc2-2-amd64-clang14-lto
Description: Binary data

61414:  sh scripts/link-vmlinux.sh "ld.lld" "-m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5" "--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1  -X --orphan-handling=warn";  true
61426:+ make -f ./scripts/Makefile.vmlinux_o
61427:  python3 ./scripts/jobserver-exec perl scripts/generate_initcall_order.pl arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a > .tmp_initcalls.lds
61428:  ld.lld -m elf_x86_64 --thinlto-cache-dir=.thinlto-cache -mllvm -import-instr-limit=5 -r -o vmlinux.o -T .tmp_initcalls.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a  lib/lib.a  arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group  --end-group  ; ./tools/objtool/objtool  --hacks=jump_label  --hacks=noinstr   --mcount  --orc  --retpoline    --static-call  --uaccess  --link     --link vmlinux.o


-dileks // 17-Jun-2022