Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO
From: Ricardo Ribalda
Date: Fri Sep 08 2023 - 17:18:07 EST
Hi Song
On Fri, 8 Sept 2023 at 01:08, Song Liu <song@xxxxxxxxxx> wrote:
>
> Hi Ricardo and folks,
>
> On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@xxxxxxxxxxxx> wrote:
> >
> > When upreving llvm I realised that kexec stopped working on my test
> > platform.
> >
> > The reason seems to be that due to PGO there are multiple .text sections
> > on the purgatory, and kexec does not supports that.
> >
> > Signed-off-by: Ricardo Ribalda <ribalda@xxxxxxxxxxxx>
>
> We are seeing WARNINGs like the following while kexec'ing a PGO and
> LTO enabled kernel:
>
> WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
> kexec_load_purgatory+0x37f/0x390
>
> AFAICT, the warning was added by this set, and it was triggered when
> we have many .text sections
> in purgatory.ro. The kexec was actually successful. So I wonder
> whether we really need the
> WARNING here. If we disable LTO (PGO is still enabled), we don't see
> the WARNING any more.
>
> I also tested an older kernel (5.19 based), where we also see many
> .text sections with LTO. It
> kexec()'ed fine. (It doesn't have the WARN_ON() in
> kexec_purgatory_setup_sechdrs).
You have been "lucky" that the code has chosen the correct start
address, you need to modify the linker script of your kernel to
disable PGO.
You need to backport a patch like this:
https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJvAeQ@xxxxxxxxxxxxxx/T/#md68b7f832216b0c56bbec0c9b07332e180b9ba2b
(assuming x86)
Regards
>
> Please help us fix this properly (as I really don't know much about kexec).
>
> Thanks in advance,
> Song
>
>
> Here is readelf -S output on purgatory.ro.
>
> With LTO:
>
> readelf -W -S purgatory.ro
> There are 48 section headers, starting at offset 0x4a10:
>
> Section Headers:
> [Nr] Name Type Address Off Size
> ES Flg Lk Inf Al
> [ 0] NULL 0000000000000000 000000
> 000000 00 0 0 0
> [ 1] .text PROGBITS 0000000000000000 000040
> 0000d0 00 AX 0 0 16
> [ 2] .data PROGBITS 0000000000000000 001000
> 001000 00 WA 0 0 4096
> [ 3] .rela.text RELA 0000000000000000 003788
> 000228 18 I 45 1 8
> [ 4] .rodata PROGBITS 0000000000000000 002000
> 0000e0 00 A 0 0 16
> [ 5] .rela.rodata RELA 0000000000000000 0039b0
> 000030 18 I 45 4 8
> [ 6] .bss NOBITS 0000000000000000 0020e0
> 001000 00 WA 0 0 4096
> [ 7] .text.purgatory PROGBITS 0000000000000000 0020e0
> 0000df 00 AX 0 0 16
> [ 8] .rela.text.purgatory RELA 0000000000000000 0039e0
> 000060 18 I 45 7 8
> [ 9] .text.warn PROGBITS 0000000000000000 0021c0
> 000001 00 AX 0 0 16
> [10] .kexec-purgatory PROGBITS 0000000000000000 0021d0
> 000120 00 WA 0 0 16
> [11] .comment PROGBITS 0000000000000000 003a40
> 000046 01 MS 0 0 1
> [12] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003a86
> 000005 00 E 0 0 1
> [13] .text.sha256_update PROGBITS 0000000000000000 0022f0
> 0008eb 00 AX 0 0 16
> [14] .rela.text.sha256_update RELA 0000000000000000
> 003a90 000060 18 I 45 13 8
> [15] .text.sha224_update PROGBITS 0000000000000000 002be0
> 00000c 00 AX 0 0 16
> [16] .rela.text.sha224_update RELA 0000000000000000
> 003af0 000018 18 I 45 15 8
> [17] .text.sha256_final PROGBITS 0000000000000000 002bf0
> 0000cd 00 AX 0 0 16
> [18] .rela.text.sha256_final RELA 0000000000000000 003b08
> 000030 18 I 45 17 8
> [19] .text.sha224_final PROGBITS 0000000000000000 002cc0
> 0000bd 00 AX 0 0 16
> [20] .rela.text.sha224_final RELA 0000000000000000 003b38
> 000030 18 I 45 19 8
> [21] .text.sha256 PROGBITS 0000000000000000 002d80
> 00011d 00 AX 0 0 16
> [22] .rela.text.sha256 RELA 0000000000000000 003b68
> 000030 18 I 45 21 8
> [23] .modinfo PROGBITS 0000000000000000 002e9d
> 000039 00 A 0 0 1
> [24] .rodata.SHA256_K PROGBITS 0000000000000000 002ee0
> 000100 00 A 0 0 16
> [25] .rodata.__sha256_final.padding PROGBITS 0000000000000000
> 002fe0 000040 00 A 0 0 16
> [26] .text.memcmp PROGBITS 0000000000000000 003020
> 00000b 00 AX 0 0 16
> [27] .text.bcmp PROGBITS 0000000000000000 003030
> 00000b 00 AX 0 0 16
> [28] .text.strcmp PROGBITS 0000000000000000 003040
> 000041 00 AX 0 0 16
> [29] .text.strncmp PROGBITS 0000000000000000 003090
> 00003a 00 AX 0 0 16
> [30] .text.strnlen PROGBITS 0000000000000000 0030d0
> 000039 00 AX 0 0 16
> [31] .text.atou PROGBITS 0000000000000000 003110
> 000035 00 AX 0 0 16
> [32] .text.simple_strtoull PROGBITS 0000000000000000 003150
> 0000b6 00 AX 0 0 16
> [33] .text.simple_strtol PROGBITS 0000000000000000 003210
> 0001b6 00 AX 0 0 16
> [34] .text.strlen PROGBITS 0000000000000000 0033d0
> 00001c 00 AX 0 0 16
> [35] .text.strstr PROGBITS 0000000000000000 0033f0
> 00005f 00 AX 0 0 16
> [36] .text.strchr PROGBITS 0000000000000000 003450
> 000022 00 AX 0 0 16
> [37] .text.kstrtoull PROGBITS 0000000000000000 003480
> 000142 00 AX 0 0 16
> [38] .text.boot_kstrtoul PROGBITS 0000000000000000 0035d0
> 00000c 00 AX 0 0 16
> [39] .rela.text.boot_kstrtoul RELA 0000000000000000
> 003b98 000018 18 I 45 38 8
> [40] .text.memset PROGBITS 0000000000000000 0035e0
> 00001f 00 AX 0 0 16
> [41] .text.memmove PROGBITS 0000000000000000 003600
> 0000a6 00 AX 0 0 16
> [42] .text.memcpy PROGBITS 0000000000000000 0036b0
> 0000a6 00 AX 0 0 16
> [43] .rodata.str1.1 PROGBITS 0000000000000000 003756
> 000032 01 AMS 0 0 1
> [44] .note.GNU-stack PROGBITS 0000000000000000 003bb0
> 000000 00 0 0 1
> [45] .symtab SYMTAB 0000000000000000 003bb0
> 000948 18 47 68 8
> [46] .shstrtab STRTAB 0000000000000000 0044f8
> 0002cd 00 0 0 1
> [47] .strtab STRTAB 0000000000000000 0047c5
> 000248 00 0 0 1
> Key to Flags:
> W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
> L (link order), O (extra OS processing required), G (group), T (TLS),
> C (compressed), x (unknown), o (OS specific), E (exclude),
> l (large), p (processor specific)
>
>
> Without LTO:
>
> readelf -W -S purgatory.ro
> There are 16 section headers, starting at offset 0x4130:
>
> Section Headers:
> [Nr] Name Type Address Off Size
> ES Flg Lk Inf Al
> [ 0] NULL 0000000000000000 000000
> 000000 00 0 0 0
> [ 1] .text PROGBITS 0000000000000000 000040
> 00131b 00 AX 0 0 16
> [ 2] .rela.text RELA 0000000000000000 003290
> 000480 18 I 13 1 8
> [ 3] .kexec-purgatory PROGBITS 0000000000000000 001360
> 000120 00 WA 0 0 16
> [ 4] .comment PROGBITS 0000000000000000 003710
> 000046 01 MS 0 0 1
> [ 5] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 003756
> 000005 00 E 0 0 1
> [ 6] .data PROGBITS 0000000000000000 002000
> 001000 00 WA 0 0 4096
> [ 7] .rodata PROGBITS 0000000000000000 003000
> 000220 00 A 0 0 16
> [ 8] .rela.rodata RELA 0000000000000000 003760
> 000030 18 I 13 7 8
> [ 9] .bss NOBITS 0000000000000000 003220
> 001000 00 WA 0 0 4096
> [10] .modinfo PROGBITS 0000000000000000 003220
> 000039 00 A 0 0 1
> [11] .rodata.str1.1 PROGBITS 0000000000000000 003259
> 000032 01 AMS 0 0 1
> [12] .note.GNU-stack PROGBITS 0000000000000000 003790
> 000000 00 0 0 1
> [13] .symtab SYMTAB 0000000000000000 003790
> 0006d8 18 15 43 8
> [14] .shstrtab STRTAB 0000000000000000 003e68
> 00009c 00 0 0 1
> [15] .strtab STRTAB 0000000000000000 003f04
> 00022b 00 0 0 1
> Key to Flags:
> W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
> L (link order), O (extra OS processing required), G (group), T (TLS),
> C (compressed), x (unknown), o (OS specific), E (exclude),
> l (large), p (processor specific)
>
> [...]
>
> > Ricardo Ribalda (4):
> > kexec: Support purgatories with .text.hot sections
> > x86/purgatory: Remove PGO flags
> > powerpc/purgatory: Remove PGO flags
> > riscv/purgatory: Remove PGO flags
> >
> > arch/powerpc/purgatory/Makefile | 5 +++++
> > arch/riscv/purgatory/Makefile | 5 +++++
> > arch/x86/purgatory/Makefile | 5 +++++
> > kernel/kexec_file.c | 14 +++++++++++++-
> > 4 files changed, 28 insertions(+), 1 deletion(-)
> > ---
> > base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f
> > change-id: 20230321-kexec_clang16-4510c23d129c
> >
> > Best regards,
> > --
> > Ricardo Ribalda Delgado <ribalda@xxxxxxxxxxxx>
> >
--
Ricardo Ribalda