Re: objtool failure caused some kernel functionality not working
From: Yonghong Song
Date: Thu Jan 30 2025 - 02:21:22 EST
On 1/29/25 11:10 PM, Yonghong Song wrote:
In Meta, when I tried to use llvm19 compiler to build a kernel with
PGO ([1]) support for 6.13 kernel, Ihit the following objtool warning:
warning: objtool: __htab_map_lookup_elem+0x3fb: can't find switch
jump table
The kernel is built successfully, but after booting the kernel,
/sys/kernel/debug/tracing/available_filter_functionsand
/sys/kernel/debug/tracing/available_filter_functions_addrs are empty.
I did some investigation and found that the warning is at func
add_jump_table(): static int add_jump_table(struct objtool_file *file,
struct instruction *insn, struct reloc *next_table) { struct symbol
*pfunc = insn_func(insn)->pfunc; struct reloc *table =
insn_jump_table(insn); struct instruction *dest_insn; unsigned int
prev_offset = 0; struct reloc *reloc = table; struct alternative *alt;
/* * Each @reloc is a switch table relocation which points to the
target * instruction. */ for_each_reloc_from(table->sec, reloc) { /*
Check for the end of the table: */ if (reloc != table && reloc ==
next_table) break; /* Make sure the table entries are consecutive: */
if (prev_offset && reloc_offset(reloc) != prev_offset + 8) break; /*
Detect function pointers from contiguous objects: */ if
(reloc->sym->sec == pfunc->sec && reloc_addend(reloc) ==
pfunc->offset) break; dest_insn = find_insn(file, reloc->sym->sec,
reloc_addend(reloc)); <===== find_insn return a NULL dest_insn with
prev_offset == 0 if (!dest_insn) break; /* Make sure the destination
is in the same function: */ if (!insn_func(dest_insn) ||
insn_func(dest_insn)->pfunc != pfunc) break; alt =
malloc(sizeof(*alt)); if (!alt) { WARN("malloc failed"); return -1; }
alt->insn = dest_insn; alt->next = insn->alts; insn->alts = alt;
prev_offset = reloc_offset(reloc); } if (!prev_offset) {
WARN_INSN(insn, "can't find switch jump table"); <===== error message
here. return -1; } return 0; } The find_insn failed for 'prev_offset
== 0' at the second iteration and finally the function return -1,
which propagated to decode_sections() in func check(). Since
decode_sections() got an error ret the later
create_mcount_loc_sections() will not be called. This caused the above
available_filter_functions issue.
Sorry for the mess-up for the above code. The summary is:
The kernel is built successfully, but after booting the kernel,
/sys/kernel/debug/tracing/available_filter_functions and
/sys/kernel/debug/tracing/available_filter_functions_addrs
are empty. I did some investigation and found that the warning is at func
add_jump_table().
The find_insn() failed for 'prev_offset == 0' at the second
iteration and finally the function return -1, which propagated to
decode_sections() in func check(). Since decode_sections() got an error
ret the later create_mcount_loc_sections() will not be called. This
caused the above available_filter_functions issue.
The following are some codes related to func__htab_map_lookup_elem():
Disassembly of section .text.hot.__htab_map_lookup_elem:
0000000000000000 <__htab_map_lookup_elem>:
0: e8 00 00 00 00 callq 0x5
<__htab_map_lookup_elem+0x5>
0000000000000001: R_X86_64_PLT32 __fentry__-0x4
5: 55 pushq %rbp
...
3f3: e9 03 fe ff ff jmp 0x1fb
<__htab_map_lookup_elem+0x1fb>
3f8: 41 ff c8 decl %r8d
3fb: 42 ff 24 c5 00 00 00 00 jmpq *(,%r8,8)
00000000000003ff: R_X86_64_32S
.rodata.hot.__htab_map_lookup_elem
403: 44 0f b6 42 0a movzbl 0xa(%rdx), %r8d
408: 41 c1 e0 10 shll $0x10, %r8d
...
45b: 44 0f b6 42 02 movzbl 0x2(%rdx), %r8d
460: 41 c1 e0 10 shll $0x10, %r8d
464: 44 01 c0 addl %r8d, %eax
467: e9 26 fe ff ff jmp 0x292
<__htab_map_lookup_elem+0x292>
46c: 41 89 d8 movl %ebx, %r8d
46f: 89 c1 movl %eax, %ecx
471: 89 c5 movl %eax, %ebp
473: 48 89 f2 movq %rsi, %rdx
476: e9 0d fe ff ff jmp 0x288
<__htab_map_lookup_elem+0x288>
47b: 44 0f b6 42 0b movzbl 0xb(%rdx), %r8d
480: 41 c1 e0 18 shll $0x18, %r8d
484: 44 01 c5 addl %r8d, %ebp
487: e9 77 ff ff ff jmp 0x403
<__htab_map_lookup_elem+0x403>
Disassembly of section .text.unlikely.bpf_percpu_hash_update:
So the func __htab_map_lookup_elem() size is 0x48c.
Here, insn 0x3fb is a jump table insn, the following is the actual
jump table:
RELOCATION RECORDS FOR [.rodata.hot.__htab_map_lookup_elem]:
OFFSET TYPE VALUE
0000000000000000 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x29e
0000000000000008 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x48c
<==== problematic one
0000000000000010 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x45b
0000000000000018 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x44f
0000000000000020 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x447
0000000000000028 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x43b
0000000000000030 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x42f
0000000000000038 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x423
0000000000000040 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x41b
0000000000000048 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x40f
0000000000000050 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x403
0000000000000058 R_X86_64_64 .text.hot.__htab_map_lookup_elem+0x47b
Note that for the second entry which has
.text.hot.__htab_map_lookup_elem+0x48c
which is right after the last insn. This caused the problem for
objtool since
jump target is not within the function, rather immediately after the
function.
There are quite some similar cases in my build.
The llvm18 does not have issues. I tried to bisect what changed in
llvm19 and
found the following llvm patch is responsible:
https://github.com/llvm/llvm-project/pull/96089
Basically, the compiler might be able to create a jump target which
actually
not possible at runtime. For example, in one of examples in the above
llvm patch, something
like
if i >= 3 goto out; /* i unsigned */
switch i, label default_unreachable:
case 0: goto label1;
case 1: goto label2;
case 2: goto label3;
label1: ...; return;
label2: ...; return;
label3: ...; return;
default_unreachable:
I think that this should be a valid code from compiler perspective.
Can we fix objtool to handle jump target which is immediately after
the func body?Thanks, Yonghong
[1]
https://patchwork.kernel.org/project/linux-kbuild/patch/20210407211704.367039-1-morbo@xxxxxxxxxx/#24246189