WARNING: at arch/x86/kernel/stacktrace.c:132 save_stack_trace_tsk_reliable

[Jiri Slaby]

Hi,

so we are testing klp with ORC. And we see
WARNING: CPU: 0 PID: 4574 at ../arch/x86/kernel/stacktrace.c:132
save_stack_trace_tsk_reliable+0x130/0x1b0
Modules linked in: ...
...
Call Trace:
 klp_try_switch_task+0x10a/0x2c0
 klp_try_complete_transition+0x121/0x1b0
 __klp_enable_patch+0xb6/0xe0
 klp_enable_patch+0x68/0x70
 livepatch_init+0x28/0x50 [klp_tc_3_live_patch_getpid]




Staring into the code, it is caused by save_stack_trace_tsk of the
CPU0's idle task (task = idle_task(0);).

Its stack trace should look like:
ffffffff811083c2 do_idle+0x142/0x1e0
ffffffff8110861d cpu_startup_entry+0x5d/0x60
ffffffff82715f58 start_kernel+0x3ff/0x407
ffffffff827153e8 x86_64_start_kernel+0x14e/0x15d
ffffffff810001bf secondary_startup_64+0x9f/0xa0
ffffffffffffffff 0xffffffffffffffff



1) To have nice secondary_startup_64 there instead of misleading
verify_cpu+0, I had to do:
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -303,6 +305,7 @@ ENTRY(secondary_startup_64)
        pushq   %rax            # target address in negative space
        lretq
 .Lafter_lret:
+       nop
 ENDPROC(secondary_startup_64)

 #include "verify_cpu.S"




2) secondary_startup_64 lives at ffffffff810001bf. It is very near
_stext (+0x1bf) and there are no orc entries for that and neither before
that. But orc_find still returns the very first orc entry to me even
though my address is lower:
ffffffff810002d0: sp:sp+8 bp:(und) type:call

The entry is for run_init_process from init/main.c.

Obviously secondary_startup_64 is from head_64.S which has no orc
entries due to OBJECT_FILES_NON_STANDARD_head_$(BITS).o := y.

So doing this had no chance:
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -169,6 +169,7 @@ startup_64:
        movq    $(early_level4_pgt - __START_KERNEL_map), %rax
        jmp 1f
 ENTRY(secondary_startup_64)
+       UNWIND_HINT_EMPTY
        /*
         * At this point the CPU runs in 64bit mode CS.L = 1 CS.D = 0,
         * and someone has loaded a mapped page table.


Anyway, I went the way to fix __orc_find anyway. It should return NULL
if the entry is not found (i.e. the found entry has higher IP), so I did:
--- a/arch/x86/kernel/unwind_orc.c
+++ b/arch/x86/kernel/unwind_orc.c
@@ -52,6 +53,8 @@ static struct orc_entry *__orc_find(int
                } else
                        last = mid - 1;
        }
+       if (orc_ip(found) > ip)
+               return NULL;

        return u_table + (found - ip_table);
 }


But now, unwind_init does not succeed, because it cannot find an entry
for LOOKUP_START_IP in the orc_lookup[i] setup loop. Obviously,
head_64.o is at LOOKUP_START_IP with no orc entry in there.

So I did this hack instead of the above:
@@ -52,6 +53,8 @@ static struct orc_entry *__orc_find(int
                } else
                        last = mid - 1;
        }
+       if (orc_init && orc_ip(found) > ip)
+               return NULL;

        return u_table + (found - ip_table);
 }

which produces the nice stack trace on the top.

So now, there are two things:
1) how to fix orc_find & unwind_init properly?
2) how to generate an EMPTY hint for secondary_startup_64 and maybe
later REGS hint after the rsp setup there? The function seems to be a
pretty killer for orc generate (among others in head_64.S):

../orc/arch/x86/kernel/head_64.o: warning: objtool:
secondary_startup_64()+0x25: sibling call from callable instruction with
modified stack frame
../orc/arch/x86/kernel/head_64.o: warning: objtool:
early_idt_handler_array()+0x4: sibling call from callable instruction
with modified stack frame
../orc/arch/x86/kernel/head_64.o: warning: objtool:
early_idt_handler_common()+0x4d: sibling call from callable instruction
with modified stack frame
../orc/arch/x86/kernel/head_64.o: warning: objtool:
secondary_startup_64()+0x9d: unsupported instruction in callable function

thanks,
-- 
js
suse labs