Re: [PATCH] ftrace/x86: fix x86-32 triple fault with graph tracing and suspend-to-ram

[Paul Menzel]

Dear Josh,


On 03/27/17 17:01, Paul Menzel wrote:

> On 03/27/17 16:54, Josh Poimboeuf wrote:
> > On x86-32, with CONFIG_FIRMWARE and multiple CPUs, if you enable
> > function graph tracing and then suspend to RAM, it will triple fault and
> > reboot when it resumes.
> >
> > The first fault happens when booting a secondary CPU:
> >
> > startup_32_smp()
> >   load_ucode_ap()
> >     prepare_ftrace_return()
> >       ftrace_graph_is_dead()
> >         (accesses 'kill_ftrace_graph')
> >
> > The early head_32.S code calls into load_ucode_ap(), which has an an
> > ftrace hook, so it calls prepare_ftrace_return(), which calls
> > ftrace_graph_is_dead(), which tries to access the global
> > 'kill_ftrace_graph' variable with a virtual address, causing a fault
> > because the CPU is still in real mode.
> >
> > The fix is to add a check in prepare_ftrace_return() to make sure it's
> > running in protected mode before continuing.  The check makes sure the
> > stack pointer is a virtual kernel address.  It's a bit of a hack, but
> > it's not very intrusive and it works well enough.
> >
> > For reference, here are a few other ways this could have potentially
> > been fixed:
> >
> > - Move startup_32_smp()'s call to load_ucode_ap() down to *after* paging
> >   is enabled.  (No idea what that would break.)
> >
> > - Track down load_ucode_ap()'s entire callee tree and mark all the
> >   functions 'notrace'.  (Probably not realistic.)
> >
> > - Pause graph tracing in ftrace_suspend_notifier_call() or bringup_cpu()
> >   or __cpu_up(), and ensure that the pause facility can be queried from
> >   real mode.
> >
> > Reported-by: Paul Menzel <pmenzel@xxxxxxxxxxxxx>
> > Signed-off-by: Josh Poimboeuf <jpoimboe@xxxxxxxxxx>
> > ---
> >  arch/x86/kernel/ftrace.c | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
>
> Thank you for debugging this. Itâs great that you were able to reproduce
> this in QEMU. Hopefully, thatâll make for an easy test case. ;-)
>
> > diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
> > index 8f3d9cf..1c5c4e2 100644
> > --- a/arch/x86/kernel/ftrace.c
> > +++ b/arch/x86/kernel/ftrace.c
> > @@ -983,6 +983,17 @@ void prepare_ftrace_return(unsigned long
> > self_addr, unsigned long *parent,
> >      unsigned long return_hooker = (unsigned long)
> >                  &return_to_handler;
> >
> > +    /*
> > +     * When resuming from suspend-to-ram, this function can be
> > indirectly
> > +     * called from early CPU startup code while the CPU is in real mode,
> > +     * which would fail miserably.  Make sure the stack pointer is a
> > +     * virtual address.
> > +     *
> > +     * This check isn't as accurate as virt_addr_valid(), but it
> > should be
> > +     * good enough for this purpose, and it's fast.
> > +     */
> > +    if (unlikely((long)__builtin_frame_address(0) >= 0)) return;
>
> The coding style requires the `return;` to be on a separate line.
>
> > +
> >      if (unlikely(ftrace_graph_is_dead()))
> >          return;
>
> Iâll test your change this evening.

With both patches applied `./analyze_suspend.py -config 
suspend-callgraph.cfg -filter i915` succeeds on a Lenovo X60t, so 
suspend and resume work perfectly, when tracing is enabled.

Tested-by: Paul Menzel <pmenzel@xxxxxxxxxxxxx>

Itâd be awesome, if you could tag both patches for inclusion into the 
stable Linux Kernel series.


Kind regards,

Paul