Re: [PATCH -tip 1/2] x86/alternative: Sync bp_patching update for avoiding NULL pointer exception

[Masami Hiramatsu]

On Mon, 2 Dec 2019 10:15:19 +0100
Peter Zijlstra <peterz@xxxxxxxxxxxxx> wrote:

> On Wed, Nov 27, 2019 at 02:56:52PM +0900, Masami Hiramatsu wrote:
> > ftracetest multiple_kprobes.tc testcase hit a following NULL pointer
> > exception.
> > 
> > BUG: kernel NULL pointer dereference, address: 0000000000000000
> > #PF: supervisor read access in kernel mode
> > #PF: error_code(0x0000) - not-present page
> > PGD 800000007bf60067 P4D 800000007bf60067 PUD 7bf5f067 PMD 0
> > Oops: 0000 [#1] PREEMPT SMP PTI
> > CPU: 6 PID: 0 Comm: swapper/6 Not tainted 5.4.0-rc8+ #23
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
> > RIP: 0010:poke_int3_handler+0x39/0x100
> > Code: 5b 5d c3 f6 87 88 00 00 00 03 75 f2 48 8b 87 80 00 00 00 48 89 fb 48 8d 68 ff 48 8b 05 80 98 72 01 83 fa 01 0f 8f 93 00 00 00 <48> 63 10 48 81 c2 00 00 00 81 48 39 d5 75 c5 0f b6 50 08 8d 4a 34
> > RSP: 0018:ffffc900001a8eb8 EFLAGS: 00010046
> > RAX: 0000000000000000 RBX: ffffc900001a8ee8 RCX: ffffffff81a00b57
> > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffc900001a8ee8
> > RBP: ffffffff81027635 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
> > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
> > FS:  0000000000000000(0000) GS:ffff88807d980000(0000) knlGS:0000000000000000
> > CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000000 CR3: 000000007a970000 CR4: 00000000000006a0
> > Call Trace:
> >  <IRQ>
> >  do_int3+0xd/0xf0
> >  int3+0x42/0x50
> > RIP: 0010:sched_clock+0x6/0x10
> > 
> > eu-addr2line told that poke_int3_handler+0x39 was alternatives:958.
> > 
> > static inline void *text_poke_addr(struct text_poke_loc *tp)
> > {
> >         return _stext + tp->rel_addr; <------ Here is line #958
> > }
> > 
> > This seems like caused by the tp (bp_patching.vec) was NULL but
> > bp_patching.nr_entries != 0. There is a small chance to do
> > this, because we have no sync after zeroing bp_patching.nr_entries
> > before clearing bp_patching.vec.
> > 
> > Steve suggested we could fix this by adding sync_core, because int3
> > is done with interrupts disabled, and the on_each_cpu() requires
> > all CPUs to have had their interrupts enabled.
> > 
> > Fixes: c0213b0ac03c ("x86/alternative: Batch of patch operations")
> > Suggested-by: Steven Rostedt (VMware) <rostedt@xxxxxxxxxxx>
> > Signed-off-by: Masami Hiramatsu <mhiramat@xxxxxxxxxx>
> > ---
> >  arch/x86/kernel/alternative.c |    8 +++++++-
> >  1 file changed, 7 insertions(+), 1 deletion(-)
> > 
> > diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
> > index 4552795a8df4..9505096e2cd1 100644
> > --- a/arch/x86/kernel/alternative.c
> > +++ b/arch/x86/kernel/alternative.c
> > @@ -1134,8 +1134,14 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries
> >  	 * sync_core() implies an smp_mb() and orders this store against
> >  	 * the writing of the new instruction.
> >  	 */
> > -	bp_patching.vec = NULL;
> >  	bp_patching.nr_entries = 0;
> > +	/*
> > +	 * This sync_core () ensures that all int3 handlers in progress
> > +	 * have finished. This allows poke_int3_handler () after this to
> > +	 * avoid touching bp_paching.vec by checking nr_entries == 0.
> > +	 */
> > +	text_poke_sync();
> > +	bp_patching.vec = NULL;
> >  }
> 
> Hurm.. is there no way we can merge that with the 'last'
> text_poke_sync() ? It seems a little daft to do 2 back-to-back IPI
> things like that.

Maybe we can add a NULL check of bp_patchig.vec in poke_int3_handler()
but it doesn't ensure the fundamental safeness, because the array
pointed by bp_patching.vec itself can be released while
poke_int3_handler() accesses it.

When I hit similar issue on unregister_kprobe, I used synchronize_rcu()
to ensure all cores passed scheduler once. This can avoid sending IPI
but will take a longer time.

Thank you,
-- 
Masami Hiramatsu <mhiramat@xxxxxxxxxx>