Re: kernel panic: stack is corrupted in udp4_lib_lookup2
From: Dmitry Vyukov
Date: Mon Jan 07 2019 - 04:05:13 EST
On Fri, Jan 4, 2019 at 7:05 PM Stefano Brivio <sbrivio@xxxxxxxxxx> wrote:
> On Fri, 4 Jan 2019 18:26:16 +0100
> Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> > On Fri, Jan 4, 2019 at 6:14 PM Stefano Brivio <sbrivio@xxxxxxxxxx> wrote:
> > >
> > > On Fri, 4 Jan 2019 12:05:04 +0100
> > > Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> > >
> > > > I've added these as tests:
> > > >
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/341
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/342
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/343
> > > > https://github.com/google/syzkaller/blob/master/pkg/report/testdata/linux/report/344
> > > >
> > > > Will try to figure out how to distinguish them from true corrupted
> > > > reports. Usually when Call Trace does not have any frames, it's a sign
> > > > of a corrupted report, and in other crashes we see the same report but
> > > > with a stack trace. But some stack-corruption-related reliably don't
> > > > have stack traces (not corrupted). But then some other
> > > > stack-corruption-related crashes do have stack traces, and for these
> > > > no stack trace again means a corrupted kernel output. Amusingly this
> > > > is one of the most complex parts of syzkaller.
> > >
> > > I'm not sure how complicated that would be, but what about some metric
> > > based on valid symbol names being reported?
> > Please elaborate. What do you mean by "valid symbol names"?
> I mean a symbol name listed in /proc/kallsyms on the running system.
> This is usually my minimum threshold for "I can do something with this
> report" -- which doesn't mean it's necessarily valid, but well, if you
> have that, it means that at least something worked in the reporting,
> and you can at least start having a look at a specific function.
> > Note that corrupted output detection solves 2 problems:
> > 1. Do we think the output is truncated to the point of being not useful?
> > E.g. sometimes kernel produces just 1 line:
> > general protection fault: 0000 [#1] PREEMPT SMP KASAN
> > This is sure a crash, but it's not too useful to report.
> Sure. In those tests above you have:
> - 341: udp6_lib_lookup2+0x622, handle_irq+0x2cb
> - 342: __sanitizer_cov_trace_pc+0x8, handle_irq+0x2cb
> - 343: __udp6_lib_err, etc.
> - 344: __udp6_lib_lookup+0x1d, etc.
> and this makes all those reports at least minimally useful.
> > 2. Do we have any reasons to think we extracted bogus crash identity?
> > E.g. crash intermixed with output from another thread so that we say
> > "something-bad in function foo", when in fact function foo come from
> > output of the second non-crashing thread.
> Okay, this looks way more complicated :)
Yeah, unfortunately, it's quite complicated.
Just today this gen popped up. You won't find any ODEBUG checks at
that stack, it's completely unrelated and come from another task.
------------[ cut here ]------------
ODEBUG: free active (active state 0) object type: timer_list hint:
WARNING: CPU: 1 PID: 45 at lib/debugobjects.c:325
CPU: 0 PID: 13619 Comm: syz-executor1 Not tainted 4.20.0+ #13
Kernel panic - not syncing: panic_on_warn set ...
Hardware name: Google Google Compute Engine/Google Compute Engine,
BIOS Google 01/01/2011
__dump_stack lib/dump_stack.c:77 [inline]
__vmalloc_node mm/vmalloc.c:1795 [inline]
__vmalloc_node_flags mm/vmalloc.c:1809 [inline]
__do_sys_write fs/read_write.c:610 [inline]
__se_sys_write fs/read_write.c:607 [inline]