Re: syzbot dashboard

[Dmitry Vyukov]

On Wed, Mar 21, 2018 at 9:32 PM, Linus Torvalds
<torvalds@xxxxxxxxxxxxxxxxxxxx> wrote:
> On Wed, Mar 21, 2018 at 9:11 AM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
>>
>> syzkaller/syzbot dashboard is now live at:
>> https://syzkaller.appspot.com
>
> Ok, this may well be the thing that makes syzbot reports useful, if
> they point to an external report instead of sending an absolutely huge
> illegible email that nobody can read.

The dashboard was initially meant to be used only for internal
monitoring (communication with kernel developers over emails) and I am
too close to the project to see obvious stuff. So thanks for the
feedback!

Do you mean to replace all attachments in syzbot emails with links to
web dashboard content?
I've heard opinions both ways (including "I am not clicking on any
external links, include all content in the email"). So we need to
decide. Provided that lots of email clients seem to inline
attachments, rather than show them as separate files, I am leaning
towards providing just links. Then emails can be short and tidy. Also
will solve the problem of "I don't see original attachments in your
forwarded email".


> That said, I do think it needs some better summarizing still.
>
> For example, landing on that front page and then going "Hmm, let's
> look at that first report", I click on that
>
>     [upstream] BUG: corrupted list in remove_wait_queue
>
> thing, and get to
>
>     https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6
>
> fine. That page itself doesn't actually tell me really anything at all, though.
>
> So I go to the first thing I see, click "log" and I get 6500 lines of
> basically line noise.
>
> Ok, so the real thing is under "report", which actually looks pretty legible.
>
> Looking at a few other of those things, I get the same feeling. Can
> you put one copy of the "report" in the main page for a bug? I assume
> they are all slightly different, but there must be some commonality to
> them that you group the syzcaller bugs by, and giving one of those
> legible reports (with all the nice filename and line information and
> basic register state) would likely be a good thing.

This is now done and deployed, the page show a sample crash at the top:
https://syzkaller.appspot.com/bug?id=c11299b410c0feaf0d861c64bcb3a67a639d17a6


> At that point, *if* a report has a reproducer, then sending reminders
> to people with "this still happens, here's a link to the syzkaller
> page for this report" might be much better received than the old huge
> and very-hard-to-read emails.


Now with your blessing we will do reminders :)
Not sure what should be periodicity, though. Obviously less frequent
than once per day. But probably more frequent than once per release.

For the bugs *without* reproducers, now we know precisely that lots of
them are fixable. For bugs *with* reproducers fix rate is 76%, but for
bugs *without* reproducers it's only down to 66% and more than 100
bugs were fixed without reproducers. So it can make sense to do
reminders for them later too.


> The reminder might well want to have that legible and short "report"
> in it too, so that people can just look at that to tell "is this
> relevant for me" particularly if they perhaps already fixed it?
>
> (I only looked at a handful of reports, but the ones I looked at all
> seemed reasonable - maybe some are less so?)

There is some, small fraction of unreasonable crashes. Part of them is
due to induced crashes after silent memory corruptions. But one can
reply with "#syz invalid" to a syzbot email, and such reports will go
off the dashboard.