Re: [RFC] syzbot process

[Dmitry Vyukov]

On Thu, Dec 21, 2017 at 6:05 PM, Greg Kroah-Hartman
<gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
> On Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:
>> Hello,
>>
>> You might have seen bug reports coming from syzbot on LKML recently.
>> syzbot is an automated system that continuously fuzzes main Linux
>> kernel branches using syzkaller fuzzer and reports all found bugs:
>> https://github.com/google/syzkaller/blob/master/docs/syzbot.md
>> So far it has reported ~280 bugs in slightly less than 2 months:
>> https://groups.google.com/forum/#!forum/syzkaller-bugs
>>
>> One of the ideas behind syzbot was maximum automation because we
>> simply could not keep track, and not even report all bugs that
>> syzkaller finds (the number has crossed 1000). Bugs were massively
>> lost, not reported, context was lost (e.g. kernel commit, config,
>> etc), we could not report similarly looking crashes, we could not test
>> proposed fixes, etc. syzbot aims at solving all of these problems.
>>
>> However, the cost is that it needs to understand statuses of bugs:
>> most importantly, what commit fixes what bug. It also has support for
>> marking a bug as "invalid", e.g. happened once but most likely was
>> caused by a previous silent memory corruption. And support for marking
>> bugs as duplicates of other bugs, i.e. the same root cause and will be
>> fixed when the target bug is fixed. These simple rules are outlined in
>> the footer of each report and also explained in more detail at the
>> referenced link:
>>
>> ----------------------------------
>> This bug is generated by a dumb bot. It may contain errors.
>> See https://goo.gl/tpsmEJ for details.
>> Direct all questions to syzkaller@xxxxxxxxxxxxxxxxx
>> Please credit me with: Reported-by: syzbot <syzkaller@xxxxxxxxxxxxxxxx>
>> syzbot will keep track of this bug report.
>> Once a fix for this bug is merged into any tree, reply to this email with:
>> #syz fix: exact-commit-title
>> If you want to test a patch for this bug, please reply with:
>> #syz test: git://repo/address.git branch
>> and provide the patch inline or as an attachment.
>> To mark this as a duplicate of another syzbot report, please reply with:
>> #syz dup: exact-subject-of-another-report
>> If it's a one-off invalid bug report, please reply with:
>> #syz invalid
>> Note: if the crash happens again, it will cause creation of a new bug  report.
>> Note: all commands must start from beginning of the line in the email body.
>> ----------------------------------
>>
>> Status tracking allows syzbot to (1) keep track of still unfixed bugs
>> (more than half actually gets lost in LKML archives if nobody keeps
>> track of them), (2) be able to ever report similarly looking crashes
>> as new bugs in future, (3) be able to test fixes.
>>
>> The problem is that these rules are mostly not followed.
>>
>> I understand that following these rules is a moderate pain and I am
>> reaching to you to discuss potential solutions for this problem. I've
>> tried hard to come up with a scheme that would eliminate sending these
>> replies altogether, but failed.
>> We can simply to agree to follow the current convention, which is not
>> too hard in the end and sounds like a fair deal -- we give you bugs,
>> you give us fixing commits.
>> Or, we could somehow employ bugzilla.kernel.org for systematic bug
>> tracking. However, I've got an impression that it's mostly unused and
>> not used systematically even when used (e.g. bugs fixed 5 years ago
>> still rest there as open).
>> One idea that was proposed is do it the other way around. Namely,
>> syzbot gives you bug id, and you need to add a tag along the lines of
>> "syzbot-fix: HASH" to the commit. I don't know if kernel community
>> finds this easier to use or not. And dups, invalid bugs and missed
>> tags still needs to be handled in some other way (e.g. the current
>> way).
>> Any other proposals, thoughts, ideas?
>
> No, don't use bugzilla :)

That was my impression as well :)

> But, use what bugzilla does do well, provide a unique identifier that
> can be referenced in git commit messages to point people at the problem
> report.
>
> Why not generate a unique one per "problem" you find?  And have a way to
> look them up somehow?
>
> Then you can put:
>         syzkaller-id: XXXXXX
> in the commit log and you can track it easier?
>
> That's what all other "tools" do that try to track kernel bug reports.
> We do that for coverity as one such example, and lots of different bug
> trackers as well.

syzbot is capable of generating unique id's itself, and later we plan
to create an UI to look up bugs, etc (a-la bugzilla) because syzbot
does have all that info. So integrating with bugzilla won't directly
help to solve _this_ problem. We could upload them to bugzilla as
well, but taking into account amount of additional work, this is not
top priority right now.

Thanks