Re: [RFC] syzbot process

[Eric Biggers]

On Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:
> However, the cost is that it needs to understand statuses of bugs:
> most importantly, what commit fixes what bug. It also has support for
> marking a bug as "invalid", e.g. happened once but most likely was
> caused by a previous silent memory corruption. And support for marking
> bugs as duplicates of other bugs, i.e. the same root cause and will be
> fixed when the target bug is fixed. These simple rules are outlined in
> the footer of each report and also explained in more detail at the
> referenced link:
> 
> ----------------------------------
> This bug is generated by a dumb bot. It may contain errors.
> See https://goo.gl/tpsmEJ for details.
> Direct all questions to syzkaller@xxxxxxxxxxxxxxxxx
> Please credit me with: Reported-by: syzbot <syzkaller@xxxxxxxxxxxxxxxx>
> syzbot will keep track of this bug report.
> Once a fix for this bug is merged into any tree, reply to this email with:
> #syz fix: exact-commit-title
> If you want to test a patch for this bug, please reply with:
> #syz test: git://repo/address.git branch
> and provide the patch inline or as an attachment.
> To mark this as a duplicate of another syzbot report, please reply with:
> #syz dup: exact-subject-of-another-report
> If it's a one-off invalid bug report, please reply with:
> #syz invalid
> Note: if the crash happens again, it will cause creation of a new bug  report.
> Note: all commands must start from beginning of the line in the email body.
> ----------------------------------
> 
> Status tracking allows syzbot to (1) keep track of still unfixed bugs
> (more than half actually gets lost in LKML archives if nobody keeps
> track of them), (2) be able to ever report similarly looking crashes
> as new bugs in future, (3) be able to test fixes.
> 
> The problem is that these rules are mostly not followed.

As others mentioned, allowing a bug ID to be in the fix's commit message,
perhaps in the Reported-by line which syzbot already suggests to include, would
make things a bit easier.

But I think the larger problem is that people in the community don't have any
visibility into the statuses of the bugs, so they don't have any motivation to
manage the statuses.

Are you planning to make a dashboard app publicly available for upstream kernel
bugs being tracked by syzbot?  I think it would be very useful for the
community, especially for finding more details about a bug, e.g. when was it
last seen, how often was it seen, has it been seen in multiple trees.  Also for
finding duplicates which may not have been sent to the correct mailing list.

syzbot also should be sending out reminders for bugs that are still open if the
crash is still occurring, and even moreso if there is a reproducer.

However, if the crash isn't still occurring, then I expect it will become
necessary to automatically invalidate the bug after some time, lest the list of
bugs grow without bound due to bugs that have already been fixed that no one has
time to debug to figure out exactly when/what the fix was, especially if there
is no reproducer.  Or perhaps the bug was only in linux-next and only existed
due to a buggy patch which was dropped or modified before it reached mainline,
so there is no "fix" commit.

Eric