[RFC] syzbot process

[Dmitry Vyukov]

Hello,

You might have seen bug reports coming from syzbot on LKML recently.
syzbot is an automated system that continuously fuzzes main Linux
kernel branches using syzkaller fuzzer and reports all found bugs:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md
So far it has reported ~280 bugs in slightly less than 2 months:
https://groups.google.com/forum/#!forum/syzkaller-bugs

One of the ideas behind syzbot was maximum automation because we
simply could not keep track, and not even report all bugs that
syzkaller finds (the number has crossed 1000). Bugs were massively
lost, not reported, context was lost (e.g. kernel commit, config,
etc), we could not report similarly looking crashes, we could not test
proposed fixes, etc. syzbot aims at solving all of these problems.

However, the cost is that it needs to understand statuses of bugs:
most importantly, what commit fixes what bug. It also has support for
marking a bug as "invalid", e.g. happened once but most likely was
caused by a previous silent memory corruption. And support for marking
bugs as duplicates of other bugs, i.e. the same root cause and will be
fixed when the target bug is fixed. These simple rules are outlined in
the footer of each report and also explained in more detail at the
referenced link:

----------------------------------
This bug is generated by a dumb bot. It may contain errors.
See https://goo.gl/tpsmEJ for details.
Direct all questions to syzkaller@xxxxxxxxxxxxxxxxx
Please credit me with: Reported-by: syzbot <syzkaller@xxxxxxxxxxxxxxxx>
syzbot will keep track of this bug report.
Once a fix for this bug is merged into any tree, reply to this email with:
#syz fix: exact-commit-title
If you want to test a patch for this bug, please reply with:
#syz test: git://repo/address.git branch
and provide the patch inline or as an attachment.
To mark this as a duplicate of another syzbot report, please reply with:
#syz dup: exact-subject-of-another-report
If it's a one-off invalid bug report, please reply with:
#syz invalid
Note: if the crash happens again, it will cause creation of a new bug  report.
Note: all commands must start from beginning of the line in the email body.
----------------------------------

Status tracking allows syzbot to (1) keep track of still unfixed bugs
(more than half actually gets lost in LKML archives if nobody keeps
track of them), (2) be able to ever report similarly looking crashes
as new bugs in future, (3) be able to test fixes.

The problem is that these rules are mostly not followed.

I understand that following these rules is a moderate pain and I am
reaching to you to discuss potential solutions for this problem. I've
tried hard to come up with a scheme that would eliminate sending these
replies altogether, but failed.
We can simply to agree to follow the current convention, which is not
too hard in the end and sounds like a fair deal -- we give you bugs,
you give us fixing commits.
Or, we could somehow employ bugzilla.kernel.org for systematic bug
tracking. However, I've got an impression that it's mostly unused and
not used systematically even when used (e.g. bugs fixed 5 years ago
still rest there as open).
One idea that was proposed is do it the other way around. Namely,
syzbot gives you bug id, and you need to add a tag along the lines of
"syzbot-fix: HASH" to the commit. I don't know if kernel community
finds this easier to use or not. And dups, invalid bugs and missed
tags still needs to be handled in some other way (e.g. the current
way).
Any other proposals, thoughts, ideas?

Thank you