Re: Auto-invalidating old syzbot reports?

[Eric Biggers]

On Tue, Jun 07, 2022 at 04:37:42PM +0200, 'Aleksandr Nogikh' via syzkaller wrote:
> Hi Eric,
> 
> Thanks for contacting us!
> These are very interesting points.
> 
> Syzbot indeed only closes old bugs without a reproducer, because if we
> have a repro, then we can periodically do a fix bisection. And yes,
> this mechanism unfortunately does not always work perfectly.
> 
> I think we could split the problem you described into two parts.
> 1) Some bugs that are "open" on the dashboard are actually no longer
> relevant and should be closed.
> 
> If you know some old opened bugs with repro, which are actually
> already fixed, could you please share them? It would be helpful to
> figure out the exact reason why they are still open.
> There are some cases when we can close bugs with a repro without
> losing too much -- e.g. for bugs from -next there was a discussion in
> https://github.com/google/syzkaller/issues/1957.
> Also, if the fix bisection fails, but the repro no longer triggers the
> crash on the HEAD, then we could probably "cancel" the repro and let
> the bug be auto-closed (actually, auto-invalidated) later?
> 

Just to give the first definitive example I could find,
https://syzkaller.appspot.com/bug?id=06c43cd0a71aec08de8ea55ca5cda816c45ab4e0
("KMSAN: uninit-value in _mix_pool_bytes") is a 3-year old bug that is still
open even though it was fixed by commit f45a4248ea4c ("net: usb: rtl8150: set
random MAC address when set_ethernet_addr() fails").

>From working on syzbot reports in the past, I can say that the "already fixed"
case for old reports is very common.  It is hard and time-consuming to actually
identify them as such though, since it requires root-causing the bug.  If it was
quick and easy to do so, there wouldn't be hundreds of such open reports...

> 2) Some bugs were reported to the mailing lists, but became forgotten.
> 
> We could periodically take maintainers as per the latest commit and
> send a reminder email to them. What do you think, would people go mad
> if we did that for each bug e.g. every 6 months? :) Only if the bug
> still happens on syzbot, of course.

That is greatly needed, but to get there we first need to get past the
assumption that every syzbot report will get properly handled by humans and thus
should never be automatically closed.  That assumption has been tried for the
last 5 years, and unfortunately it isn't working.  (If responding to syzbot
reports was being properly funded, it would be possible, but it's not.)  It
looks like you agree, as per your suggestion that only crashes that still happen
in syzbot should be reminded about.  I think syzbot actually needs to go further
and close the old bug reports, not merely suppress reminders about them...

In any case, reminders *must* include the latest crash details in a way that
clearly shows that the bug is still relevant.

> At some point we were also considering sending aggregated reminders
> (e.g. sth like "we still have X open bugs for the subsystem you
> maintain/have actively contributed to, here they are:"), but to do
> that, we first need to learn how to more or less reliably classify the
> bugs into the subsystems.

Well syzbot already identifies subsystems via the stack trace; it's just not as
good as a human expert, and probably never will be since the correct subsystem
can be very non-obvious.  For a short time, I was actually manually classifying
the subsystems for syzbot reports and sending out reminders --- see
https://lore.kernel.org/linux-kernel/?q=%22open+syzbot+bugs%22 --- but I gave up
due to lack of support from my job for doing that work, combined with receiving
somewhat less engagement than I had hoped.

Perhaps the best solution would be to crowdsource by providing a self-service
"#syz subsystem $FOO" command.

- Eric