Re: [RFC] syzbot process

[Ozgur]

28.12.2017, 13:41, "Dmitry Vyukov" <dvyukov@xxxxxxxxxx>:
> On Fri, Dec 22, 2017 at 4:32 AM, Eric Biggers <ebiggers3@xxxxxxxxx> wrote:
>> ÂOn Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:
>>> ÂHowever, the cost is that it needs to understand statuses of bugs:
>>> Âmost importantly, what commit fixes what bug. It also has support for
>>> Âmarking a bug as "invalid", e.g. happened once but most likely was
>>> Âcaused by a previous silent memory corruption. And support for marking
>>> Âbugs as duplicates of other bugs, i.e. the same root cause and will be
>>> Âfixed when the target bug is fixed. These simple rules are outlined in
>>> Âthe footer of each report and also explained in more detail at the
>>> Âreferenced link:
>>>
>>> Â----------------------------------
>>> ÂThis bug is generated by a dumb bot. It may contain errors.
>>> ÂSee https://goo.gl/tpsmEJ for details.
>>> ÂDirect all questions to syzkaller@xxxxxxxxxxxxxxxxx
>>> ÂPlease credit me with: Reported-by: syzbot <syzkaller@xxxxxxxxxxxxxxxx>
>>> Âsyzbot will keep track of this bug report.
>>> ÂOnce a fix for this bug is merged into any tree, reply to this email with:
>>> Â#syz fix: exact-commit-title
>>> ÂIf you want to test a patch for this bug, please reply with:
>>> Â#syz test: git://repo/address.git branch
>>> Âand provide the patch inline or as an attachment.
>>> ÂTo mark this as a duplicate of another syzbot report, please reply with:
>>> Â#syz dup: exact-subject-of-another-report
>>> ÂIf it's a one-off invalid bug report, please reply with:
>>> Â#syz invalid
>>> ÂNote: if the crash happens again, it will cause creation of a new bug report.
>>> ÂNote: all commands must start from beginning of the line in the email body.
>>> Â----------------------------------
>>>
>>> ÂStatus tracking allows syzbot to (1) keep track of still unfixed bugs
>>> Â(more than half actually gets lost in LKML archives if nobody keeps
>>> Âtrack of them), (2) be able to ever report similarly looking crashes
>>> Âas new bugs in future, (3) be able to test fixes.
>>>
>>> ÂThe problem is that these rules are mostly not followed.
>>
>> ÂAs others mentioned, allowing a bug ID to be in the fix's commit message,
>> Âperhaps in the Reported-by line which syzbot already suggests to include, would
>> Âmake things a bit easier.
>>
>> ÂBut I think the larger problem is that people in the community don't have any
>> Âvisibility into the statuses of the bugs, so they don't have any motivation to
>> Âmanage the statuses.
>>
>> ÂAre you planning to make a dashboard app publicly available for upstream kernel
>> Âbugs being tracked by syzbot? I think it would be very useful for the
>> Âcommunity, especially for finding more details about a bug, e.g. when was it
>> Âlast seen, how often was it seen, has it been seen in multiple trees. Also for
>> Âfinding duplicates which may not have been sent to the correct mailing list.
>
> Hi Eric,
>
> Good question. I would very much like to open the UI, and I hope to do
> it in near future, but we need to do some additional work to make it
> possible. The good news is that information is already accumulating
> and we can do pings, etc.

Hello Dmitry,

I think not useful to be a GUI, for example it can be console based ui we can conenct and get information and fixed patches.
So syzbot is perfectly, I founded a patc last time :)

https://09738734946362323617.googlegroups.com/attach/3c6ef7059f77c/patch.txt?part=0.2&view=1&vt=ANaJVrFm49WFVkkKiomlnsrdfnv4P-0znjiC4agFB72ibq9_6iqg1rmZtw9-DxS5VvoOoKx8Ikl88sYEQQ45X0vjrwFkKDRaZELV-oU9DVmmrRAMSfStn24

And, I have a my suggestions:

Please keep to short url addresses and I think syzbot use to .txt file attached.
.txt is not good.

Ozgur

>> Âsyzbot also should be sending out reminders for bugs that are still open if the
>> Âcrash is still occurring, and even moreso if there is a reproducer.
>
> Agree. The reasons why this hasn't happen yet are:
> 1. syzbot is being built up as it's running, I am overwhelmed with
> hundreds of bugs and also doing lots of work which may be not directly
> visible but important (e.g. improving quality of generated
> reproducers, increasing percent of cases when reproducers are created,
> improving bug title extraction logic, implementing patch testing by
> request, now this new Reported-by-based process, etc).
> 2. Just sending an email for each open bug every week is simple, but I
> afraid it won't be warmly welcomed. The open questions are: how
> frequently syzbot should ping? should repro/no repro affect this? what
> to do if it stopped happening? stopped happenning for how long? and
> what if it happened just few times, so we can't really conclude if it
> still happens or not (but we've seen very bad races manifesting this
> way)? how should it interact with the following point?
>
>> ÂHowever, if the crash isn't still occurring, then I expect it will become
>> Ânecessary to automatically invalidate the bug after some time, lest the list of
>> Âbugs grow without bound due to bugs that have already been fixed that no one has
>> Âtime to debug to figure out exactly when/what the fix was, especially if there
>> Âis no reproducer. Or perhaps the bug was only in linux-next and only existed
>> Âdue to a buggy patch which was dropped or modified before it reached mainline,
>> Âso there is no "fix" commit.
>
> Good point. I think we will need to do this in some form in future.
> Again open questions:
> Â- what is the precise formula behind "isn't still occurring"?
> Â- should we only close "no repro" bugs?
> Â- should we re-test bugs with repro? (re-testing is not 100% precise,
> so we will lose some real subtle bugs this way)
>
> Thanks