Re: [RFC] syzbot process

[Ozgur]

28.12.2017, 14:45, "Dmitry Vyukov" <dvyukov@xxxxxxxxxx>:
> On Thu, Dec 28, 2017 at 11:51 AM, Ozgur <ozgur@xxxxxxxxxx> wrote:
>> Â28.12.2017, 13:41, "Dmitry Vyukov" <dvyukov@xxxxxxxxxx>:
>>> ÂOn Fri, Dec 22, 2017 at 4:32 AM, Eric Biggers <ebiggers3@xxxxxxxxx> wrote:
>>>> ÂÂOn Thu, Dec 21, 2017 at 01:52:40PM +0100, Dmitry Vyukov wrote:
>>>>> ÂÂHowever, the cost is that it needs to understand statuses of bugs:
>>>>> ÂÂmost importantly, what commit fixes what bug. It also has support for
>>>>> ÂÂmarking a bug as "invalid", e.g. happened once but most likely was
>>>>> ÂÂcaused by a previous silent memory corruption. And support for marking
>>>>> ÂÂbugs as duplicates of other bugs, i.e. the same root cause and will be
>>>>> ÂÂfixed when the target bug is fixed. These simple rules are outlined in
>>>>> ÂÂthe footer of each report and also explained in more detail at the
>>>>> ÂÂreferenced link:
>>>>>
>>>>> ÂÂ----------------------------------
>>>>> ÂÂThis bug is generated by a dumb bot. It may contain errors.
>>>>> ÂÂSee https://goo.gl/tpsmEJ for details.
>>>>> ÂÂDirect all questions to syzkaller@xxxxxxxxxxxxxxxxx
>>>>> ÂÂPlease credit me with: Reported-by: syzbot <syzkaller@xxxxxxxxxxxxxxxx>
>>>>> ÂÂsyzbot will keep track of this bug report.
>>>>> ÂÂOnce a fix for this bug is merged into any tree, reply to this email with:
>>>>> ÂÂ#syz fix: exact-commit-title
>>>>> ÂÂIf you want to test a patch for this bug, please reply with:
>>>>> ÂÂ#syz test: git://repo/address.git branch
>>>>> ÂÂand provide the patch inline or as an attachment.
>>>>> ÂÂTo mark this as a duplicate of another syzbot report, please reply with:
>>>>> ÂÂ#syz dup: exact-subject-of-another-report
>>>>> ÂÂIf it's a one-off invalid bug report, please reply with:
>>>>> ÂÂ#syz invalid
>>>>> ÂÂNote: if the crash happens again, it will cause creation of a new bug report.
>>>>> ÂÂNote: all commands must start from beginning of the line in the email body.
>>>>> ÂÂ----------------------------------
>>>>>
>>>>> ÂÂStatus tracking allows syzbot to (1) keep track of still unfixed bugs
>>>>> ÂÂ(more than half actually gets lost in LKML archives if nobody keeps
>>>>> ÂÂtrack of them), (2) be able to ever report similarly looking crashes
>>>>> ÂÂas new bugs in future, (3) be able to test fixes.
>>>>>
>>>>> ÂÂThe problem is that these rules are mostly not followed.
>>>>
>>>> ÂÂAs others mentioned, allowing a bug ID to be in the fix's commit message,
>>>> ÂÂperhaps in the Reported-by line which syzbot already suggests to include, would
>>>> ÂÂmake things a bit easier.
>>>>
>>>> ÂÂBut I think the larger problem is that people in the community don't have any
>>>> ÂÂvisibility into the statuses of the bugs, so they don't have any motivation to
>>>> ÂÂmanage the statuses.
>>>>
>>>> ÂÂAre you planning to make a dashboard app publicly available for upstream kernel
>>>> ÂÂbugs being tracked by syzbot? I think it would be very useful for the
>>>> ÂÂcommunity, especially for finding more details about a bug, e.g. when was it
>>>> ÂÂlast seen, how often was it seen, has it been seen in multiple trees. Also for
>>>> ÂÂfinding duplicates which may not have been sent to the correct mailing list.
>>>
>>> ÂHi Eric,
>>>
>>> ÂGood question. I would very much like to open the UI, and I hope to do
>>> Âit in near future, but we need to do some additional work to make it
>>> Âpossible. The good news is that information is already accumulating
>>> Âand we can do pings, etc.
>>
>> ÂHello Dmitry,
>>
>> ÂI think not useful to be a GUI, for example it can be console based ui we can conenct and get information and fixed patches.
>
> Hi Ozgur,

Hello,

> We will do web UI first as it's something that's already partially
> there and syzbot itself is not a console process, it's a cloud
> service. It's also handy because there are lots of contextual
> information and in a web UI one can just just click links to navigate
> or download a blob. Later we could do an API for console clients, etc
> if there is an interest in developing these types of UIs. But
> generally UI is not the main business of syzbot, it's only a side
> thing that helps it achieve the main goal, so it's doesn't have a team
> of people assigned to it. But you are welcome to contribute, it's all
> open-source:
> https://github.com/google/syzkaller/tree/master/dashboard/app

I understand.

>> ÂSo syzbot is perfectly, I founded a patc last time :)
>>
>> Âhttps://09738734946362323617.googlegroups.com/attach/3c6ef7059f77c/patch.txt?part=0.2&view=1&vt=ANaJVrFm49WFVkkKiomlnsrdfnv4P-0znjiC4agFB72ibq9_6iqg1rmZtw9-DxS5VvoOoKx8Ikl88sYEQQ45X0vjrwFkKDRaZELV-oU9DVmmrRAMSfStn24
>>
>> ÂAnd, I have a my suggestions:
>>
>> ÂPlease keep to short url addresses.
>
> Well, that's an URL generated by google groups, we don't have control
> over it. You also received the patch as an attachment in the syzbot
> email.

I know, understand. sure.

>> Âand I think syzbot use to .txt file attached.
>> Â.txt is not good.
>
> Why are not .txt attachments good? What do you propose to use?

I think I'm misunderstood that is good to have text output in a file but  not useful if the file extension is ".txt" 
Not comfortable use it for mutt / vim and diff.

I think needs to be an new extension, would be like this ".log" or ".syz" :)


> Thanks
>
>>>> ÂÂsyzbot also should be sending out reminders for bugs that are still open if the
>>>> ÂÂcrash is still occurring, and even moreso if there is a reproducer.
>>>
>>> ÂAgree. The reasons why this hasn't happen yet are:
>>> Â1. syzbot is being built up as it's running, I am overwhelmed with
>>> Âhundreds of bugs and also doing lots of work which may be not directly
>>> Âvisible but important (e.g. improving quality of generated
>>> Âreproducers, increasing percent of cases when reproducers are created,
>>> Âimproving bug title extraction logic, implementing patch testing by
>>> Ârequest, now this new Reported-by-based process, etc).
>>> Â2. Just sending an email for each open bug every week is simple, but I
>>> Âafraid it won't be warmly welcomed. The open questions are: how
>>> Âfrequently syzbot should ping? should repro/no repro affect this? what
>>> Âto do if it stopped happening? stopped happenning for how long? and
>>> Âwhat if it happened just few times, so we can't really conclude if it
>>> Âstill happens or not (but we've seen very bad races manifesting this
>>> Âway)? how should it interact with the following point?
>>>
>>>> ÂÂHowever, if the crash isn't still occurring, then I expect it will become
>>>> ÂÂnecessary to automatically invalidate the bug after some time, lest the list of
>>>> ÂÂbugs grow without bound due to bugs that have already been fixed that no one has
>>>> ÂÂtime to debug to figure out exactly when/what the fix was, especially if there
>>>> ÂÂis no reproducer. Or perhaps the bug was only in linux-next and only existed
>>>> ÂÂdue to a buggy patch which was dropped or modified before it reached mainline,
>>>> ÂÂso there is no "fix" commit.
>>>
>>> ÂGood point. I think we will need to do this in some form in future.
>>> ÂAgain open questions:
>>> ÂÂ- what is the precise formula behind "isn't still occurring"?
>>> ÂÂ- should we only close "no repro" bugs?
>>> ÂÂ- should we re-test bugs with repro? (re-testing is not 100% precise,
>>> Âso we will lose some real subtle bugs this way)
>>>
>>> ÂThanks