Re: [Ksummit-discuss] bug-introducing patches

[Fengguang Wu]

On Mon, May 14, 2018 at 10:48:03AM +0200, Boris Brezillon wrote:
> +Fengguang
>
> On Mon, 14 May 2018 10:40:10 +0200
> Geert Uytterhoeven <geert@xxxxxxxxxxxxxx> wrote:
>
> > Hi Boris,
> >
> > On Mon, May 14, 2018 at 10:34 AM, Boris Brezillon
> > <boris.brezillon@xxxxxxxxxxx> wrote:
> > > On Mon, 14 May 2018 10:29:04 +0200
> > > Geert Uytterhoeven <geert@xxxxxxxxxxxxxx> wrote:
> > >> On Mon, May 14, 2018 at 10:12 AM, Boris Brezillon
> > >> <boris.brezillon@xxxxxxxxxxx> wrote:
> > >> > On Mon, 14 May 2018 10:00:30 +0200
> > >> > Geert Uytterhoeven <geert@xxxxxxxxxxxxxx> wrote:
> > >> >> On Tue, May 1, 2018 at 10:00 PM, Sasha Levin
> > >> >> <Alexander.Levin@xxxxxxxxxxxxx> wrote:
> > >> >> > On Tue, May 01, 2018 at 03:44:50PM -0400, Theodore Y. Ts'o wrote:
> > >> >> >>On Tue, May 01, 2018 at 04:38:21PM +0000, Sasha Levin wrote:
> > >> >> > What's worse is that that commit is tagged for stable, which means
> > >> >> > that (given Greg's schedule) it may find it's way to -stable users
> > >> >> > even before some -next users/bots had a chance to test it out.
> > >> >>
> > >> >> I just noticed a case where a commit was picked up for stable, while a
> > >> >> bot had flagged it as a build regression 18 hours earlier (with a CC to
> > >> >> lkml).
> > >> >
> > >> > Also, this patch has been on a tree that I know is tested by Fengguang's
> > >> > robots for more than a week (and in linux-next for 2 days, which, I
> > >> > agree, is probably not enough), and still, I only received the bug
> > >> > report when the patch reached mainline. Are there tests that are only
> > >> > run on Linus' tree?
> > >>
> > >> Have your received a success report from Fengguang's bot, listing all
> > >> configs tested (the broken one should be included; it is included in the
> > >> configs tested on my branches)?
> > >
> > > Yes I did (see below).
> > >
> > > -->8--
> > > From: kbuild test robot <lkp@xxxxxxxxx>
> > > To: Boris Brezillon <boris.brezillon@xxxxxxxxxxx>
> > > Subject: [bbrezillon-0day:mtd/fixes] BUILD SUCCESS  fc3a9e15b492eef707afd56b7478001fdecfe53f
> > > Date: Mon, 07 May 2018 20:05:52 +0800
> > > User-Agent: Heirloom mailx 12.5 6/20/10
> > >
> > > tree/branch: https://github.com/bbrezillon/linux-0day  mtd/fixes
> > > branch HEAD: fc3a9e15b492eef707afd56b7478001fdecfe53f  mtd: rawnand: Make sure we wait tWB before polling the STATUS reg
> > >
> > > elapsed time: 49m
> > >
> > > configs tested: 142
> >
> > But the failed config (m68k/allmodconfig) is not listed?
>
> Yes, that's my point. It seems that some configs are only rarely
> (never?) tested on my linux-0day tree (probably because they take longer
> to build), and I should only take kbuild robot results as an indication
> not a guarantee.

Yeah sorry, there is no 100% guarantee. There are 2 main aspects to
this problem.

- Response time vs coverage. Most build errors can be caught within 1
 day. The build success notification email is typically sent within
 half day (a reasonable feedback time). At this time, it can only be
 a rough indication not a guarantee. After sending the 0day build
 success notification, the build tests will actually continue for
 about 1 week to increase test coverage. 

- Merge-test-bisect based workflow. If one branch is hard to merge
 with others, especially if it's based on old kernel, it'll receive
 much less test coverage. Branches with known build/boot errors will
 be excluded from further merges, too.

Thanks,
Fengguang