Re: bug-introducing patches (or: -rc cycles suck)

[Willy Tarreau]

Hi Sasha,

On Mon, Apr 30, 2018 at 05:58:30PM +0000, Sasha Levin wrote:
>  - For some reason, the odds of a -rc commit to be targetted for -stable is
>  over 20%, while for merge window commits it's about 3%. I can't quite
>  explain why that happens, but this would suggest that -rc commits end up
>  hurting -stable pretty badly.

Often, merge window collects work that has been done during the previous
cycle and which is prepared to target this merge window. Fixes that happen
during this period very likely tend to either be remerged with the patches
before they are submitted if they concern the code to be submitted, or are
delayed to after the work gets merged. As a result few of the pre-rc1 patches
get backported while the next ones mostly contain fixes. By the way, you
probably also noticed it when backporting patches to your stable releases,
the mainline commit almost never comes from a merge window.

> 2. Maintainers need to stop writing patches, commiting them, and pushing them
> in without reviews. In -rc cycles there is quite a large number of commits
> that were either written by maintainers, commited, and merged upstream the
> same day. These patches are very likely to introduce a new bug.

Developers are humans before anything else. We probably all address most
bug reports the same way : "ah, of course, stupid me, now that's fixed".
Keep in mind that for the developer, the pressure has lowered now that
the code got merged, and that mentally the fix is "on top" of the initial
work and no more part of it. It often means a narrower mental image of
how the fix fits in the whole code.

I think that you'll also notice that fixes that address bugs introduced
during the merge window of the same version will more often introduce
bugs than the ones which address 6-months old bugs which require some
deeper thinking. In short it indicates that we tend to believe we are
better than we really are, especially very late at night.

> I don't really have a proposal beyond "tighten up -rc cycles", but I think
> it's a discussion worth having. We have enough data to show what parts of
> kernel development work, and what parts are just hurting us.

I'm inclined to believe that making individuals aware of their own
mistakes can help. I personally like to try to understand how I managed
to introduce a bug, it's always useful. Very often it's around "I was
pretty sure it didn't require testing, the change was so obvious". We
all know this feeling when you write 100 lines in a new file, you
compile, and it builds without any warning and apparently works, and
suddenly you think "uh oh, what did I do wrong?" and you have no idea
where to start to look for possible mistakes.

Probably that some statistics on mistake classifications and maybe some
affected subsystems (if that doesn't blame anyone) could be useful.

Just my two cents,
Willy