Re: Linux 2.6.21

From: Adrian Bunk
Date: Thu Apr 26 2007 - 13:11:42 EST

On Thu, Apr 26, 2007 at 08:47:26AM -0700, Linus Torvalds wrote:
> On Thu, 26 Apr 2007, Adrian Bunk wrote:
> >
> > There is a conflict between Linus trying to release kernels every
> > 2 months and releasing with few regressions.
> No.
> Regressions _increase_ with longer release cycles. They don't get fewer.
> The fact is, we have a -stable series for a reason. The reason is that the
> normal development kernel can work in three ways:
> (a) long release cycles, with two subcases:
> (a1) huge changes (ie a "long development series". This is what we
> used to have. There's no way to even track the regressions,
> because things just change too much.
> (a2) keep the development limited, just stretch out the
> "stabilization phase". This simply *does*not*work*. You might
> want it to work, but it's against human psychology. People
> get bored, and start wasting their time discussing esoteric
> scheduler issues which weren't regressions at all.
> (b) Short and staggered release cycle: keep changes limited (like a2),
> but recognize when it gets counter-productive, and cut a release so
> that the stable team can continue with it, while most developers (who
> wouldn't have worked on the stable kernel _anyway_) don't get
> frustrated.


They get frustrated because they focussed on developing new features
instead of fixing regressions, and now it takes longer until their new
features get merged because noone fixed the regressions...


> And yes, we've gone for (b). With occasional "I'm not taking any half-way
> scary things at _all_" releases, like 2.6.20 was.
> > Trying to avoid regressions might in the worst case result in an -rc12
> > and 4 months between releases. If the focus is on avoiding regressions
> > this has to be accepted.
> No. You are ignoring the reality of development. The reality is that you
> have to balance things. If you have a four-month release cycle, where

I'm not saying it always have to be 4 months.

> three and a half months are just "wait for reports to trickle in from
> testers", you simply won't get _anything_ done. People will throw their
> hands up in frustration and go somewhere else.

"wait for reports to trickle in from testers" is exactly the opposite of
our problem.

I started the regression lists originally to prove the fairy tale
"noone tests -rc kernels" some kernel developers spread as wrong.

Look at the facts:
8 out of 14 regressions in my current list were reported in March or earlier.
And for many regressions fixed it took several weeks until debugging
by a kernel developer was started.

We do not lack testers for getting bug reports quickly.
We lack developer manpower for debugging the many regression reports.

> > 0 regressions is never realistic (especially since many regressions
> > might not be reported during -rc), but IMHO we could do much better than
> > what happened in 2.6.20 and 2.6.21.
> 2.6.20 was actually really good. Yes, it had some regressions, but I do
> believe that it was one of the least buggy releases we've had. The process
> _worked_.

In the country of the blind the one-eyed man is king...

> 2.6.21 was much less pleasant, but the timer thing really was
> > I'm not satisfied with the result, and the world won't stop turning when
> > I'm not tracking 2.6.22-rc regressions.
> True. However, it's sad that you feel like you can't bother to track them.
> They were _very_ useful. The fact that you felt they weren't is just
> becasue I think you had unrealistic expectations, and you think that the
> stable people shouldn't have to have anything to do.
> You're maintaining 2.6.16 yourself - do you not see what happens when you
> decide that "zero regressions" is the target? You have to stop
> development. And while that may sound like a good thing at any particular
> time, it's a total *disaster* in the long run (not even very long,
> actually: in the two-to-three release cycle kind of run), because while
> you are in a "regression fix" mode, people still go on developing, and
> you're just causing problems for the _next_ release by holding things up
> too long.
> That's the *real* reality: 5 to 7 _million_ lines of diffs in a release
> every two to three months. Do you really think those changes stop just
> because of a release process? No. If you drag out the releases to be 4+
> months, you'll just have 10-15 million lines of changes instead (or, more
> likely, you'll have developers who can't be bothered any more, and you may
> have just 2 million lines, and three years later you have a kernel that
> isn't relevant any more. Look at any of the other Unixes).

There's not a realistic chance for 0 regressions, and 4 months was
a worst case, not the average case.

But I am not happy with the current state of released kernels.

> In other words, there's a _reason_ we have staggered development. We have
> the "crazy development trees" (aka -mm and various other trees), we have
> the "development tree" (aka Linus' tree), and we have the -stable tree. If
> the stable tree has a dozen known issues that they'll have to sort out
> over the next two months, that's *fine*. That's kind of the point of the
> stable tree.

And all the people who have to upgrade to 2.6.21 for getting an
important security fix run into a dozen known (and many unknown)

I don't think that's fine.

> And you would helpe them with the 2.6.22-stable releases if you'd maintain
> that list. Even if it is _designed_ not to go down to zero.
> I suspect that you got overly optimistic from the fact that 2.6.20 really
> _was_ an easy release. It was designed that way. You feel that it was bad
> or average, but that's actually because of _your_ unrealistic
> expectations, not becasue there was anything wrong with 2.6.20.

If we had the developer manpower to get each reported regression
debugged and fixed [1] within three weeks, 2.6.21 might be in the shape
I would have liked it to be today.

But there are the three interdependent variables time, developer
manpower and quality. And few developer manpower and few time results in
a lower quality of the release I'm not happy with.

Life has taught me that sometimes I'm right, sometimes I'm wrong, and
sometimes both sides have a possible solution. We might agree to
disagree, and you are the one who's opinion counts. I can only say that
I am not happy with the result, and that I do therefore not spend my
time on maintaining regression lists for 2.6.22 - and maintaining such
lists is not something special noone else could do equally well.

> Linus


[1] "fixed" can also be e.g. "patch reverted" or "not a bug"


"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at