> > That works, but adding a mechanized regression test would
> > be painless, and might give us earlier warnings if and when
> > a few bad things (like ext2 fs corruption) creep back into
>
> Automatic regression tests very rarely help. Most of the bugs that
> get into a shipping kernel now are ones I can't reproduce even given a
> description let alone find randomly
>
> > corporations love. Red Hat, for instance, might well feel
> > inclined to set up a mechanized regression test to give
> > it a little added certainty that it's not about to ship a lemon.
>
> Guess which turns up lemons best, the automated testing or the beta program.
The usual experience (not necessarily with Linux) is these two types of testing
produce sets of bug reports which only partially overlap. Each seems to be
better at flushing out certain types of problem. Only an idiot organisation
would place full reliance on programmed regression testing (like a lot of idiot
organisations actually do), but betas tend to miss problems in boring bits the
beta testers don't exercise well. I guess you might say programmed tests give
broader coverage, but betas give greater depth. The fact that regression tests
are only partially effectice shows the limited value of test coverage analysis,
which some people so love.
One thing programmed regression tests do well is to reduce reinventing the
wheel. A lot of test results are open to discussion, to decide what the
"correct" outcome is. If the final answer gets embedded in the test plan it
tends not to get wastefully re-discussed so often.
The key issue is not whether regression tests are useful , which they are, but
whether they are cost effective, which is questionable in the Linux case. Try
skipping them on a large distributed system, like a cellular switch, and see
what the effect is.
Steve
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/