Re: page fault scalability patch V12 [0/7]: Overview andperformance tests

[cliff white]

On Wed, 01 Dec 2004 23:26:59 -0800
"Martin J. Bligh" <mbligh@xxxxxxxxxxx> wrote:

> --Andrew Morton <akpm@xxxxxxxx> wrote (on Wednesday, December 01, 2004 23:02:17 -0800):
> 
> > Jeff Garzik <jgarzik@xxxxxxxxx> wrote:
> >> 
> >> Andrew Morton wrote:
> >> > We need to be be achieving higher-quality major releases than we did in
> >> > 2.6.8 and 2.6.9.  Really the only tool we have to ensure this is longer
> >> > stabilisation periods.
> >> 
> >> 
> >> I'm still hoping that distros (like my employer) and orgs like OSDL will 
> >> step up, and hook 2.6.x BK snapshots into daily test harnesses.
> > 
> > I believe that both IBM and OSDL are doing this, or are getting geared up
> > to do this.  With both Linus bk and -mm.
> 
> I already run a bunch of tests on a variety of machines for every new 
> kernel ... but don't have an automated way to compare the results as yet, 
> so don't actually look at them much ;-(. Sometime soon (quite possibly over 
> Christmas) things will calm down enough I'll get a couple of days to write 
> the appropriate perl script, and start publishing stuff.

We've had the most success when one person has an itch to scratch, and works
with us to scratch it. We (OSDL) worked with Sebastien at Bull, and we're very 
glad he had the time to do such excellent work. We worked with Con Kolivas, likewise.

We've done tools to automate LTP comparisons ( bryce@xxxxxxxx  has posted results )
and reaim, we've been able to post some regression to lkml, and tied in with developers
to get bugs fixed. But OSDL has been limited by manpower.
 
One of the issues with the performance tests is the amount of data produced - 
 for example, the deep IO tests produce ton's o'  numbers, but the developer community wants
a single "+/- 5%" type response-  we need some opinions and help on how to do the data reduction 
necessary. 

What would be really kewl is some test/analysis code that could be re-used, so the Martin's of the future
have a good starting place. 
cliffw
OSDL




> 
> > However I have my doubts about how useful it will end up being.  These test
> > suites don't seem to pick up many regressions.  I've challenged Gerrit to
> > go back through a release cycle's bugfixes and work out how many of those
> > bugs would have been detected by the test suite.
> > 
> > My suspicion is that the answer will be "a very small proportion", and that
> > really is the bottom line.
> 
> Yeah, probably. Though the stress tests catch a lot more than the 
> functionality ones. The big pain in the ass is drivers, because I don't
> have a hope in hell of testing more than 1% of them.
> 
> M.
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@xxxxxxxxxx  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"aart@xxxxxxxxx";> aart@xxxxxxxxx </a>
> 


-- 
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/