Re: Regression testing?

Chuck Lever (cel@monkey.org)
Thu, 21 Jan 1999 16:13:19 -0500 (EST)


dan-

regression testing is a great idea. i've already ported SPEC's SDM to
Linux (yes, a few minor fixes were required to get it to work properly on
Linux; i can post them here for anyone who is interested), and have a copy
of SPECweb96.

SPECweb96 is pretty hard to use. i would think using something like
httperf might be easier and more relevant. also realize that the web
server's architecture (threading and I/O model) make a big difference, so
the web server's version would have to be fixed across all tests to make
comparisons valid.

testing 2.2.0-preX is a good idea, but there is no performance history.
someone trying this would need to benchmark a few of the popular kernel
versions that are already out there (2.1.132 and 2.0.36 come to mind) in
order to gather really useful information. and somehow we'd need to
derive a reasonable "common" hardware configuration.

SDM is an excellent validation test, i've found, because of its careful
checking of script output. SDM can drive repeatable offered loads against
specific filesystems, too, like NFS, arla, reiserfs, or ext2fs, and will
easily expose small misbehaviors.

i've been able to push a dual 200Mhz PPro pretty hard with it (64 scripts,
after increasing NR_TASKS and rebuilding the kernel). however, SDM uses a
bunch of programs and utilities, so it should be considered to be a stress
of a particular distribution, rather than stressing just the kernel
itself.

On Wed, 20 Jan 1999, Dan Kegel wrote:

> Date: Wed, 20 Jan 1999 08:41:54 -0800
> From: Dan Kegel <dank@alumni.caltech.edu>
> To: linux-kernel@vger.rutgers.edu
> Subject: Regression testing?
>
> Does anyone have access to some hefty benchmarks
> suites like SPEC's, or regression tests
> like the POSIX compliance test?
> It might be a good idea to run 2.2.0-pre-x
> through these to look for suprises
> before going final with it.
>
> The Posix test suite is downloadable from
> http://www.itl.nist.gov/div897/ctg/posix_form.htm
> See also http://www.standards.ieee.org/regauth/posix/
> Even if we didn't actually pass all the tests,
> it'd be good to know that we didn't oops.
>
> The SPEC benchmarks can be purchased at
> http://www.spec.org/cgi-bin/osgorder
> for $700 or so. The hard part is
> actually having the patience to run them.
> (I suspect if Linux or Alan asked nicely, they'd
> donate a benchmark or two. Or we could take
> up a collection. Or Alan's boss could pay.)
> Three of the SPEC tests look interesting:
>
> SPECweb96 stresses out HTTP networking.
> SPEC SFS97 stresses out NFS networking.
> SPEC SDM (System Development: Multitasking)
> stresses the kernel by simulating a large number of
> users running make, cp, diff, etc.
>
> The point of running these benchmarks is not really
> to say we're fast, but to see whether we can
> actually get all the way through the benchmark
> (which might take a day to run) without having
> a functional error or a crash.
> - Dan
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.rutgers.edu
> Please read the FAQ at http://www.tux.org/lkml/
>

- Chuck Lever

--
corporate:	<chuckl@netscape.com>
personal:	<chucklever@netscape.net> or <cel@monkey.org>

- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/