> One thing that seems to be a major problem to me would be the fact
> that unlike the cryptography people, we are talking about doing things
> that may down the machine, and even while operating properly may make
> the machine unusable while it is running. The crypto people have
> software which is sitting nicely in the background.
True on both counts, and I think that's what we have to live with to do
serious testing. I don't think this is really a problem - I think we can
find plenty of people in the community willing to dedicate a machine to
kernel testing, especially if it's largely automated.
I'm putting together a tree structure on my system for the test suite and
here's what I have in mind so far:
linux/ Yep, the same directory your source is in
test/
Documentation/
sources/ describe where to get any packages included
arch/ architecture-specific tests
bench/ benchmarks
lmbench/ for starters, maybe include BYTE?
drivers/ driver-specific tests
fs/ filesystem tests, possibly building loopback devs
kernel/ things that exercise the scheduler, syscalls, etc.
mm/
net/ network tests, possibly using loopback
posix/ PCTS - Posix Conformance Test Suite
results/ Place to put result logs
The idea is to untar this package inside your source tree, run "make
config" to set up your kernel as usual, but with a new section to choose
your testing parameters. Parameters include a test cycle time limit, an
aggresiveness factor for stress testing, whether or not to include
benchmarks or to submit results, etc. Build your kernel and reboot as
usual, then run "make test".
This then gathers (at least) the following information:
kernel version
linux/.config file to decide what components to test, etc.
version of major tools including bintools, gcc, etc.
OS distribution info
/proc/(meminfo|devices|cpuinfo|etc..)
And then starts a series of conformance, regression, stress tests, and
benchmarks, submitted as perl, C, or shell scripts. The output of these
individual tests is tabulated and optionally sent off to a remote system
for indexing.
I've started on this, but we're in the process of upgrading our shell
machine from an ancient and heavily customized Slackware to Redhat, so it
might be a while.
-- "Love the dolphins," she advised him. "Write by W.A.S.T.E.."