Larry, thank you for your post. This is exactly the post I was looking
for to seed conversation on this subject. Thank you for your testimony
into the value of regression testing. Regression testing isn't a
panacea, but it can have great value if leveraged correctly.
In regard to your framework, this is very similar to some ideas we had
for our project. We (SGI) have a regression testing framework in use
internally that is very similar in basis to your framework. The major
difference is that it has been in use and development for years now.
Its quite grandiose and complete -- I think we might open source it if
LTP came to need it. For now, your framework is more along the line of
what we had in mind for LTP.
The exit status method is probably the simplest way of reporting test
results. It makes it relatively easy for anyone to write tests (take
note you lkml lurkers). Basically, writing a simple test goes something
like this:
Setup - get the proc environment ready to perform the test operation
Test - perform the test operation
Analysis - determine the success/failure of the operation
Cleanup - get ready to go away, cleanup scratch space, make sure system
is in a clean state
Exit - report results via exit status 0-PASS,!0-FAIL
Larry, I may hit you up for your test harness. It may be a great place
for us to start. We already have 90+ tests, and managing them is
already a problem.
Again, anyone interested in helping us with our project is encouraged to
participate:
http://oss.sgi.com/projects/ltp/
--aaron
Larry McVoy wrote:
>
> I'm a huge fan of this idea. We have a very simple, but effective, test
> harness in BitKeeper. You are welcome to it if you want it. All we do is
> have directory full of "t.files" which are shell scripts. There is a
> setup program called "doit" which clears out /tmp/.regression-$USER,
> sets up the environment, goes to that directory, and then runs the
> script. If the script exits 0, it does the next one, else it stops.
>
> We've got 12,000 lines of shell scripts we use to test BitKeeper and it
> has been priceless. In theory, we don't fix a bug without adding a
> regression, in practice that is true about 80% of the time. We're quite
> happy that it is that high. The reason it is that high is that once the
> harness is there, you can create a new test like so:
>
> cat > t/t.small
> echo $N Try wrapping and unwrapping some data .......................$NL
> ls / > data
> bk uuwrap < data > data.uu
> bk unuuwrap < data.uu | cmp -s - data
> if [ $? != 0 ]; then echo Data is different when unwrapped; exit 1; fi
> echo OK
> ^D
>
> I strongly urge you to do the work to create a similar simple harness.
> If it is trivial for people to create tests, they _might_ create them.
> If it is anything other than trivial, it's just another good idea that
> won't happen.
>
> If I get to Linux World I'll look you up, otherwise you can ping me
> for the test harness.
>
> On Tue, Aug 08, 2000 at 03:06:28PM -0500, Nathan Straz wrote:
> > SGI would like to announce the Linux Test Project. The goal of this
> > project is to create a formalized test system for the Linux kernel.
> >
> > We have released a set of 96 tests on the project's website
> > (http://oss.sgi.com/projects/ltp/). These tests exercise file systems
> > and system calls and can be used for stress testing or sanity tests.
> >
> > We would like to discuss the following topics with the community.
> >
> > - The testing philosophy that is most important to the kernel
> > developers. What approach best fits the development process?
> > Regression? Functional? Stress? Performance?
> >
> > - What is needed immediately? Building a test suite for the kernel is
> > going to take time. What tests or tools are most important?
> >
> > - We need to plan a development road map that works with the Linux
> > kernel development road map.
> >
> > We are hoping to hold an unscheduled BOF at LinuxWorld on Wednesday.
> > Aaron Laffin and Richard Logan will be there to discuss these issues.
> > If you are interested in testing and are attending LinuxWorld, please
> > keep an eye open for our BOF.
> >
> > Linux Test Project http://oss.sgi.com/projects/ltp/
> > LTP Mailing list ltp@oss.sgi.com
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.rutgers.edu
> > Please read the FAQ at http://www.tux.org/lkml/
>
> --
> ---
> Larry McVoy lm@bitmover.com http://www.bitmover.com/lm
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.rutgers.edu
> Please read the FAQ at http://www.tux.org/lkml/
-- Aaron Laffin Silicon Graphics, Inc. OS Test Development Email: alaffin@sgi.com Voice: 651-683-5756 USA/MN/CRP/F5233/SSBU- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Tue Aug 15 2000 - 21:00:16 EST