Re: Benchmark comparisons - a summary of the last year

David C Niemi (niemi@wauug.erols.com)
Thu, 30 May 1996 09:53:53 -0400 (EDT)

On Thu, 30 May 1996, Carlo E. Prelz wrote:
...
> Q: Where can I find the benchmark software?
> A: I am using the unix benchmark set that was put together by Byte
> magazine at the time it was not terminally microsoft-biased. You can
> find the software at
>
> ftp://tsx-11.mit.edu/pub/linux/sources/test-suites/benchmark.tar.Z
...
> Q: The disk IO results are a bit strange.
> A: I know. Some results are missing altogether for all releases up to
> 1.3.13: at that time I discovered a bug in the software that meant
> that the past results were rubbish.
>
> There was also a period where the IO results were varying between two
> platform levels. This behavior ended at release 1.3.53, when the
> performances grew to a higher level. Here is the graph for File Copy
> (30 seconds):
...

There are some additional problems with the original disk I/O benchmarks,
beyond the truly egregious flaw which you already fixed:
- No "partial credit" is given for partly read or written blocks.
This results in greater variance and "plateaus" like you saw.

- The sample size is just right to fit in buffer cache on some
machines and to not fit on others

- Sync() is not called very effectively between calls.

I attempted to address all these issues. I also did major revisious to
the scoring algorithm and rebaselined it at 10.0 for a SPARCstation 20/61
running Solaris 2.3 with a SPARC Storage Array (yes, many PCs running
Linux are still a lot faster than the baseline).

Anyway, if anyone is interested in my "fixed" version, please take a look
at:

ftp://wauug.erols.com/pub/bench/unixbench-4.0-DELTA.tgz

Niemi@wauug.erols.com 703-810-5538 Reston, Virginia, USA
------ Money talks, but it is wrong half of the time. -----