>
> HOWEVER, despite the fact that I don't think it matters in real life, I'm
> all for people doing benchmarks, and crying out when one or the other is
> slower in something. Not because I think it makes much of a difference for
> normal users, but because it's good for development. It's a great way to
> motivate people to do better (show that the competition can do better at
> something, and you force us to try to improve ourselves).
>
That is similar to my position on competition. I do not think that a
"single" OS, compiler, etc. should be predominant in the free domain. It
makes the software fat & lazy.
>
> Indeed. Even on the same machine the placement of the partitions can make
> a noticeable difference for disk tests, so benchmarking is not a trivial
> thing.. It might still be interesting to see some kind of benchmarking
> done, just for "some data" as opposed to "THE data".
>
> If somebody wants to do benchmarking,I'd suggest using at least
> - lmbench (nice microbenchmark)
> - bonnie (reasonable disk performance benchmark)
> - webstone (or something similar. But use "apache" as the server, not
> some braindead horror like NCSA).
> - ???
>
Above list is ok, but CERTAINLY not sufficient.
> (the three mentioned should cover different areas, all very reasonable,
> but have I missed some important area?)
>
Above list has no testing under VM load. That is where memory scheduling
policies start taking effect. There are other forms of loading also, like
tty subsystems (some people still use those.) Another, is how well do big disk
farms work (lots of sustained concurrent I/O)? How does the system work
with many many TCP connections (little benchmarks like lmbench are interesting
but don't show performance in an ISP or large workstation situation)?
John