Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results

From: Ryan Anderson (
Date: Mon Sep 23 2002 - 09:02:41 EST

On Mon, Sep 23, 2002 at 08:30:21PM +1000, Con Kolivas wrote:
> Quoting Ingo Molnar <>:
> > On Mon, 23 Sep 2002, Con Kolivas wrote:
> >
> > how many times are you running each test? You should run them at least
> > twice (ideally 3 times at least), to establish some sort of statistical
> > noise measure. Especially IO benchmarks tend to fluctuate very heavily
> > depending on various things - they are also very dependent on the initial
> > state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful
> > measurement result would be something like:
> Yes you make a very valid point and something I've been stewing over privately
> for some time. contest runs benchmarks in a fixed order with a "priming" compile
> to try and get pagecaches etc back to some sort of baseline (I've been trying
> hard to make the results accurate and repeatable).

Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.

Or you could run the contest 3 times, rebooting between each run....
(automating that is a little harder, of course.)

IANAS, however.


Ryan Anderson sometimes Pug Majere - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to More majordomo info at Please read the FAQ at

This archive was generated by hypermail 2b29 : Mon Sep 23 2002 - 22:00:38 EST