Re: Benchmarks - 1.3.11

Darin Johnson (djohnson@cs.ucsd.edu)
Mon, 24 Jul 1995 09:52:31 -0700


noone@nowhere.com writes:
> Its all fine and well to say
> that task switching times or something has increased + or - 0.82% over
> the previous version, however when there haven't been any changes that
> could make a difference in such a thing, then you have to figre that
> you are actually detecting something else.

Probably just detecting the fact that it's on an asynchronous
interrupt driven system with a relatively large clock increment. For
instance, things hiccup when bdflush runs, or it takes slightly longer
for a disk to rotate, or the outfile files are in a different
location, and so forth. For any user mode code, slight changes can
effect how things page or fit in the cache, but that probably doesn't
apply too much if the test code isn't changing. Also, turn off as
many background processes as possible (cron, bdflush).

Maybe simpler than calculating the variance, why not just track the
smallest and largest results for each test, and see if these change
from version to version? How about a "realistic" test, that is,
something similar to what really gets run a lot, for instance,
compiling the kernel with -pipe and other things; the variance would
be high, but it would be interesting to see (and improvements to
things like paging might show up dramatically).