RE: Direct io on block device has performance regression on 2.6.x kernel

From: Chen, Kenneth W
Date: Wed Mar 09 2005 - 22:55:02 EST

Andrew Morton wrote Wednesday, March 09, 2005 6:26 PM
> What does "1/3 of the total benchmark performance regression" mean? One
> third of 0.1% isn't very impressive. You haven't told us anything at all
> about the magnitude of this regression.

2.6.9 kernel is 6% slower compare to distributor's 2.4 kernel (RHEL3). Roughly
2% came from storage driver (I'm not allowed to say anything beyond that, there
is a fix though).

2% came from DIO.

The rest of 2% is still unaccounted for. We don't know where.

> How much system time? User time? All that stuff.
20.5% in the kernel, 79.5% in user space.

> But the first thing to do is to work out where the cycles are going to.
You've seen the profile. That's where all the cycle went.

> Also, I'm rather peeved that we're hearing about this regression now rather
> than two years ago. And mystified as to why yours is the only group which
> has reported it.

2.6.X kernel has never been faster than the 2.4 kernel (RHEL3). At one point
of time, around 2.6.2, the gap is pretty close, at around 1%, but still slower.
Around 2.6.5, we found global plug list is causing huge lock contention on
32-way numa box. That got fixed in 2.6.7. Then comes 2.6.8 which took a big
dip at close to 20% regression. Then we fixed 17% regression in the scheduler
(fixed with cache_decay_tick). 2.6.9 is the last one we measured and it is 6%
slower. It's a constant moving target, a wild goose to chase.

I don't know why other people have not reported the problem, perhaps they
haven't got a chance to run transaction processing db workload on 2.6 kernel.
Perhaps they have not compared, perhaps they are working on the same problem.
I just don't know.

- Ken

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at