On Mon, Dec 07, 2009 at 03:39:49PM +0100, Christian Ehrhardt wrote:As far as i could clarify it our controllers don't support such a opportunistic merging.
[...]
I don't know what controller is in use there but does it
opportunistically merge requests if they are on physically contiguous
pages? If so, can it be disabled?
In the meanwhile I was able to verify that this also applies to random reads which are still reads but have zero read ahead requests.The regression appears as up to 76% loss in throughput at 16 processes (processes are scaled from 1 to 64, performance is bad everywhere - 16 is just the peak - avg loss is about 40% throughput).
I already know that giving the system just a bit (~64m+) more memory solves the issue almost completely, probably because there is almost no "memory pressure" left in that cases.
I also know that using direct-I/O instead of I/O through page cache doesn't have the problem at all.
This makes sense because it's a sequentual read load, so readahead is a
factor and that is why __GFP_COLD is used - the data is not for
immediate use so doesn't need to be cache hot.
I thought that too, but now comes the funny part.Comparing sysstat data taken while running with the kernels just with & without the bisected patch shows nothing obvious except that I/O seems to take much longer (lower interrupt ratio etc).
Maybe the controller is spending an age trying to merge requests because
it should be able to but takes a long time figuring it out?
With the effective throughput dropping by such a large amount while hardware latency improves by 30% I agree and suggest the issue is in the driver.The patch alone looks very reasonable, so I'd prefer understanding and fixing the real issue instead of getting it eventually reverted due to this regression being larger than the one it was intended to fix.
In the patch it is clear that hot pages (cold==0) freed in rmqueue_bulk should behave like before. So maybe the question is "are our pages cold while they shouldn't be"?
Well I don't really have a clue yet to explain how patch e084b exactly causes that big regression, ideas welcome :-)
Only theory I have at the moment is that the controller notices it can
merge requests and either spends a long time figuring out how to do the
merging or performs worse with merged IO requests.
If the problem is in the driver, oprofile might show where the problem lies