Re: Drive performance bottleneck

From: Lincoln Dale
Date: Fri Feb 04 2005 - 18:10:21 EST


At 08:32 PM 4/02/2005, Andrew Morton wrote:
Something funny is happening here - it looks like there's plenty of CPU
capacity left over.
[..]
Could you monitor the CPU load during the various tests? If the `dd'
workload isn't pegging the CPU then it could be that there's something
wrong with the I/O submission patterns.

as an educated guess, i'd say that the workload is running out of memory bandwidth ..

lets say the RAM is single-channel DDR400. peak bandwidth = 3.2Gb/s (400 x 10^6 x 64 bits / 1000000000). its fair to say that peak bandwidth is pretty rare thing to achieve with SDRAM given real-world access patterns -- lets take a conservative "it'll be 50% efficient" -- so DDR400 realistic peak = 1.6Gbps.

as far as memory-accesses go, a standard user-space read() from disk results in 4 memory-accesses (1. DMA from HBA to RAM, 2. read in copy_to_user(), 3. write in copy_to_user(), 4. userspace accessing that data).
1.6Gbps / 4 = 400MB/s -- or roughly what Ian was seeing.

sg_dd uses a window into a kernel DMA window. as such, two of the four memory acccesses are cut out (1. DMA from HBA to RAM, 2. userspace accessing data).
1.6Gbps / 2 = 800MB/s -- or roughly what Ian was seeing with sg_dd.

DIRECT_IO should achieve similar numbers to sg_dd, but perhaps not quite as efficient.


cheers,

lincoln.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/