Re: Where is the performance bottleneck?
From: Holger Kiehl
Date: Wed Aug 31 2005 - 11:52:34 EST
On Wed, 31 Aug 2005, Jens Axboe wrote:
On Wed, Aug 31 2005, Holger Kiehl wrote:
On Wed, 31 Aug 2005, Jens Axboe wrote:
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us sy id
wa^M
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
12 66
1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
Using kernel 2.6.12.5 again, here the results:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 8009648 4764 40592 0 0 0 0 1011 32 0 0 100 0
0 0 0 8009648 4764 40592 0 0 0 0 1011 34 0 0 100 0
0 0 0 8009648 4764 40592 0 0 0 0 1008 61 0 0 100 0
0 0 0 8009648 4764 40592 0 0 0 0 1006 26 0 0 100 0
0 8 0 8006372 4764 40592 0 0 120192 0 1944 1929 0 1 89 10
2 8 0 8006372 4764 40592 0 0 319488 0 3502 4999 0 2 75 24
0 8 0 8006372 4764 40592 0 0 319488 0 3506 4995 0 2 75 24
0 8 0 8006372 4764 40592 0 0 319744 0 3504 4999 0 1 75 24
0 8 0 8006372 4764 40592 0 0 319488 0 3507 5009 0 2 75 23
0 8 0 8006372 4764 40592 0 0 319616 0 3506 5011 0 2 75 24
0 8 0 8005124 4800 41100 0 0 319976 0 3536 4995 0 2 73 25
0 8 0 8005124 4800 41100 0 0 323584 0 3534 5000 0 2 75 23
0 8 0 8005124 4800 41100 0 0 323968 0 3540 5035 0 1 75 24
0 8 0 8005124 4800 41100 0 0 319232 0 3506 4811 0 1 75 24
0 8 0 8005504 4800 41100 0 0 317952 0 3498 4747 0 1 75 24
0 8 0 8005504 4800 41100 0 0 318720 0 3495 4672 0 2 75 23
1 8 0 8005504 4800 41100 0 0 318720 0 3509 4707 0 1 75 24
0 8 0 8005504 4800 41100 0 0 318720 0 3499 4667 0 2 75 23
0 8 0 8005504 4808 41092 0 0 318848 40 3509 4674 0 1 75 24
0 8 0 8005380 4808 41092 0 0 318848 0 3497 4693 0 2 72 26
0 8 0 8005380 4808 41092 0 0 318592 0 3500 4646 0 2 75 23
0 8 0 8005380 4808 41092 0 0 318592 0 3495 4828 0 2 61 37
0 8 0 8005380 4808 41092 0 0 318848 0 3499 4827 0 1 62 37
1 8 0 8005380 4808 41092 0 0 318464 0 3495 4642 0 2 75 23
0 8 0 8005380 4816 41084 0 0 318848 32 3511 4672 0 1 75 24
0 8 0 8005380 4816 41084 0 0 320640 0 3512 4877 0 2 75 23
0 8 0 8005380 4816 41084 0 0 322944 0 3533 5047 0 2 75 24
0 8 0 8005380 4816 41084 0 0 322816 0 3531 5053 0 1 75 24
0 8 0 8005380 4816 41084 0 0 322944 0 3531 5048 0 2 75 23
0 8 0 8005380 4816 41084 0 0 322944 0 3529 5043 0 1 75 24
0 0 0 8008360 4816 41084 0 0 266880 0 3112 4224 0 2 78 20
0 0 0 8008360 4816 41084 0 0 0 0 1012 28 0 0 100 0
Holger
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/