2.6.5 io scheduler problem? (cfq/deadline/noop)

From: Antonio Larrosa Jiménez
Date: Wed Jul 13 2005 - 09:53:15 EST

Hello,

I have a problem when reading and writing at the same time to an array of scsi
hard disks (an HP Smart Array 30) connected to an ADAPTEC ASR-2230S RAID5
controller using the aacraid module.

I'm using a SLES 9 system (kernel 2.6.5) which enables the cfq scheduler by
default (I checked that it's really using cfq).

I really don't understand why, but when running a bash script that contains:

#!/bin/sh
sync
time dd if=test.read of=/dev/null bs=32k count=32k &
time dd if=/dev/zero of=test.write bs=32k count=32k

(test.read is a file generated just like test.write is generated)

I got this when running vmstat 1 :

0 0 12 9512 5820 916920 0 0 0 0 1021 169 0 0 100 0
0 1 12 9448 5820 917180 0 0 108 0 1043 176 0 0 100 0
1 3 12 9508 5800 912780 0 0 8828 122504 1587 1002 0 25 48 27
0 4 12 8464 5808 913032 0 0 128 51940 1212 323 0 4 48 49
0 4 12 10276 5820 910940 0 0 0 30892 1278 297 0 2 50 49
0 4 12 9828 5844 910656 0 0 0 40600 1232 317 0 2 50 48
0 4 12 9764 5812 910168 0 0 0 45244 1192 247 0 3 35 62
0 4 12 10788 5840 908580 0 0 0 45468 1213 267 0 2 45 52
0 4 12 12196 5440 907160 0 0 0 37160 1216 281 0 2 50 48
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 3 12 9204 5472 909728 0 0 0 38580 1217 275 0 2 48 50
0 4 12 10356 5500 907880 0 0 0 46748 1197 306 0 2 50 48
0 4 12 15580 5504 902156 0 0 0 42028 1286 330 0 3 50 48
0 4 12 8692 5544 908356 0 0 0 43048 1203 234 0 3 49 49
0 4 12 12348 5552 904448 0 0 32 38692 1216 305 0 2 50 48
0 4 12 9852 5592 907008 0 0 0 49200 1230 292 0 3 50 48
0 2 12 14052 5700 902480 0 0 0 31060 1288 340 0 1 63 37
0 2 12 14052 5700 902480 0 0 0 42408 1220 280 0 0 60 40
0 2 12 13912 5700 902740 0 0 128 43116 1242 262 0 0 63 36
0 2 12 13764 5700 902740 0 0 128 41832 1311 362 0 0 56 44
0 2 12 13764 5700 902740 0 0 0 41832 1266 268 0 0 61 39
0 2 12 13764 5720 902720 0 0 0 42700 1260 287 0 0 60 40
0 1 12 13764 5720 902720 0 0 0 40164 1267 311 0 1 56 44
0 1 12 9412 5720 907140 0 0 4352 0 1156 200 0 0 75 25
0 1 12 8460 5708 908712 0 0 42904 0 1374 889 0 1 75 24
0 1 12 9612 5680 907960 0 0 38676 0 1360 794 0 2 74 24
0 1 12 10508 5680 907440 0 0 43032 20 1503 945 0 2 75 24
0 1 12 10376 5680 907960 0 0 42020 0 1348 870 0 1 75 24
0 1 12 10328 5664 908496 0 0 37912 0 1334 763 0 1 75 24
0 1 12 8908 5652 910328 0 0 38936 0 1328 832 0 1 75 24
0 1 12 8972 5636 910864 0 0 41884 0 1364 836 0 2 75 24
0 1 12 9740 5636 910344 0 0 41116 172 1385 823 0 1 75 24
0 1 12 8844 5632 911908 0 0 40736 0 1356 854 0 1 75 24
0 1 12 8716 5624 912436 0 0 36892 2048 1444 821 0 1 75 24
0 1 12 10252 5596 911424 0 0 42912 2180 1433 896 0 1 75 24
1 1 12 10764 5572 911448 0 0 40216 2160 1359 785 0 1 75 24
0 1 12 8588 5564 914056 0 0 41752 20 1367 850 0 1 75 24
0 1 12 9548 5528 913572 0 0 42908 128 1390 1136 0 1 75 24
0 1 12 8908 5480 914660 0 0 34192 256 1305 725 0 1 75 24
0 1 12 8780 5444 915476 0 0 41640 128 1348 879 0 1 75 24
0 1 12 9548 5464 915196 0 0 43044 256 1466 919 0 1 75 24
0 1 12 10468 5512 914888 0 0 38812 276 1331 817 0 2 75 24
0 1 12 8460 5524 917476 0 0 41620 256 1368 832 0 1 75 24
0 1 12 9800 5548 916412 0 0 38296 196 1355 808 0 1 75 24
0 1 12 166392 5488 759952 0 0 37020 41820 1476 863 0 3 73 24
0 1 12 159796 5488 766712 0 0 6912 0 1286 286 0 0 75 25
0 1 12 118092 5524 808536 0 0 41744 20 1368 838 0 1 75 24
0 1 12 75084 5552 851668 0 0 43036 0 1360 791 0 1 75 24
0 1 12 33868 5560 892740 0 0 41224 0 1365 786 0 1 75 24
0 1 12 9484 5592 916888 0 0 38816 0 1449 875 0 1 75 24
0 0 12 10216 5604 916616 0 0 18124 0 1182 447 0 0 87 13
0 0 12 10216 5632 916588 0 0 0 28 1050 185 0 0 100 0
0 0 12 10240 5632 916588 0 0 0 0 1066 155 0 0 100 0


Then I changed my /boot/grub/menu.lst to include elevator=deadline, but
curiously, after restarting, I got:

> dmesg | grep sch
Using cfq io scheduler

Which is quite odd, since I specified deadline in the kernel parameters.
Anyway, I did a test and I got:

0 0 0 11100 19460 944360 0 0 0 0 1023 46 0 0 100 0
0 0 0 11100 19460 944360 0 0 0 0 1018 38 0 0 100 0
0 0 0 11100 19460 944360 0 0 0 0 1019 42 0 0 100 0
0 2 0 10108 19584 944756 0 0 2532 172 1098 195 0 0 90 10
0 2 0 9020 19792 945588 0 0 27708 36 1277 584 0 1 61 38
0 2 0 10940 19968 943592 0 0 28464 0 1389 629 0 1 64 35
0 2 0 9596 20084 945036 0 0 26160 0 1352 602 0 1 62 37
0 2 8 18108 20224 936048 0 0 26016 0 1287 550 0 1 56 43
0 2 8 9148 20376 944996 0 0 27560 0 1269 557 0 1 59 40
0 2 8 9852 20556 944036 0 0 26404 16 1261 558 0 1 61 38
0 2 8 9276 20732 944380 0 0 27184 0 1400 608 0 1 58 42
1 3 8 10684 20612 938520 0 0 11932 108320 1501 973 0 27 42 30
0 4 8 11132 20552 937800 0 0 256 55604 1324 230 0 3 62 35
1 3 8 12924 20492 936040 0 0 128 41436 1189 145 0 2 57 40
0 4 8 9852 20460 939452 0 0 0 38772 1189 161 0 2 50 48
0 4 8 15108 20332 934380 0 0 256 47072 1190 169 0 3 50 48
0 4 8 9084 20044 940388 0 0 128 38516 1230 186 0 2 50 48
1 3 8 8956 19616 940816 0 0 128 36900 1228 190 0 2 48 50
0 4 8 13436 19328 936684 0 0 0 47580 1223 196 0 2 50 48
0 4 8 10276 19300 939832 0 0 256 46008 1200 197 0 3 50 48
0 4 8 11452 19276 938296 0 0 128 38812 1191 153 0 2 39 59
0 4 8 12668 19108 937164 0 0 128 41080 1185 162 0 2 30 67
0 4 8 14320 18612 936360 0 0 128 42200 1190 178 0 3 48 49
0 4 8 8828 18396 942556 0 0 0 38696 1192 157 0 2 50 48
0 4 8 14012 18048 938224 0 0 0 38692 1218 187 0 3 40 58
0 2 8 13428 18068 938464 0 0 128 36556 1223 204 0 1 61 38
0 2 8 13364 18068 938724 0 0 128 42988 1222 178 0 0 63 36
0 2 8 13108 18068 938984 0 0 256 41832 1221 165 0 0 64 36
0 2 8 12980 18068 938984 0 0 128 42344 1250 187 0 0 57 43
0 2 8 12852 18068 939244 0 0 128 37476 1234 179 0 0 62 38
0 2 8 12724 18088 939224 0 0 116 42628 1226 167 0 0 74 25
0 1 8 12484 18088 939484 0 0 272 43176 1243 174 0 0 63 37
0 1 8 11332 18088 940784 0 0 1152 0 1126 66 0 0 75 25
0 1 8 8836 17744 944248 0 0 40616 0 1337 734 0 1 75 24
0 1 8 10244 17476 943996 0 0 42920 0 1355 762 0 1 75 24
0 1 8 9156 17204 946088 0 0 43052 16 1477 837 0 1 74 24
0 1 8 8452 16988 947864 0 0 40872 0 1432 788 0 1 75 24
0 1 8 8708 16620 948752 0 0 37924 0 1347 693 0 1 75 24
0 1 8 10052 16404 948708 0 0 43048 0 1356 778 0 1 75 24
0 1 8 9156 16164 950248 0 0 41000 0 1341 730 0 1 74 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 8 10180 15968 950192 0 8 42924 32 1490 826 0 2 75 24
0 1 8 9284 15596 952124 0 0 43048 0 1357 750 0 1 75 24
0 1 8 9284 15384 952856 0 0 35876 0 1389 676 0 1 75 24
0 1 8 8516 15156 954904 0 0 42924 0 1354 760 0 2 74 24
0 1 8 9732 14920 954360 0 0 43048 0 1354 762 0 1 75 24
0 1 8 9540 14708 955612 0 0 39976 148 1365 714 0 1 75 24
0 1 8 10948 14356 955184 0 0 42920 0 1387 781 0 2 74 24
0 1 8 8580 14136 958524 0 0 38952 1580 1378 707 0 1 75 24
0 1 8 9668 13912 958488 0 0 39976 2468 1392 730 0 1 74 24
0 1 8 8836 13696 960264 0 0 42920 2580 1381 740 0 1 75 24
0 1 8 9476 13484 960216 0 0 43052 1060 1368 766 0 1 75 24
0 1 8 9156 13260 961480 0 0 42024 128 1349 744 0 2 75 24
0 1 8 10116 13032 961188 0 0 36772 256 1309 658 0 1 75 24
0 0 8 9668 12920 962080 0 0 19280 128 1176 382 0 0 89 11
0 0 8 9732 12920 962080 0 0 0 0 1049 60 0 0 100 0
0 0 8 9756 12936 962064 0 0 0 16 1024 46 0 0 100 0


Then, I set elevator=noop in the kernel parameters in grub, and still got:
> dmesg | grep sche
Using cfq io scheduler

An when running vmstat, the results were similar to the first ones.

0 0 0 9672 5364 966256 0 0 0 0 1023 50 0 0 100 0
0 0 0 9696 5364 966256 0 0 0 0 1020 40 0 0 100 0
0 0 0 9696 5364 966256 0 0 0 0 1019 38 0 0 100 0
0 1 0 8976 5472 966668 0 0 292 99208 1424 218 0 1 88 11
1 3 0 124388 5588 853452 0 0 9308 35904 1242 667 0 12 61 27
0 5 0 4656 4912 970608 0 0 128 43452 1248 446 0 14 25 61
0 5 0 4004 4952 971348 0 0 128 44052 1202 214 0 4 18 78
0 5 0 4340 4984 971056 0 0 128 32956 1188 166 0 3 25 72
0 5 0 5824 5028 969452 0 0 256 38300 1204 182 0 4 28 68
0 5 0 4664 5044 970476 0 0 128 16052 1111 97 0 1 28 71
0 5 0 7756 5084 967316 0 0 128 57792 1297 222 0 4 31 65
0 5 0 8788 4112 967248 0 0 128 31208 1174 142 0 2 20 77
0 4 0 14440 4120 961780 0 0 256 34572 1207 160 0 2 35 63
0 4 0 18752 4160 957320 0 0 128 37216 1190 166 0 3 37 60
0 4 0 13244 4192 962488 0 0 0 34440 1158 137 0 2 44 55
0 5 0 12444 4216 963244 0 0 256 29076 1240 179 0 2 40 58
0 4 8 17652 4244 958008 0 0 128 39808 1231 198 0 2 40 57
0 4 8 10124 4280 964992 0 0 128 38912 1212 174 0 3 34 63
0 4 8 11020 4308 964444 0 0 256 31516 1146 144 0 2 27 71
0 5 8 13124 4344 961808 0 0 256 41112 1208 185 0 3 24 73
0 5 8 11524 4368 963084 0 0 128 23316 1141 148 0 2 51 48
0 5 8 8508 4416 966156 0 0 128 47152 1219 186 0 3 47 50
0 2 8 9084 4444 965608 0 0 220 36104 1221 197 0 0 69 31
0 2 8 9084 4516 965536 0 0 0 41464 1237 220 0 0 59 40
0 2 8 8956 4516 965536 0 0 128 42600 1217 173 0 0 67 33
0 2 8 8828 4516 965796 0 0 128 42664 1223 170 0 0 61 39
0 2 8 8828 4516 965796 0 0 0 41580 1217 154 0 0 67 33
0 2 8 8836 4516 965796 0 0 0 38244 1203 156 0 0 61 39
0 2 8 8836 4516 965796 0 0 0 41832 1221 154 0 0 55 45
0 1 8 9604 4520 965012 0 0 1284 5764 1155 89 0 0 75 26
0 1 8 10256 4556 964456 0 0 38820 0 1337 708 0 1 75 24
0 1 8 9484 4584 965468 0 0 40232 0 1334 714 0 1 75 24
0 1 8 9932 4624 965688 0 0 43052 0 1354 770 0 1 75 24
1 1 8 9676 4664 966168 0 0 43176 0 1373 791 0 1 75 24
0 1 8 9100 4660 966952 0 0 38436 0 1320 683 0 1 75 24
0 1 8 8652 4680 967972 0 0 40232 8 1400 767 0 1 75 24
0 1 8 9228 4724 967928 0 0 43052 0 1356 759 0 1 75 24
0 1 8 9740 4764 967628 0 0 43176 0 1372 775 0 1 75 24
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 1 8 8780 4820 968872 0 0 40232 16 1337 742 0 1 75 24
0 1 8 10124 4856 967796 0 0 38180 0 1320 694 0 1 75 24
0 1 8 8460 4896 970104 0 8 43052 8 1407 782 0 2 75 24
0 1 8 9996 4936 968764 0 0 40104 0 1334 720 0 1 75 24
1 0 8 8652 4976 970544 0 0 43176 0 1356 775 0 1 75 24
0 1 8 8396 5036 971264 0 0 43308 1980 1387 772 0 1 75 24
0 1 8 12364 5100 968080 0 0 35776 2000 1327 646 0 2 75 24
0 1 8 9484 5140 971160 0 0 43048 3560 1469 804 0 1 75 24
0 1 8 10380 5172 970088 0 0 31904 25300 1411 601 0 2 74 24
0 1 8 8972 5196 971884 0 0 24088 11748 1276 450 0 1 75 24
1 0 8 17164 5264 963756 0 0 33696 5992 1319 603 0 1 75 24
0 1 8 9036 5304 972036 0 0 42920 0 1387 778 0 1 74 24
0 1 8 9164 5288 971792 0 0 43180 0 1357 776 0 1 75 24
0 1 8 10292 5316 970984 0 0 21412 0 1211 454 0 1 75 24
0 1 8 10320 5352 970688 0 0 6072 0 1096 198 0 1 75 24
0 1 8 8588 5352 972508 0 0 5912 280 1160 188 0 1 75 24
0 0 8 10124 5364 970936 0 0 524 0 1030 66 0 1 97 3
0 0 8 10140 5364 970936 0 0 0 0 1019 42 0 0 100 0
0 0 8 10140 5364 970936 0 0 0 0 1021 38 0 0 100 0
0 0 8 10148 5368 970932 0 0 4 0 1026 60 0 0 100 0

In the three cases, the real time spent in the operation was approximately the
same.

Am I setting the scheduler wrong? Anybody knows why writes/reads are
"serialized" even when using cfq ?

Thanks,

--
Antonio Larrosa
antlarr@xxxxxxxxxx
TEDIAL - Tecnologías Digitales Audiovisuales, S.L.
Parque Tecnológico de Andalucía
C/ Severo Ochoa, nº 10, 1ª planta
29590 Campanillas, Málaga
Tfno. +34 951 012 624
Fax +34 951 012 610
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/