On Fri, Aug 24, 2001 at 05:19:07PM -0300, Rik van Riel <riel@conectiva.com.br> wrote:
> Actually, no. FIFO would be ok if you had ONE readahead
> stream going on, but when you have multiple readahead
Do we all agree that read-ahead is actually the problem? ATM, I serve ~800
files, read()ing them in turn. When I increase the number of threads I
have more reads at the same time in the kernel, but the absolute number of
read() requests decreases.
So if read-ahead is the problem, then there must be some interdependency
between read-aheads for serial requests and read-aheads for concurrent
requests.
For example, I could imagine that read() executes, returns to
userspace and at the same time the kernel thinks "nothing to do, let's
readahead". While, in the concurrent case, there is hardly a time when no
read() is running. But read-ahead does not seem to work that way.
I usually have around 200MB of free memory, this leaves ~200k/handle
(enough for read-ahead). let's see what happens with vmstat. this is after i
kill -STOP httpd:
0 2 0 0 4140 15448 185972 0 0 0 264 1328 88 1 3 96
0 2 0 0 4136 15448 185976 0 0 0 0 793 62 0 3 97
0 2 0 0 4136 15448 185976 0 0 0 0 538 62 0 1 98
0 2 0 0 4128 15448 185984 0 0 0 0 382 62 1 1 98
0 2 0 0 4128 15448 185984 0 0 0 0 312 66 0 1 99
1 1 0 0 3056 15448 185088 0 0 48 132 2465 321 24 28 48
2 1 1 0 3056 15448 186392 0 0 4704 0 3017 925 11 21 68
1 1 2 0 3804 15440 185516 0 0 4708 0 3909 1196 28 38 34
1 1 0 0 3056 15440 186212 0 0 5392 0 4004 1579 21 32 47
1 1 1 0 3064 15436 186220 0 0 6668 0 3676 1273 19 42 39
0 1 1 0 3056 15468 186168 0 0 4488 1424 3889 1342 16 34 50
0 1 2 0 3056 15468 186116 0 0 3372 0 3854 1525 20 34 46
1 1 1 0 3060 15468 186084 0 0 4096 0 4088 1641 21 37 41
0 1 2 0 3056 15468 186044 0 0 6744 0 3679 1415 22 33 45
1 1 1 0 3072 15468 186040 0 0 4700 0 3713 1429 19 32 50
at some point I killall -CONT myhttpd (it seems more than the 2mb/s is
being read, though, although I only server about 2.3mb/s). now let's see
when i dynamically add 15 reader threads:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 1 1 0 3056 14636 193104 0 0 5248 0 3531 1428 11 35 53
0 1 2 0 3056 14604 193072 0 0 5936 0 3730 1416 10 33 57
0 1 0 0 3056 14604 193068 0 0 5628 0 4025 1504 16 34 50
# added 15 threads
0 16 0 0 3060 14580 192952 0 0 5412 0 3394 1010 12 31 57
1 16 1 0 3060 14580 192944 0 0 2580 268 2253 636 8 23 69
0 16 0 0 3056 14580 192944 0 0 380 852 1570 488 5 13 82
0 16 0 0 3064 14580 192880 0 0 2652 0 1470 583 6 14 80
0 16 0 0 3056 14560 192888 0 0 2944 0 2068 693 5 23 72
0 16 0 0 3056 14560 192888 0 0 544 0 1732 550 5 15 80
1 16 0 0 3056 14560 192884 0 0 2892 0 1513 741 4 14 82
0 16 1 0 3056 14560 192884 0 0 2236 552 1894 742 5 17 77
1 16 0 0 3056 14560 192880 0 0 1100 128 1699 604 4 15 81
woaw. blocks in decreases a slot. now let's remove the extra threads again:
0 16 1 0 3056 14552 193720 0 0 4944 660 1463 721 3 20 77
0 16 0 0 3056 14552 193648 0 0 1756 136 1451 726 4 18 79
0 16 0 0 3056 14552 193588 0 0 440 0 1221 565 3 13 84
0 16 0 0 3056 14552 193584 0 0 3308 0 1278 632 4 9 88
1 16 0 0 3056 14536 193608 0 0 3040 0 2469 1168 7 21 72
0 16 0 0 3056 14536 193608 0 0 2320 0 1844 730 5 15 80
1 16 1 0 3056 14536 193612 0 0 1660 596 1557 559 12 24 64
# here the server starts rto reap threads. this is a slow and inefficient process
# that take svery long and block the server.
1 16 0 0 3056 14536 193612 0 0 2188 164 831 440 25 30 45
1 16 0 0 3056 14536 193612 0 0 2324 0 506 329 23 30 47
1 16 0 0 3056 14536 193612 0 0 1452 0 460 401 24 30 46
# many similar lines snipped
1 16 0 0 3056 14516 193692 0 0 3932 0 510 621 20 38 42
1 16 0 0 3056 14516 193692 0 0 1744 0 338 369 23 31 46
1 16 0 0 3056 14568 193872 0 0 1292 476 383 392 20 32 48
2 0 2 0 3056 14616 175104 0 0 5748 0 3670 1342 24 37 39
2 1 1 0 3560 14616 174708 0 0 5028 0 3539 989 22 43 35
0 1 0 0 93948 14660 175764 0 0 1604 0 3341 667 10 22 68
1 0 0 0 92244 14844 176212 0 0 524 0 3240 424 12 12 76
0 1 0 0 90532 15212 176404 0 0 200 1524 3308 426 16 14 71
0 1 1 0 84600 15212 179096 0 0 2712 0 2710 669 26 11 63
0 1 2 0 77768 15212 183132 0 0 4012 0 3041 889 19 18 63
0 1 0 0 68724 15212 189440 0 0 6284 0 3110 998 21 21 58
1 1 0 0 58892 15212 195984 0 0 6528 0 3149 975 28 25 47
2 1 0 0 50636 15248 201316 0 0 5316 1368 3321 968 20 28 52
1 1 0 0 38520 15248 210004 0 0 8664 0 3250 910 28 26 46
0 1 1 0 28100 15248 218520 0 0 8508 0 3186 777 20 28 52
1 1 1 0 18848 15248 227028 0 0 8500 0 3090 704 15 26 59
0 1 0 0 10752 15248 233752 0 0 6732 0 3223 774 20 27 53
back to 2.1mb/s, but reading much more.
this certainly looks like overzealus read-ahead, but I should have the
memory available for read-ahead. So is it "just" the use-once-optimization
that throws away read-ahead pages? If yes, then why do I see exactly
the same performance under 2.4.5pre4, which didn't have (AFAIK) the
use-once-opt.?
--
-----==- |
----==-- _ |
---==---(_)__ __ ____ __ Marc Lehmann +--
--==---/ / _ \/ // /\ \/ / pcg@goof.com |e|
-=====/_/_//_/\_,_/ /_/\_\ XX11-RIPE --+
The choice of a GNU generation |
|
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Fri Aug 31 2001 - 21:00:15 EST