Hey Robert -
I don't have any insight on the kernel's elevator scheduling algorithm,
but I'm intrigued by the imbalance you are seeing in client throughput.
It sounds like this could be a combined effect of an anomoly in the
elevator scheduling and an unstable equilibrium in your test
environment. Here is my guess (shot in the dark!) as to what might be
going on:
Since the clients are reading and writing the same 30MB area of the disk
over and over, clients whose files are in the middle of the disk arm
sweep will be favored by the elevator scheduling, b/c they can be
serviced both coming and going. The clients with slightly slower server
response times (due to the placement of files on disk) will have to wait
longer before sending a subsequent request. In the mean time, the other
clients will have been able to issue more reuqests, so by the time the
slower client gets its next request in, it is likely that it will be at
the end of the elevator loop, and hence be delayed further...
As I said this is pure conjecture - I haven't tested this out at all.
So, for the FS guys: does this seem like a plausible explanation? If
so, should the elevator algorithm be tweaked, so as not to play
favorites with processes writing to the middle of the disk? (Perhaps
have it always scan the disk in the same direction?)
Best regards,
-Rob von Behren
Robert Cohen wrote:
>
> I would like to report on my findings about Linux file serving
> performance. I will start with results for 2.2 kernels and finish with
> 2.4 results.
>
> I have been doing some evaluation of Linux as a Appleshare file server
> using Netatalk. The benchmark I am using is mac program called lantest,
> but the part of it I am using just writes a 30 Meg file and then reads
> it back and repeats this continuously.
> And I vary the number of client machines I run it on simultaneously.
> All tests are done over 100Mbit ethernet.
>
> I started with Kernel version 2.2.17(preX).
> With 1 client, performance is fine 5-6 Megabytes/sec.
> But on a 128 Meg server with 2 clients, I hit a performance wall
> and the cumulative throughput for the write test drops to less than a
> meg per second.
> It appears to be seek bound as I can hear the disk seeking madly.
> There appears to be a effect from the memory size since a 256 Meg server
> can handle 2 clients, but goes seek bound with 4 clients.
> However, 4 clients are only writing files with a total size of 120 Meg
> so it should be able to cache it all anyway.
>
> I decided to check it was disk seeks that were the performance
> limitation.
> So I ran the test with 2 clients writing to separate disks.
> The performance is OK even with as little as 32 Megs of memory.
> So there are no bottlenecks in the network code or the buffer cache etc.
>
> The mildly odd thing is that performance does seem to be effected by the
> fact that its a networked test. The file serving gets handled by a
> separate process for each client. I can strace these processes and see
> that they are just doing an 8k read from the network followed by an 8k
> write to disk. If I use the IOzone throughput test with multiple
> processes doing 8k reads and writes of 30 Meg files, I don't see the
> same sort of performance problems.
>
> My current theory is that waiting for the network read is causing a
> context switch to the next process hence interleaving the writes from
> all the processes. Under IOzone, each process is probably doing multiple
> 8k writes in its timeslice.
>
> Since the benchmark is going seek bound, it sounds like the elevator
> algorithm isn't doing its job. The 2.2 kernel is supposed to have an
> elevator, it just isn't guaranteed to be fair.
> So we should still see high total throughput, just not evenly spread.
> But even total throughput is very low; rarely more than 150 blocks in 5
> seconds according to vmstat. Less than 1 Meg per second aggregate
> according to the clients.
> So this is a problem for the 2.2 kernels. I tested a 2.0 kernel as well
> and saw the same problem there, so this isn't new for 2.2.
>
> Anyway, the 2.4 kernels are supposed to have a funky new elevator.
> So I decided to try some 2.4 kernels. I have tried 2.4.0-test4 and
> 2.4.0-test5pre2, 2.4.0-test1-ac22-class and 2.4.0-test1-ac22-riel.
>
> Kernels test1-ac22-class and test1-acc22-riel perform quite well even
> with 5 clients on a 128 Meg server.
> The new elevator really helps with write performance.
> The blocks written per 5 secs result from output is over 1000.
> Aggregate client output is over 10 Megs per second.
> However there is still some tendency towards client starvation.
> Often one client will drop to less than 300 K per second for around a
> minute.
>
> The more recent test4 and test5pre2 don't fair quite so well.
> They handle 2 clients on a 128 Meg server fine, so they're doing better
> than 2.2 but they choke and go seek bound with 4 clients.
> So something has definitely taken a turn for the worse since test1-ac22.
>
> I was hoping to do some experimentation with elvtune and see how much
> effect various elevator settings have on the performance. However, I
> have been unable to get elvtune to run with any of the 2.4 kernels. The
> ioctl which it uses gives an IO error. If anyone has suggestions as to
> why elvtune isnt working, please let me know.
>
> --
> Robert Cohen
> Unix Support
> TLTSU - Australian National University
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.rutgers.edu
> Please read the FAQ at http://www.tux.org/lkml/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Sun Jul 23 2000 - 21:00:14 EST