I would like to report on my findings about Linux file serving
performance. I will start with results for 2.2 kernels and finish with
2.4 results.
I have been doing some evaluation of Linux as a Appleshare file server
using Netatalk. The benchmark I am using is mac program called lantest,
but the part of it I am using just writes a 30 Meg file and then reads
it back and repeats this continuously.
And I vary the number of client machines I run it on simultaneously.
All tests are done over 100Mbit ethernet.
I started with Kernel version 2.2.17(preX).
With 1 client, performance is fine 5-6 Megabytes/sec.
But on a 128 Meg server with 2 clients, I hit a performance wall
and the cumulative throughput for the write test drops to less than a
meg per second.
It appears to be seek bound as I can hear the disk seeking madly.
There appears to be a effect from the memory size since a 256 Meg server
can handle 2 clients, but goes seek bound with 4 clients.
However, 4 clients are only writing files with a total size of 120 Meg
so it should be able to cache it all anyway.
I decided to check it was disk seeks that were the performance
limitation.
So I ran the test with 2 clients writing to separate disks.
The performance is OK even with as little as 32 Megs of memory.
So there are no bottlenecks in the network code or the buffer cache etc.
The mildly odd thing is that performance does seem to be effected by the
fact that its a networked test. The file serving gets handled by a
separate process for each client. I can strace these processes and see
that they are just doing an 8k read from the network followed by an 8k
write to disk. If I use the IOzone throughput test with multiple
processes doing 8k reads and writes of 30 Meg files, I don't see the
same sort of performance problems.
My current theory is that waiting for the network read is causing a
context switch to the next process hence interleaving the writes from
all the processes. Under IOzone, each process is probably doing multiple
8k writes in its timeslice.
Since the benchmark is going seek bound, it sounds like the elevator
algorithm isn't doing its job. The 2.2 kernel is supposed to have an
elevator, it just isn't guaranteed to be fair.
So we should still see high total throughput, just not evenly spread.
But even total throughput is very low; rarely more than 150 blocks in 5
seconds according to vmstat. Less than 1 Meg per second aggregate
according to the clients.
So this is a problem for the 2.2 kernels. I tested a 2.0 kernel as well
and saw the same problem there, so this isn't new for 2.2.
Anyway, the 2.4 kernels are supposed to have a funky new elevator.
So I decided to try some 2.4 kernels. I have tried 2.4.0-test4 and
2.4.0-test5pre2, 2.4.0-test1-ac22-class and 2.4.0-test1-ac22-riel.
Kernels test1-ac22-class and test1-acc22-riel perform quite well even
with 5 clients on a 128 Meg server.
The new elevator really helps with write performance.
The blocks written per 5 secs result from output is over 1000.
Aggregate client output is over 10 Megs per second.
However there is still some tendency towards client starvation.
Often one client will drop to less than 300 K per second for around a
minute.
The more recent test4 and test5pre2 don't fair quite so well.
They handle 2 clients on a 128 Meg server fine, so they're doing better
than 2.2 but they choke and go seek bound with 4 clients.
So something has definitely taken a turn for the worse since test1-ac22.
I was hoping to do some experimentation with elvtune and see how much
effect various elevator settings have on the performance. However, I
have been unable to get elvtune to run with any of the 2.4 kernels. The
ioctl which it uses gives an IO error. If anyone has suggestions as to
why elvtune isnt working, please let me know.
-- Robert Cohen Unix Support TLTSU - Australian National University- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Sun Jul 23 2000 - 21:00:13 EST