Yes, 2.6.11. I have tuned max_backlog and some other TCP and networking
related settings to give more buffers etc to networking tasks. I have not
tried any significant disk-IO while doing these tests.
I finally got my systems set up so I can run my WAN emulator at full 1Gbps:
I am getting right at 986Mbps throughput with 30ms round-trip latency
(15ms in both directions).
So, latency does not seem to be the problem either.
I think the problem can be narrowed down to:
1) Non-optimal kernel network tunings on your server.
2) Disk-IO (my disk is small and slow compared to a 'real' server, not sure I can
really test this side of things, and I have not tried as of yet.)
3) Your clients have much more latency and/or don't have enough bandwidth
to fully load your server. Since you didn't answer before: I assume you
do not have a reliable test bed and are just hoping that enough clients connect
to do your benchmarking.
4) There is something strange with sendfile and/or your application's coding.
My suggestion would be to eliminate these variables by coming up with a repeatable
test bed, alternative traffic generators, WAN/Network emulators for latency, etc.