Re: BUG: Slowdown on 3000 socket-machines tracked down

From: Christian Schmid
Date: Sun Mar 06 2005 - 20:59:51 EST


Ben Greear wrote:
Christian Schmid wrote:

Hello.

After weeks of work, I can now give a detailed report about the bug and when it appears:

Attached is another traffic-image. This one is with 2.6.10 and a 3/1 split, preemtive kernel, so all defaults.


What are the units on your graph. You say "MB" several places, but
do you mean Mb (ie, Mega-bit) instead?

The unit on this graph is kilobytes. So 80000 there means 80 megabytes per second.

I have a tool that can also generate TCP traffic on a large number of
sockets. If I can understand what you are trying to do, I may be able
to reproduce the problem. My biggest machine at present has only
2GB of RAM, however...not sure if that matters or not.

It should not matter. Low-memory is both just 1 GB if you have default 32 bit with 3/1 split.

Are you sending traffic in only one direction, or more of a full-duplex
configuration?

Its a full-duplex. Its a download-service with 3000 downloaders all over the world.

Is each socket running the same bandwidth?

No. It ranges from 3 kb/sec to 100 kb/sec. 100 kb/sec is the limit because of the send-buffer limits.

What is this bandwidth?

1000 MBit

Are you setting the send & rcv buffers in the socket creation
code? (To what values if so?)

Yes. send-buffer to 64 kbytes and receive buffer to 16 kbytes.

How many bytes are you sending with each call to write()/sendto() whatever?

I am using sendfile-call every 100 ms per socket with the poll-api. So basically around 40 kb per round.

Is there any significant latency between your sender and receiver machine?
If so, how much?

3000 different downloaders, 3000 different locations, 3000 different machines ;)

What is the physical transport...GigE? 1500 MTU?

Yes.

Chris

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/