Re: multicast performance

From: Matt Garman
Date: Tue Feb 09 2010 - 12:13:45 EST



Resurrecting a long-dead thread regarding multicast latencies.
Original post is at the bottom.

On Thu, Nov 05, 2009 at 05:02:25PM +0100, Marco P. wrote:
> - Multicast performance decreases with the number of senders.
> I.e. if a single process can send data at 800Mbit/s, then two senders
> (most likely) won't be able to send data at 400Mbit/s each.
> (just saying that since you're using "many" servers)

That makes sense. However, we're sending lots of small packets, so
it's more an issue of latency than throughput.

> - Timestamping after receive() can be very imprecise.
> You should probably use in-kernel timestamping for that purpose.

How is this done? Is there a function call or a socket parameter?
And how does one see the kernel timestamps to assess latencies?

Thanks again,
Matt


> On 5 Nov, 2009, at 16:15 , Matt Garman wrote:
>
> >We are using multicast quite heavily for some internal messaging
> >across many Linux (CentOS 4.x, RHEL kernel 2.6.9) servers.
> >
> >Effectively, we create our message, timestamp it (high resolution
> >timestamp via gettimeofday()), and call send(). The receiver gets
> >the message via recv(), adds another timestamp, and pushes it into a
> >queue. The queue is processed on a separate thread.
> >
> >Note that all our machines are time sync'ed via NTP.
> >
> >Part of that thread processing is analyzing the elapsed time between
> >send() and recv(). We are seeing increasing delays recently.
> >Though our traffic rate has increased, our machine load has not.
> >Our machines are all modern (Conroe or i7 architecture) dual-socket
> >quad core (eight total cores) Xeon boxes with 8 GB of RAM. The
> >machines' load is always under 1.00, and memory usage under 5%.
> >
> >We are fairly certain the networking hardware is not the culprit, as
> >it is well under 50% load.
> >
> >We *suspect* the Linux kernel scheduling may have something to do
> >with this. But this is only a hunch, and we do not know enough
> >about Linux internals to know where to being looking more closely.
> >
> >Whenever I've done a net search for "linux multicast", the results
> >are always dated by several years, often talking about 2.4 and
> >earlier kernels. Given Linux's development pace, I'm afraid to put
> >much stock in this information.
> >
> >So, a few questions:
> >
> > - Does anyone have any experience dealing with issues similar to
> > what I described above?
> > - Can anyone point me to some recent information on Linux's
> > multicast implementation?
> > - Can anyone make some general comments about Linux, multicast,
> > scheduling, etc that might be relevant?
> > - I'm also looking for tests and/or benchmarks to help me ask a
> > more specific question.
> >
> >I realize this is a somewhat vague question, but I'm at the "extreme
> >ignorance" point where I really don't know where to start or even
> >what questions to ask.
> >
> >Any thoughts or suggestions would be most appreciated!
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html