Re: big picture UDP/IP performance question re 2.6.18 -> 2.6.32

From: Ben Hutchings
Date: Fri Oct 07 2011 - 16:07:16 EST

On Fri, 2011-10-07 at 14:37 -0400, starlight@xxxxxxxxxxx wrote:
> At 02:09 PM 10/7/2011 -0400, chetan loke wrote:
> b) Use non-commodity(?) NICs(from vendors
> >you mentioned): where it might have some on-board
> >memory(cushion) and so it can absorb the spikes
> >and can also smoothen out too many
> >PCI-transactions for bursty (and small payload -
> >as in 64 byte traffic). But wait, when you use the
> >libs provided by these vendors, then their
> >driver(especially the Rx path) is not so much
> >working in inline mode as NIC drivers in case a)
> >above. This driver with a special Rx-path purely
> >exists for managing your mmap'd queues.So
> >of-course it's going to be faster that the
> >traditional inline drivers. In this partial-inline
> >mode, the adapter might i) batch the packets and
> >ii) send a single notification to the
> >host-side. With that single event you are now
> >processing 1+ packets.

Solarflare's user-mode queues have their own wakeup moderation timers,
similar to interrupt moderation timers.

Note also that NICs designed for user-mode networking may allow DMA
descriptor/event rings and doorbell registers to be mapped into user
processes. This allows packets to be passed continuously without any
need to call into the kernel.

> Kernel bypass is probably the best answer for
> what we do. Problem has been lack of maturity
> in their driver software. Looks like it's reaching
> a point where they cover our use case. As mentioned
> earlier, Solarflare could not match the Intel
> 82599 + ixgbe for this app last year. Was a
> disaster.

Ouch. What's the application?

> Myricom is focused on UDP (better
> for us), but only just added multi-core IRQ
> doorbell wakeups in recent months. Previously
> one had to accept all IRQs on a single core or
> poll, neither of which works for us.

Our hardware does support spreading wakeups for user-mode queues across
multiple kernel-mode event queues (hence multiple CPUs), although not in
a very flexible way. I believe most Onload users prefer to keep their
threads polling for events rather than waiting for wakeups, though, so
it doesn't yet use this feature.


Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at