Re: High contention on the sk_buff_head.lock

From: Rick Jones
Date: Wed Mar 18 2009 - 18:20:13 EST


Andi Kleen wrote:
Thanks. I will test to see how this affects this lock contention the
next time the broadcom hardware is available.


The other strategy to reduce lock contention here is to use TSO/GSO/USO.
With that the lock has to be taken less often because there are less packets
travelling down the stack. I'm not sure how well that works with netperf style workloads though.

All depends on what the user provides with the test-specific -m option for how much data they shove into the socket each time "send" is called, and I suppose if they use a test-specific -D option to set TCP_NODELAY in the case of a TCP test when they have small values of -m. Eg

netperf -t TCP_STREAM ... -- -m 64K
vs
netperf -t TCP_STREAM ... -- -m 1024
vs
netperf -t TCP_STREAM ... -- -m 1024 -D
vs
netperf -t UDP_STREAM ... -- -m 1024

etc etc.

If the netperf test is:

netperf -t TCP_RR ... -- -r 1 (single-byte request/response)

then TSO/GSO/USO won't matter at all, and probably still wont matter even if the user has ./configure'd netperf with --enable-burst and does:

netperf -t TCP_RR ... -- -r 1 -b 64
or
netperf -t TCP_RR ... -- -r 1 -b 64 -D

which was basically what I was doing for the 32-core scaling stuff I posted about a few weeks ago. That was running on multi-queue NICs, so looking at some of the profiles of the "no iptables" data may help show how big/small the problem is, keeping in mind that my runs (either the XFrame II runs, or the Chelsio T3C runs before them) had one queue per core in the system...and as such may be a best case scenario as far as lock contention on a per-queue basis goes.

ftp://ftp.netperf.org/

happy benchmarking,

rick jones

BTW, that setup went "poof" and had to go to other nefarious porpoises. I'm not sure when I can recreate it, but I still have both the XFrame and T3C NICs when the HW comes free again.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/