I thought I'd see if anyone else has seen such a case or could offer a
remedy. ;)
I saw that there are fixes in the 2.3.4+ code branch for a network
bottom half scalability issue. But, in that case, I couldn't find any
mention of performance being worse with MP than with UP. So, this may
or may not be the same problem. Informed comments would be greatly
appreciated.
TEST SUMMARY:
The dual cpu machine was configured with two NICs, each on a different
isolated network. IP forwarding was enabled. Using a standalone
packet-generator/analyzer box, we sent a continuous series of identical
UDP packets on one network with a destination address on the other
network. We used UDP port 9 (discard) so that all packets were
discarded by the destination host. So, all traffic was in one
direction. The analyzer box has two NICs. One is used to send
packets. The other listens (in promiscuous mode) for the forwarded
packets on the destination network. Packet loss statistics are
displayed on the analyzer.
RESULTS:
Booting a UP kernel, we were able to reach a steady state where 45000
(67 byte) packets/sec could be forwarded without packet loss. However,
when we switched to an SMP kernel, we could only reach ~22500 (67 byte)
packets/sec. Any more than that and we started losing packets on the
destination net.
CPU usage from "top" and "vmstat" indicated that at 45000 pps, the UP
kernel was using 94% system time and 6% remained idle (see note 1
below). At 22500 pps, the SMP kernel was using ~59% (aggregated)
system time with 41% idle.
I was able to reach 26000 pps with the SMP kernel after binding both
network IRQs to the same cpu (see note 2 below). That reduced cpu usage
to ~50% system with ~50% idle. Could this this be the same or different
lock granularity problem fixed in 2.3.4+?
We were hoping to stick with a stable kernel as that is what we'll
need to use for deployment. But, if someone is pretty sure this is
fixed in a "stable enough" 2.3.x kernel, then we might be able to try
that.
NOTES:
1) Unless there is a runnable user process, interrupt handling is not
accounted for as system time. The system can be 100%busy handling
interrupts, yet appear idle when looking at "top", "vmstat", etc. So, I
wrote a cpu consumer program "while (1) {;}" and ran it "nice -20
./loop". That way, interrupt handling activity appears as system time
and "free" cpu appears as "nice" time in the cpu usage statistics.
Crude, but it seems to work.
2) I modified io_apic.c to bind the two network interrupts to the same
cpu and then to different cpus. Got some "interesting" results in each
case, but it didn't fix the poor MP performance problem.
3) Here are the configuration details of the System Under Test:
Motherboard: Intel N440BX dual-Slot 1
Chipset: Intel 440BX
Integrated video: Cirrus Logic GD5480
CPU: 2 x Intel Pentium II/400
NIC1: Integrated Intel EtherExpress Pro 10/100
NIC2: Intel EtherExpress Pro 10/100 (PCI card)
Linux Distribution: RedHat 6.0
Kernels: generic kernels off CD (2.2.5-15 and 2.2.5-15smp). We also
built various stripped down kernels, but they didn't fix the poor MP
routing performance problem.
TIA,
---Fred P.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/