Poor UDP performance using 2.6.21-rc5-rt5

[Dave Sperry]

Hi
I have a dual core Opteron machine that exhibits poor UDP performance 
(RT consumes more than 2X cpu) with the 2.6.21-rc5-rt5 as compared to 
2.6.21-rc5. Top shows the IRQ handler consuming a lot of CPU.

The mother board is a Supermicro H8DME-2 with one dual core Opteron 
installed. The networking is provided by the on board nVidia MCP55Pro chip.

The RT test is done using netperf 2.4.3 with the server on an IBM LS20 
blade running RHEL4U2 and the Supermicro running netperf under RHEL5 
with 2.6.21-rc5-rt5. 

The Non-RT test was done on the exact same setup except 2.6.21-rc5-rt5 
was loaded on the SuperMicro board.

Cyclesoak was used to measure CPU utilization in all cases.



Here are the RT results
########################################################3
## 2.6.21-rc5-rt5
#######################################################
$ !netper
netperf -l 100 -H 192.168.70.11 -t UDP_STREAM -- -m 1025
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
192.168.70.11 (192.168.70.11) port 0 AF_INET
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

126976    1025   100.00    8676376      0     711.46
135168           100.00    8676376            711.46

########## cyclesoak during test
$ ./cyclesoak
using 2 CPUs
System load: -0.1%
System load: 40.5%
System load: 51.6%
System load: 51.5%
System load: 50.9%
System load: 50.7%
System load: 50.8%
System load: 50.7%
System load: 50.6%

######## top during test
top - 13:26:48 up 8 min,  4 users,  load average: 1.74, 0.46, 0.15
Tasks: 149 total,   4 running, 145 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us, 16.8%sy, 50.6%ni,  0.0%id,  0.0%wa, 25.6%hi,  6.3%si,  
0.0%st
Mem:   2035444k total,   465888k used,  1569556k free,    28840k buffers
Swap:  3068372k total,        0k used,  3068372k free,   318668k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3865 eadi      39  19  6804 1164  108 R  100  0.1   0:38.25 cyclesoak
2715 root     -51  -5     0    0    0 S   51  0.0   0:09.52 IRQ-8406
3867 eadi      25   0  6440  632  480 R   34  0.0   0:06.03 netperf       
  19 root     -51   0     0    0    0 S   13  0.0   0:02.33 softirq-net-tx/
3866 eadi      39  19  6804 1164  108 R    1  0.1   0:20.47 cyclesoak
3167 root      25   0 29888 1180  888 S    0  0.1   0:00.93 automount
3861 eadi      15   0 12712 1076  788 R    0  0.1   0:00.19 top
   1 root      18   0 10308  668  552 S    0  0.0   0:00.67 init      
   2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0   
   3 root      RT   0     0    0    0 S    0  0.0   0:00.00 posix_cpu_timer
   4 root     -51   0     0    0    0 S    0  0.0   0:00.00 softirq-high/0
   5 root     -51   0     0    0    0 S    0  0.0   0:00.00 softirq-timer/0
   6 root     -51   0     0    0    0 S    0  0.0   0:00.00 softirq-net-tx/
   7 root     -51   0     0    0    0 S    0  0.0   0:00.00 softirq-net-rx/
   8 root     -51   0     0    0    0 S    0  0.0   0:00.00 softirq-block/0

########################
The baseline results:
RHEL5 with 2.6.21-rc5 kernel
##############################

$  netperf -l 100 -H 192.168.70.11 -t UDP_STREAM -- -m 1025

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
192.168.70.11 (192.168.70.11) port 0 AF_INET
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

126976    1025   100.00    11405485      0     935.24
135168           100.00    11405485            935.24

#######################################
$ ./cyclesoak
using 2 CPUs
System load:  7.6%
System load: 29.6%
System load: 29.6%
System load: 28.9%
System load: 24.9%
System load: 25.0%
System load: 24.8%
System load: 24.9%

#######################################
top:top - 13:52:22 up 10 min,  6 users,  load average: 1.46, 0.43, 0.17
Tasks: 118 total,   4 running, 114 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  9.8%sy, 75.7%ni,  0.0%id,  0.0%wa,  5.8%hi,  8.1%si,  
0.0%st
Mem:   2057200k total,   459128k used,  1598072k free,    29020k buffers
Swap:  3068372k total,        0k used,  3068372k free,   318968k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
3882 eadi      39  19  6804 1164  108 R  100  0.1   0:52.11 cyclesoak
3881 eadi      39  19  6804 1164  108 R   65  0.1   0:38.47 cyclesoak
3883 eadi      15   0  6436  632  480 R   35  0.0   0:18.26 netperf
3879 eadi      15   0 12580 1052  788 R    0  0.1   0:00.15 top
   1 root      18   0 10308  664  552 S    0  0.0   0:00.48 init      
   2 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0
   3 root      34  19     0    0    0 S    0  0.0   0:00.01 ksoftirqd/0
   4 root      RT   0     0    0    0 S    0  0.0   0:00.00 watchdog/0
   5 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1
   6 root      34  19     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1

Any thoughts on how to fix this?

Thanks,
-Dave











-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/