_High_ CPU usage while routing (mostly) small UDP packets
From: Karsten Desler
Date: Mon Dec 06 2004 - 15:56:39 EST
Hi,
I'm running a dual Opteron 244 router with two active e1000 interfaces,
that mostly deals with small (<200b) udp packets.
I'm using Linux 2.6.10-rc3 (32bit), but I tried 64bit with early 2.6.9-rc
versions, and it didn't make much of a difference.
The irq of eth0 is bound to cpu1 while eth1 is bound to cpu0.
NAPI is enabled.
Current packetload on eth0 (and reversed on eth1):
115kpps tx
135kpps rx
There are about 200 iptables rules, but the common packet only has to
traverse about 20.
conntrack is not loaded.
eth0 and eth1 are running on the same 66MHz/64bit PCI bus.
Kernel-Profiling is running, I don't know how much that contributes to the
overall load.
I don't know if that has anything to do with it, but the systemclock is
getting out of sync _fast_ (openntpd can't keep up).
ntpdate -b ntp.soohrt.org; sleep 60; ntpdate ntp.soohrt.org:
6 Dec 21:40:39 ntpdate[30146]: adjust time server 134.100.177.5 offset 0.000092 sec
6 Dec 21:41:39 ntpdate[30218]: adjust time server 134.100.177.5 offset 0.006971 sec
Is that the expected cpu usage?
I'd appreciate _any_ pointers, thanks in advance,
Karsten
Current cpu usage:
Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 46.3% id, 0.0% wa, 0.0% hi, 53.7% si
Cpu1 : 0.0% us, 0.0% sy, 0.0% ni, 21.4% id, 0.0% wa, 0.0% hi, 78.6% si
vmstat 5:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 1804612 100488 104464 0 0 0 32 5965 167 3 68 30 0
0 1 0 1804548 100488 104464 0 0 0 157 5994 18 0 67 33 0
1 1 0 1804684 100488 104464 0 0 0 63 5998 19 0 67 33 0
0 1 0 1804684 100492 104460 0 0 0 4 5985 10 0 68 33 0
0 1 0 1804620 100492 104460 0 0 0 12 6032 15 0 68 32 0
lspci -vt:
-[00]-+-06.0-[03]--+-00.0 Advanced Micro Devices [AMD] AMD-8111 USB
[...]
+-0a.1 Advanced Micro Devices [AMD] AMD-8131 PCI-X APIC
+-0b.0-[01]--+-01.0 Intel Corp. 82545EM Gigabit Ethernet Controller (Fiber) <- eth0
| +-03.0 Intel Corp. 82546GB Gigabit Ethernet Controller <- eth1
| \-03.1 Intel Corp. 82546GB Gigabit Ethernet Controller
arp -n|grep -c incomplete:
73
arp -n|grep -vc incomplete:
778
iptables -nL|grep -v Chain|grep -vc source:
199
readprofile -r; sleep 60; readprofile|sort -n +2:
76 __do_softirq 0.3654
73 dst_alloc 0.4148
489 ip_rcv 0.4187
96 fib_semantic_match 0.4615
15 memset 0.4688
8 _write_lock 0.5000
37 match 0.5781
72 handle_IRQ_event 0.6429
138 fn_hash_lookup 0.7188
401 qdisc_restart 0.7371
13 _read_unlock 0.8125
28 _spin_lock_bh 0.8750
70 e1000_rx_checksum 0.8750
536 ip_rcv_finish 0.9306
33 kfree_skbmem 1.0312
248 e1000_intr 1.0333
30 _read_lock 1.8750
1198 ip_forward 1.9199
910 ip_finish_output2 1.9612
282 rt_hash_code 2.2031
286 pfifo_fast_enqueue 2.2344
36 _spin_unlock 2.2500
1657 dev_queue_xmit 2.3014
230 ip_output 2.3958
905 netif_receive_skb 2.4592
2852 e1000_clean_rx_irq 2.6213
697 nf_hook_slow 2.7227
674 e1000_alloc_rx_buffers 2.8083
228 ip_forward_finish 2.8500
187 ipt_hook 3.8958
346 kmem_cache_free 4.3250
357 pfifo_fast_dequeue 4.4625
626 local_bh_enable 4.8906
250 e1000_irq_enable 5.2083
425 kmem_cache_alloc 5.3125
3100 e1000_clean_tx_irq 5.6985
1014 nf_iterate 5.7614
284 ipt_route_hook 5.9167
701 kfree 6.2589
1063 __kmalloc 8.3047
1605 skb_release_data 10.0312
2692 eth_type_trans 11.2167
4013 __kfree_skb 15.6758
4554 alloc_skb 18.9750
20017 ipt_do_table 24.0589
10700 ip_route_input 30.3977
1341 _spin_lock 83.8125
1483 _read_unlock_bh 92.6875
3345 _read_lock_bh 104.5312
3185 _spin_unlock_irqrestore 199.0625
44402 default_idle 693.7812
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/