dst cache overflow

From: Oleksandr Samoylyk
Date: Mon Oct 13 2008 - 10:04:47 EST

Next message: Christoph Lameter: "Re: [PATCH 1/1] hugetlbfs: handle pages higher order than MAX_ORDER"
Previous message: Alan Cox: "Re: I have returned!"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Dear community,

I've a problem of unexpected loosing of network connection on a server running GNU/Linux.

It's a 8-core / 8 GB RAM PPTP "aggregator" with about 2500 sessions and 200 Mb/s or 30 kpps of Internet traffic.

There are two GigE Intel NICs (Intel(R) PRO/1000 Network Driver - version 7.3.20-k2-NAPI).

I've attached their IRQs to different CPUs with smp_affinity.

TSO is off.

Kernel version: linux-2.6.24.3 (server image from ubuntu hardy).

I'm booting with:
noapic acpi=off panic=5 rhash_entries=1048575

I got the following in logs:

Oct 7 22:26:50 linux kernel: [ 0.000000] CPU 2:
Oct 7 22:26:50 linux kernel: [ 0.000000] Modules linked in: oprofile af_packet xt_tcpmss act_police cls_u32 sch_sfq sch_ingress sch_htb xt_multiport xt_TCPMSS xt_state
xt_limit xt_tcpudp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables pptp pppox ppp_generic slhc parport_pc lp parport loop i
TCO_wdt iTCO_vendor_support pcspkr i5000_edac shpchp edac_core pci_hotplug evdev ext3 jbd mbcache sg sr_mod cdrom sd_mod ata_piix pata_acpi aic94xx libsas scsi_transport_sas a
ta_generic e1000 libata scsi_mod fbcon tileblit font bitblit softcursor fuse
Oct 7 22:26:50 linux kernel: [ 0.000000] Pid: 29, comm: events/2 Not tainted 2.6.24-19-server #1
Oct 7 22:26:50 linux kernel: [ 0.000000] RIP: 0010:[libsas:_spin_lock_irqsave+0x15/0x30] [libsas:_spin_lock_irqsave+0x15/0x30] _spin_lock_irqsave+0x15/0x30
Oct 7 22:26:50 linux kernel: [ 0.000000] RSP: 0018:ffff81022568de18 EFLAGS: 00000282
Oct 7 22:26:50 linux kernel: [ 0.000000] RAX: 0000000000000282 RBX: ffff810164d62400 RCX: 0000000000000000
Oct 7 22:26:50 linux kernel: [ 0.000000] RDX: 00000000000e4e67 RSI: 00000000124839fe RDI: ffff810164d626b4
Oct 7 22:26:50 linux kernel: [ 0.000000] RBP: ffffffff802345b3 R08: 0000000000000000 R09: 0000000000000000
Oct 7 22:26:50 linux kernel: [ 0.000000] R10: 0000000000000000 R11: ffff8101f04d5d00 R12: ffff81022568ddc0
Oct 7 22:26:50 linux kernel: [ 0.000000] R13: 0000000000000003 R14: 0000000000000286 R15: 0000000000000001
Oct 7 22:26:50 linux kernel: [ 0.000000] FS: 0000000000000000(0000) GS:ffff810228001b80(0000) knlGS:0000000000000000
Oct 7 22:26:50 linux kernel: [ 0.000000] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Oct 7 22:26:50 linux kernel: [ 0.000000] CR2: 00007f54a9cd2000 CR3: 0000000000201000 CR4: 00000000000006e0
Oct 7 22:26:50 linux kernel: [ 0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 7 22:26:50 linux kernel: [ 0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 7 22:26:50 linux kernel: [ 0.000000]
Oct 7 22:26:50 linux kernel: [ 0.000000] Call Trace:
Oct 7 22:26:50 linux kernel: [ 0.000000] [pptp:skb_dequeue+0x21/0x3d0] skb_dequeue+0x21/0x80
Oct 7 22:26:50 linux kernel: [ 0.000000] [pptp:do_buf_work+0x38/0x150] :pptp:do_buf_work+0x38/0x150
Oct 7 22:26:50 linux kernel: [ 0.000000] [pptp:_buf_work+0x0/0x20] :pptp:_buf_work+0x0/0x20
Oct 7 22:26:50 linux kernel: [ 0.000000] [run_workqueue+0xcc/0x170] run_workqueue+0xcc/0x170
Oct 7 22:26:50 linux kernel: [ 0.000000] [worker_thread+0x0/0x110] worker_thread+0x0/0x110
Oct 7 22:26:50 linux kernel: [ 0.000000] [worker_thread+0x0/0x110] worker_thread+0x0/0x110
Oct 7 22:26:50 linux kernel: [ 0.000000] [worker_thread+0xa3/0x110] worker_thread+0xa3/0x110
Oct 7 22:26:50 linux kernel: [ 0.000000] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30
Oct 7 22:26:50 linux kernel: [ 0.000000] [worker_thread+0x0/0x110] worker_thread+0x0/0x110
Oct 7 22:26:50 linux kernel: [ 0.000000] [worker_thread+0x0/0x110] worker_thread+0x0/0x110
Oct 7 22:26:50 linux kernel: [ 0.000000] [kthread+0x4b/0x80] kthread+0x4b/0x80
Oct 7 22:26:50 linux kernel: [ 0.000000] [child_rip+0xa/0x12] child_rip+0xa/0x12
Oct 7 22:26:50 linux kernel: [ 0.000000] [kthread+0x0/0x80] kthread+0x0/0x80
Oct 7 22:26:50 linux kernel: [ 0.000000] [child_rip+0x0/0x12] child_rip+0x0/0x12
Oct 7 22:26:50 linux kernel: [ 0.000000]
Oct 7 22:26:51 linux kernel: [ 0.000000] NETDEV WATCHDOG: eth0: transmit timed out
Oct 7 22:26:53 linux kernel: [ 0.000000] printk: 19 messages suppressed.
Oct 7 22:26:53 linux kernel: [ 0.000000] dst cache overflow
Oct 7 22:26:56 linux kernel: [ 0.000000] NETDEV WATCHDOG: eth0: transmit timed out
Oct 7 22:26:58 linux kernel: [ 0.000000] printk: 19 messages suppressed.
Oct 7 22:26:58 linux kernel: [ 0.000000] dst cache overflow
Oct 7 22:27:01 linux kernel: [ 0.000000] NETDEV WATCHDOG: eth0: transmit timed out
Oct 7 22:27:02 linux kernel: [ 0.000000] CPU 2:

The server lost network connectivity until reboot.

I guess it's due to "dst cache overflow".

Some of custom sysctl variables:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.core.netdev_max_backlog = 4096
net.ipv4.conf.default.arp_filter = 1
net.ipv4.ip_default_ttl = 255
net.ipv4.tcp_max_syn_backlog = 4096

net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0

net.nf_conntrack_max = 1743087
net.ipv4.netfilter.ip_conntrack_max = 1743087
net.ipv4.tcp_max_orphans = 131072

net.ipv4.netfilter.ip_conntrack_generic_timeout = 300
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 10
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 5
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 30
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 20
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 30

net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 16384
net.ipv4.neigh.default.gc_thresh3 = 32768

net.ipv4.route.max_size = 1048576
net.ipv4.route.gc_thresh = 131072
net.ipv4.route.gc_elasticity = 4
net.ipv4.route.gc_interval = 1
net.ipv4.route.secret_interval = 3600

fs.file-max = 2097152

kernel.pid_max = 4194303

net.core.somaxconn = 640000

vm.min_free_kbytes = 65536

kernel.panic = 5

vm.swappiness = 0

What can I do to prevent such situations?

Any advice will be appreciate. :)

Thanks!

--
Oleksandr Samoylyk
OVS-RIPE
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Christoph Lameter: "Re: [PATCH 1/1] hugetlbfs: handle pages higher order than MAX_ORDER"
Previous message: Alan Cox: "Re: I have returned!"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]