Re: many rx packets dropped since 2.6.37

From: Eric Dumazet
Date: Thu Apr 26 2012 - 10:07:40 EST

On Thu, 2012-04-26 at 15:53 +0200, Chris wrote:
> 2012/4/25 Eric Dumazet <eric.dumazet@xxxxxxxxx>:
> > On Wed, 2012-04-25 at 16:38 +0200, Chris wrote:
> >> 2012/4/25 Eric Dumazet <eric.dumazet@xxxxxxxxx>:
> >> >> Why do we see dropped packets only in virtual environments and bonded
> >> >> network interfaces and not on real network interfaces?
> >> >
> >> > This has to be investigated, eventually.
> >>
> >> Do you have the opportunity to test it? Just install fedora 16 (linux
> >> 3.3) or ubuntu 12.04 (linux 3.2) as guest under kvm and use as virtual
> >> network card virtio_net or e1000.
> >>
> >> You will see many lost rx packets and i have no idea why???
> >
> >
> > You can use drop_monitor / dropwatch to figure out
> >
> > https://fedorahosted.org/dropwatch/
> >
> > http://prefetch.net/blog/index.php/2011/07/11/using-netstat-and-dropwatch-to-observe-packet-loss-on-linux-servers/
>
> Hello Eric,
>
> Okay this is a Fedora-16 VM under KVM with 4 minutes uptime only and dropwatch!
>
> 349 dropped packets in 4 minutes uptime??
>

yes, apparently.

Mostely because of :

610 with invalid addresses
71 dropped because of missing route
and IPReversePathFilter: 82

It seems you have some work to do on your network config

> [root@test ~]# uname -r
> 3.3.2-6.fc16.x86_64
> [root@test ~]# dropwatch -l kas
> Initalizing kallsyms db
> dropwatch> start
> Enabling monitoring...
> Kernel monitoring activated.
> Issue Ctrl-C to stop monitoring
> 1 drops at netlink_unicast+1bc

this is a false positive, bugfix here :
http://git.kernel.org/?p=linux/kernel/git/davem/net-next.git;a=commitdiff;h=bfb253c9b277acd9f85b1886ff82b1dd5fbff2ae

> 3 drops at ip_rcv_finish+f0
> 1 drops at __netif_receive_skb+583
> 1 drops at ip_rcv+d3
> 9 drops at ip_rcv_finish+f0
> 9 drops at ip_rcv_finish+f0
> 1 drops at __netif_receive_skb+583
> 6 drops at ip_rcv_finish+f0
> 1 drops at nf_hook_slow+12b

do you have drops at iptables level ?

> 4 drops at ip_rcv_finish+f0
> 1 drops at __netif_receive_skb+583
> 4 drops at ip_rcv_finish+f0
> 1 drops at nf_hook_slow+12b
> 3 drops at ip_rcv_finish+f0
> 3 drops at nf_hook_slow+12b
> 2 drops at __netif_receive_skb+583
> 1 drops at ip6_mc_input+1f6
> 7 drops at ip_rcv_finish+f0
> 1 drops at ip_rcv_finish+f0
> 1 drops at netlink_unicast+1bc
> 1 drops at skb_queue_purge+20
> 2 drops at __netif_receive_skb+583
> 1 drops at ip_rcv_finish+f0
> 1 drops at nf_hook_slow+12b
> 1 drops at ip_rcv_finish+f0
> 1 drops at __netif_receive_skb+583
> 19 drops at ip_rcv+d3
> 2 drops at ip_rcv_finish+f0
> 37 drops at ip_rcv+d3
> 2 drops at __netif_receive_skb+583
> 43 drops at ip_rcv+d3
> 5 drops at ip_rcv_finish+f0
> 1 drops at __netif_receive_skb+583
> 1 drops at nf_hook_slow+12b
> 1 drops at ip6_mc_input+1f6
> 7 drops at ip_rcv_finish+f0
> 39 drops at ip_rcv+d3
> 2 drops at ip6_mc_input+1f6
> 1 drops at netlink_unicast+1bc
> 1 drops at skb_queue_purge+20
> 29 drops at ip_rcv+d3
> 7 drops at ip_rcv_finish+f0
> 2 drops at __netif_receive_skb+583
> 2 drops at ip6_mc_input+1f6
> 54 drops at ip_rcv+d3
> 3 drops at __netif_receive_skb+583
> 2 drops at nf_hook_slow+12b
> 3 drops at ip_rcv_finish+f0
> 1 drops at ip6_mc_input+1f6
> 36 drops at ip_rcv+d3
> 6 drops at ip_rcv_finish+f0
> 2 drops at __netif_receive_skb+583
> 1 drops at nf_hook_slow+12b
> 1 drops at icmpv6_rcv+305
> 37 drops at ip_rcv+d3
> 1 drops at __netif_receive_skb+583
> 2 drops at ip_rcv_finish+f0
> 4 drops at ip_rcv_finish+f0
> 39 drops at ip_rcv+d3
> 2 drops at __netif_receive_skb+583
> 1 drops at netlink_unicast+1bc
> 1 drops at skb_queue_purge+20
> 1 drops at nf_hook_slow+12b
> 38 drops at ip_rcv+d3
> 3 drops at __netif_receive_skb+583
> 2 drops at ip6_mc_input+1f6
> 1 drops at ip_rcv_finish+f0
> 46 drops at ip_rcv+d3
> 1 drops at __netif_receive_skb+583
> 1 drops at ip_rcv_finish+f0
> ^CGot a stop message
> dropwatch> exit
> Shutting down ...
> [root@test ~]# ifconfig | grep dropped
> RX packets:161562 errors:0 dropped:349 overruns:0 frame:0
> TX packets:2140 errors:0 dropped:0 overruns:0 carrier:0
> RX packets:4 errors:0 dropped:0 overruns:0 frame:0
> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
> [root@test ~]# uptime
> 15:47:00 up 4 min, 3 users, load average: 0.01, 0.04, 0.03
> [root@test ~]# netstat -s
> Ip:
> 1553 total packets received
> 610 with invalid addresses
> 0 forwarded
> 0 incoming packets discarded
> 345 incoming packets delivered
> 478 requests sent out
> 71 dropped because of missing route
> Icmp:
> 4 ICMP messages received
> 0 input ICMP message failed.
> ICMP input histogram:
> echo requests: 2
> echo replies: 2
> 4 ICMP messages sent
> 0 ICMP messages failed
> ICMP output histogram:
> echo request: 2
> echo replies: 2
> IcmpMsg:
> InType0: 2
> InType8: 2
> OutType0: 2
> OutType8: 2
> Tcp:
> 1 active connections openings
> 2 passive connection openings
> 0 failed connection attempts
> 0 connection resets received
> 2 connections established
> 2353 segments received
> 2096 segments send out
> 0 segments retransmited
> 0 bad segments received.
> 0 resets sent
> Udp:
> 51 packets received
> 0 packets to unknown port received.
> 0 packet receive errors
> 51 packets sent
> 0 receive buffer errors
> 0 send buffer errors
> UdpLite:
> TcpExt:
> 1 TCP sockets finished time wait in fast timer
> 3 delayed acks sent
> 2 packets directly queued to recvmsg prequeue.
> 2 packets directly received from prequeue
> 2009 packets header predicted
> 165 acknowledgments not containing data received
> IPReversePathFilter: 82
> IpExt:
> InMcastPkts: 13
> InBcastPkts: 60
> InOctets: 146431
> OutOctets: 63310
> InMcastOctets: 416
> InBcastOctets: 12060
>
> --
> Chris


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/