Dropped packets on lo

From: Steve Hill
Date: Mon Nov 29 2010 - 10:34:05 EST

I have a Squid proxy server which talks to an ICAP server running on the same machine. Squid talks to the ICAP server through the loopback interface ( or ::1 - it doesn't seem to make any difference whether I'm using IPv6 or IPv4).

Everything works fine for a while (anything from a few hours to a few weeks), but every so often the system appears to start dropping packets on these Squid->ICAP connections. The connections start going very slow and flakey and a tcpdump shows lots of TCP retransmissions, which also appears to include the SYN packets of new connections being established between Squid and the ICAP server.

The problem continues to affect the connections between Squid and the ICAP server indefinately until either Squid or the ICAP server is restarted - doesn't seem to matter which. Connections between other processes seem to be unaffected; even the TCP connection between the ICAP server and the LDAP server (also running on the same machine) seems to be fine.

The machine doesn't seem to be under undue load at the time of the problem. This problem has been observed on several similar systems, whilst seemingly not affecting other similarly configured machines. All these systems are running CentOS 5.5 with the 2.6.18-194.8.1.el5 x86_64 kernel.

While the problem is ongoing, netstat also shows a few connections in the ESTABLISHED state that aren't owned by any process (the "PID/Program name" column just shows "-"). All the other ESTABLISHED connections show no packets in the rx/tx queues.

I'm at a loss to understand what is happening - the reason I am posting to this mailing list is because I can't see any reason why a misbehaving userland process would cause the dropped loopback packets that I am seeing - surely this must be a problem within the network stack itself? But at the same time, I can't understand why a problem deep within the stack would only affect the connections between these two processes whilst leaving everything else working ok.

Any help or advice that can be offered would be greatfully received - I have run out of ideas for further debugging I can do. I can provide tcpdump logs, etc.


- Steve
xmpp:steve@xxxxxxxxxxx sip:steve@xxxxxxxxxxx http://www.nexusuk.org/

To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html