Strange throughput anomaly: gateway fast, behind gateway slow

From: Ralf Gerlich
Date: Thu May 05 2005 - 16:30:25 EST


Hello,

I'm currently observing pretty strange throughput anomalies. My network is connected to the Internet via a Linux masquerading gateway using a PPPoE DSL-connection. I noticed shortly that the connection does not come anywhere near the speed it should deliver, independent of the server I'm connecting to, the file I'm downloading or the box which serves as client for the download - with one exception: The gateway.

When downloading on one of the hosts behind the gateway I constantly get about 15-22kbyte/s (starting from about 30kbyte/s), while downloading the very same file from the very same host gives me around 100kbyte/s. I tried several servers (Debian mirrors in that case) and it's all the same.

I made some experiments with a specific Linux host behind the gateway. I created some traces of the traffic going via the PPP line and on the local network and the main differences I could find - aside from the source ports and the actual TCP sequence numbers - were the following:

1. The host behind the gateway advertise an MTU of 1460 in their initial SYN, which is clamped down to 1452 on the gateway. The gateway itself advertises an MTU of 1452 straight away.

2. Due to the difference in the MTU the host behind the gateway advertises an initial window of 5840 (=4*1460) while the gateway advertises an initial window of 5808 (=4*1452).

3. The data packets coming from the server when downloading on the gateway alternate between 1420 and 20 bytes payload.

4. The data packets coming from the server when downloading on the host behind the gateway all have 1420 bytes payload.

5. The delay from ACK to next data packet when downloading on the host behind the gateway alternates between about 10ms and 80-100ms, even though the pakets are all of the same size.

6. The delay between a 20byte packet and a 1420byte packet when downloading on the gateway is about 1ms and from a 1420byte packet to a 20byte packet it's around 12ms.

7. When downloading on the host behind the gateway, the window grows faster than when downloading on the gateway itself.

Changing the MTU on the host behind the gateway from 1500 to 1492 doesn't change anything on the throughput. Only the differences in (1) and (2) vanish and the transfer to the host behind the gateway also alternates between 1420 and 20 bytes payload. The alternation between high delay and low delay on 1420byte packets is still there.

I've tried this several times and it's always the same. The delay alteration seem to be induced by something outside of our network, but I can't see any differences in the pakets sent and received besides the ones I've listed.

I'm fairly desperate. I would have thought that a faster growing window was "positive" for throughput and sending small packets in between would ruin throughput. Also: Why are these 20byte-packets sent anyway?

Perhaps anybody with a bit more in-detail knowledge of the Linux TCP/IP stack and TCP/IP in general can point me where to look next.

Thank you in advance,
Ralf
-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html