atl1e: TSO is broken

[Anders Boström]

Short desription
----------------

TCP Segmentation Offload (TSO) result in broken IPv4-packets sent out
from Atheros AR8121/AR8113/AR8114 with the atl1e driver.


Work around
-----------

Turn off TSO. Requires 2.6.33-rc1 or later.


Long desription
----------------

When I run NFS over TCP (default options) and read large files from a
server with Atheros AR8121/AR8113/AR8114 Ethernet chip, I only get
~25Mbyte/s performance. I get ~5000 retransmitted packets per GByte
data, according to RetransSegs in /proc/net/snmp . wireshark in the
client show that the server send out a sequence of frames. All but the
last one are 1500 bytes IP-packets. The last one is shorter, but the
IP-header still say 1500 byte. The client then requests retransmit,
and the retransmitted frame arrives with correct IP-header.

If I mount NFS using UDP instead, performance is ~110Mbyte/s.

TCP Segmentation Offload (TSO) is default enabled in the atl1e
Ethernet-driver. When I run a patched 2.6.30.10, enabling ethtool to
turn off TSO (using ac936929092dc6a5409b627c4c67305ab9b626b3 by Ben
Hutchings), and turn off TSO, the problem disappears. Performance is
~110Mbyte/s and no broken IP-packets arrive.


Capture of 146-byte Ethernet frame with bad IP-header:

No.     Time        Source                Destination           Protocol Info
  98329 11.034129   flash.netinsight.se   sid.netinsight.se     RPC      Continuation

Frame 98329 (146 bytes on wire, 146 bytes captured)
    Arrival Time: Jan 15, 2010 13:35:16.224491000
    [Time delta from previous captured frame: 0.000009000 seconds]
    [Time delta from previous displayed frame: 0.000009000 seconds]
    [Time since reference or first frame: 11.034129000 seconds]
    Frame Number: 98329
    Frame Length: 146 bytes
    Capture Length: 146 bytes
    [Frame is marked: False]
    [Protocols in frame: eth:ip:tcp:rpc]
    [Coloring Rule Name: TCP]
    [Coloring Rule String: tcp]
Ethernet II, Src: AsustekC_ae:69:6d (00:26:18:ae:69:6d), Dst: sid.netinsight.se (00:18:f3:52:22:3f)
Internet Protocol, Src: flash.netinsight.se (10.100.0.88), Dst: sid.netinsight.se (10.100.1.25)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
    Total Length: 1500
    Identification: 0x331e (13086)
    Flags: 0x02 (Don't Fragment)
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0xebc5 [correct]
    Source: flash.netinsight.se (10.100.0.88)
    Destination: sid.netinsight.se (10.100.1.25)
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: accessbuilder (888), Seq: 93989617, Ack: 516997, Len: 80
Remote Procedure Call

0000  00 18 f3 52 22 3f 00 26 18 ae 69 6d 08 00 45 00   ...R"?.&..im..E.
0010  05 dc 33 1e 40 00 40 06 eb c5 0a 64 00 58 0a 64   ..3.@.@....d.X.d
0020  01 19 08 01 03 78 a4 07 57 23 e6 50 1f 1b 80 10   .....x..W#.P....
0030  01 f5 dd 8d 00 00 01 01 08 0a 05 28 ca 7e 38 93   ...........(.~8.
0040  67 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00   g...............
0050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0060  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0070  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0080  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
0090  00 00                                             ..



Software info:

I've tested with Debian 2.6.26 (stable) and 2.6.30 (testing), as well
as 2.6.30.10 from kernel.org. Same result.

Hardware info:

lspci -vvv:

03:00.0 Ethernet controller: Attansic Technology Corp. Atheros AR8121/AR8113/AR8114 PCI-E Ethernet Controller (rev b0)
        Subsystem: ASUSTeK Computer Inc. Device 831c
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 27
        Region 0: Memory at fbfc0000 (64-bit, non-prefetchable) [size=256K]
        Region 2: I/O ports at ec00 [size=128]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
                Address: 00000000fee0f00c  Data: 4189
        Capabilities: [58] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag- AttnBtn+ AttnInd+ PwrInd+ RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 unlimited, L1 unlimited
                        ClockPM- Suprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [180] Device Serial Number ff-18-26-00-6d-69-ae-ff
        Kernel driver in use: ATL1E
        Kernel modules: atl1e

ethtool -i eth0:

driver: ATL1E
version: 1.0.0.7-NAPI
firmware-version: L1e
bus-info: 0000:03:00.0


/ Anders
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/