TCP checksum change in RPC replies within XEN, NFS lockup (SLES10)

[Ulrich Windl]

Hello,

my apologies for not being sure whom to tell this problem, but it is very strange. 
Let me tell the story:

I'm using XEN (3.0.2) with SLES10 (x86_64, SunFire X4100). On one machine I have 
three virtual machines ("DomU") that are very identically configured (SLES10 
x86_64 also). There is also a SLES9 (i386) acting as a multi-homed NFS server.

I can mount and access a read-only NFS filesystem on the server from Dom0 
(hypervisor), and from two of the three DomUs without any problem, but from the 
third DomU mount hangs and is unkillable (except kill -9). This is how I started 
to debug the problem.

To make things short: I haven't found the solution, but a some problems: Running 
tcpdump/etheral on the client (inside DomU), and on the NFS server, I found out 
that a significant number (almost all) of RPC reply packets have an invalid TCP 
cjhecksum on the NFS server, but not on the NFS client. When actually comparing 
the packets, I found that only the checksum is different. Example:

--- nfs-client9.txt	2006-11-29 12:56:59.176133729 +0100
+++ nfs-server8.txt	2006-11-29 12:56:25.812623453 +0100
@@ -1,10 +1,10 @@
 No.     Time            Source                Destination           Protocol Info
-      9 15:10:15.488963 132.199.176.153       132.199.177.13        Portmap  V2 
DUMP Reply (Call In 7)[Unreassembled Packet]
+      8 15:10:15.497059 132.199.176.153       132.199.177.13        Portmap  V2 
DUMP Reply (Call In 6)[Unreassembled Packet [incorrect TCP checksum]]
 
 0000  00 16 3e f3 45 0d 00 c0 9f 27 44 a6 08 00 45 00   ..>.E....'D...E.
 0010  01 c4 d0 3f 40 00 40 06 fd be 84 c7 b0 99 84 c7   ...?@.@.........
 0020  b1 0d 00 6f 94 48 89 33 9b af 3f e4 5a 65 80 18   ...o.H.3..?.Ze..
-0030  16 a0 27 8a 00 00 01 01 08 0a 5a a1 4b 92 01 2f   ..'.......Z.K../
+0030  16 a0 6c ec 00 00 01 01 08 0a 5a a1 4b 92 01 2f   ..l.......Z.K../
 0040  a9 da 00 00 01 8c 65 e9 c4 df 00 00 00 01 00 00   ......e.........
 0050  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00   ................
 0060  00 01 00 01 86 a0 00 00 00 02 00 00 00 06 00 00   ................


I' NOT saysing that _all_ TCP checksums are bad, but significantly those RPC reply 
packets seem to be affected. OK so far.

I don't know why the NFS mount is actually hanging, but the last packet exchange 
seems to be:

Server sends ACK to RPC reply with bad checksum:
Transmission Control Protocol, Src Port: nfs (2049), Dst Port: 1023 (1023), Seq: 
2306928188, Ack: 1069064470, Len: 0

Client receives:

Transmission Control Protocol, Src Port: nfs (2049), Dst Port: 1023 (1023), Seq: 
2306928188, Ack: 1069064470, Len: 0

Some time later I see the client sending an NFS GETATTR packet, but that's 
probably when the kill came in (31 seconds later).

The other odd thing is that the multihomes NFS server has a route to 132.199.0.0 
for both ethernet interfaces, but only one of those, eth0, has IP 132.199.176.153.
However when an ARP request is sent for 132.199.176.153, there are two ansers:

ARP 132.199.176.153 is at 00:c0:9f:27:44:a6
ARP 132.199.176.153 is at 00:02:b3:d9:91:a7

Only the first one is correct. However that problem may be unrelated.

Back to the issue, I doubt that XEN will just overwrite the TCP checksum of some 
specific RPC packets. Personally I could image is much more that there is some 
problem in the RPC processing that might cause this. Sorry for the poor problem 
description.

Just in case, the Novell/SUSE kernel versions are:
Client: 2.6.16.21-0.25-xen
Server: 2.6.5-7.282-default

Upon request I could provide the packet files as well as two PDFs that show the 
packet flow.

Regards,
Ulrich
P.S: I'm subscribed to xen-users, but not to the kernel list, so maybe CC: me for 
kernel-list replies. Thanks!

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/