Re: PROBLEM: invalid udp checksum with ip6gre-in-udp

[Benoît Monin]

Hi again,

05/09/2024 Benoît Monin :
> Hi all,
> 
> I am having issue with GRE-in-UDP (GRE with fou encapsulation) over 
> IPv6: the outer UDP checksum is only valid if an inner checksum is 
> present and valid. This problem is only present over IPv6, not IPv4, 
> and reproducible with different kernel versions and cpu architecture.
> 
> Here is the test setup I used:
> 
>     +----------+                                    +------------+
>     |          | fd00::2/64              fd00::1/64 |            |
>     |  tester  | 10.0.0.33/24          10.0.0.11/24 |    DUT     |
>     |          |------------------------------------|            |
>     +----------+                                    +------------+
> 
> Two machines are connected with an ethernet cable, and each side is 
> setup with an ipv4 and an ipv6 address.
> 
> On the device under test, two GRE-in-UDP tunnels are setup, one over 
> ipv6 and one over ipv4, aimed at the tester. Each tunnel is configured 
> with an ipv4 address:
> 
>     modprobe -a fou fou6
> 
>     ip link add gre6 type ip6gre local fd00::1 remote fd00::2 encap fou encap-sport 1234 encap-dport 4567
>     ip link set up gre6
>     ip address add 172.20.6.1/24 dev gre6
> 
>     ip link add gre4 type gre local 10.0.0.11 remote 10.0.0.33 encap fou encap-sport 3456 encap-dport 6789 encap-csum
>     ip link set up gre4
>     ip address add 172.20.4.1/24 dev gre4
> 
> On the tester, no setup is done, it is only running tcpdump to capture 
> the traffic emitted by the DUT.
> 
> The following commands are used on the device under test to send 
> packets via the tunnels:
> 
>     ping -c1 -W0.1 172.20.6.2
>     ./send_udp 172.20.6.2 5555 "ip6gre-in-udp with inner udp csum"
>     SO_NO_CHECK=1 ./send_udp 172.20.6.2 5555 "ip6gre-in-udp without inner udp csum"
>     ping -c1 -W0.1 172.20.4.2
>     ./send_udp 172.20.4.2 5555 "gre-in-udp with inner udp csum"
>     SO_NO_CHECK=1 ./send_udp 172.20.4.2 5555 "gre-in-udp without inner udp csum"
> 
> Three packets are sent in each GRE tunnels :
> * One ICMP echo request
> * One UDP packet with a valid checksum
> * One UDP packet with the checksum set to 0
> 
> Here is a link to the pcap containing the six packets generated by the 
> previous commands:
> https://onaip.mooo.com/pub/tmp/bug_ip6gre-in-udp.pcap
> 
> Some details about the captured packets:
> IP6 fd00::1 > fd00::2: 1234 > 4567: [bad udp cksum 0xfa75 -> 0xe680!]
> IP6 fd00::1 > fd00::2: 1234 > 4567: [udp sum ok]
> IP6 fd00::1 > fd00::2: 1234 > 4567: [bad udp cksum 0xfa62 -> 0xb57c!]
> IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> IP 10.0.0.11.3456 > 10.0.0.33.6789: [udp sum ok]
> 
> For the tunnel over ipv6, only the UDP packet sent with a valid 
> checksum get encapsulated with a valid checksum. For the ping and the 
> UDP packet with a zero checksum, the outer UDP checksum matches the 
> partial checksum of the pseudo-header.
> 
> For ipv4, the UDP checksum of the encapsulation are all valid.
> 
> The device under test used for the capture is an x86-64 machine with a 
> realtek ethernet adapter (r8169 driver) running a 6.10.7 kernel.
> 
> The problem was also seen on an arm64 board (freescale ls1046) with 
> dpaa ethernet driver running a 4.14 kernel. on this hardware, the 
> packets that would have an invalid checksum are not emitted and the tx 
> error counter of the ethernet interface increases.
> 
> In all cases, disabling hardware checksumming for ipv6 with ethtool can 
> be used as a work-around:
> 
>     ethtool -K eth0 tx-checksum-ipv6 off
> 
> This and the partial checksum value seems to point to an error in the 
> handling of hardware checksumming in the particular case of fou6 
> encapsulation, but I have not been able to figure out what could be 
> causing it.
> 
> Did I miss a configuration parameter for ip6gre-in-udp? Any advise on 
> how to debug that would be appreciated.
> 
I did some more digging with 6.11-rc7, and it is not a problem in the 
common code or udp encapsulation, I just got "lucky" with my tests...

First in fou6.c, fou6_build_udp constructs the upd header and calls 
udp6_set_csum. If the inner packet has a valid checksum, the outer udp 
get computed by reusing it, otherwise the partial checksum is set.

Next the outer ipv6 is built in ip6tunnel.c:ip6_tnl_xmit. With the 
default value of encaplimit (4), an destination options header is 
inserted to pass that value. This means that ipv6_hdr(skb)->nexthdr is 
set to 60 (NEXTHDR_DEST), not 17 (IPPROTO_UDP).

So when sending a packet without a valid checksum in a ip6gre-in-udp, 
we pass a skb with a partial udp checksum and an ipv6 header with an 
extension to ndo_start_xmit.

Finally, what are the odds of testing two different hardware and 
finding two similar bugs?

For the LS1046 platform, the Tx checksum is done in 
dpaa_enable_tx_csum. For ipv6, the layer 4 protocol is extracted from 
the skb with l4_proto = ipv6h->nexthdr. Since the value does not match 
IPPROTO_UDP nor IPPROTO_TCP, the function errors out and the packet is 
discarded.

For the PC with a realtek 8169 card, the code is similar in 
rtl8169_tso_csum_v2, with ip_protocol = ipv6_hdr(skb)->nexthdr. The 
error case then triggers a WARN_ON_ONCE(1) and the packet is still sent 
on the wire with a partial udp checksum.

There seems to be quite a few places in drivers/net/ethernet that use 
ipv6_hdr(skb)->nexthdr as the IP protocol, with only some of them 
calling ipv6_skip_exthdr afterward to take care of the extension 
headers. So my proposal is to add a small helper (ipv6_protocol?) and 
fix the drivers where needed.

I'll try to come up with a patch set, unless someone has a better idea.

-- 
Benoît