Re: [PATCH] net: fix csum calculation for encapsulated packets
From: 沈安琪(凛玥)
Date: Thu Aug 22 2024 - 23:28:58 EST
在 8/22/24 下午6:05, Paolo Abeni 写道:
On 8/19/24 13:17, 沈安琪(凛玥) wrote:
This commit fixes the issue that when a packet is encapsulated, such as
sending through a UDP tunnel, the outer TCP/UDP checksum is not
correctly recalculated if (1) checksum has been offloaded to hardware
and (2) encapsulated packet has been NAT-ed again, which causes the
packet being dropped due to the invalid outer checksum.
Previously, when an encapsulated packet met some NAT rules and its
src/dst ip and/or src/dst port has been modified,
inet_proto_csum_replace4 will be invoked to recalculated the outer
checksum. However, if the packet is under the following condition: (1)
checksum offloaded to hardware and (2) NAT rule has changed the src/dst
port, its outer checksum will not be recalculated, since (1)
skb->ip_summed is set to CHECKSUM_PARTIAL due to csum offload and (2)
pseudohdr is set to false since port number is not part of pseudo
header.
I don't see where nat is calling inet_proto_csum_replace4() with
pseudohdr == false: please include more detailed description of the
relevant setup (ideally a self-test) or at least a backtrace leading
to the issue.
The relevant setup we found this issue was:
1. we setup a VXLAN tunnel and set a MASQUERADE rule with --random-fully
enabled on client side (tx side).
2. we enabled. NIC checksum offload.
We used the following bpftrace script to get the call trace:
bpftrace -o trace.log -e '#include <linux/skbuff.h>
#include <linux/net.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <linux/if_ether.h>
#include <linux/types.h>
#include <net/sock.h>
#include <net/route.h>
#include <net/dst.h>
kprobe:inet_proto_csum_replace4
{
$skb = (struct sk_buff *)arg1;
$ip_hdr = (struct iphdr *)($skb->head + $skb->network_header);
$proto = $ip_hdr->protocol;
if ($proto == 17) {
$udp_hdr = (struct udphdr *)($skb->head + $skb->transport_header);
printf("udp check: 0x%04x; \n", $udp_hdr->check);
printf("skb ip_summed: %d, skb encapsulation: %d, skb
encap_csum_hdr: %d; \n", $skb->ip_summed, $skb->encapsulation,
$skb->encap_hdr_csum);
printf("from: 0x%08x, to: 0x%08x; \n", arg2, arg3);
printf("pseudohdr: %d; \n", arg4);
printf("enter kprobe:inet_proto_csum_replace4 : %s\n\n", kstack);
}
}
'
And get the following call trace:
udp check: [...];
skb ip_summed: 3, skb encapsulation: 1, skb encap_csum_hdr: 0;
from: ...[old src ip], to: ...[NAT-ed src ip];
pseudohdr: 1;
enter kprobe:inet_proto_csum_replace4 :
inet_proto_csum_replace4+1
l4proto_manip_pkt+1166
nf_nat_ipv4_manip_pkt+90
nf_nat_manip_pkt+141
nf_nat_ipv4_out+76
nf_hook_slow+57
ip_output+225
iptunnel_xmit+356
vxlan_xmit_one+3184
vxlan_xmit+823
xmit_one.constprop.0+149
dev_hard_start_xmit+80
__dev_queue_xmit+962
ip_finish_output2+417
ip_send_skb+56
udp_send_skb+337
udp_sendmsg+2404
sock_sendmsg+51
____sys_sendmsg+487
___sys_sendmsg+117
__sys_sendmsg+89
do_syscall_64+45
entry_SYSCALL_64_after_hwframe+68
udp check: [...];
skb ip_summed: 3, skb encapsulation: 1, skb encap_csum_hdr: 0;
from: ...[old src port], to: ...[new src port];
pseudohdr: 0;
enter kprobe:inet_proto_csum_replace4 :
inet_proto_csum_replace4+1
l4proto_manip_pkt+1195
nf_nat_ipv4_manip_pkt+90
nf_nat_manip_pkt+141
nf_nat_ipv4_out+76
nf_hook_slow+57
ip_output+225
iptunnel_xmit+356
vxlan_xmit_one+3184
vxlan_xmit+823
xmit_one.constprop.0+149
dev_hard_start_xmit+80
__dev_queue_xmit+962
ip_finish_output2+417
ip_send_skb+56
udp_send_skb+337
udp_sendmsg+2404
sock_sendmsg+51
____sys_sendmsg+487
___sys_sendmsg+117
__sys_sendmsg+89
do_syscall_64+45
entry_SYSCALL_64_after_hwframe+68
We trace the current implementation and found in __udp_manip_pkt
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/netfilter/nf_nat_proto.c#n57
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/include/net/checksum.h#n164
will invoke inet_proto_csum_replace4 with pseudohdr as false.
This leads to the outer TCP/UDP checksum invalid since it does
not change along with the port number change.
In this commit, another condition has been added to recalculate outer
checksum: if (1) the packet is encapsulated, (2) checksum has been
offloaded, (3) the encapsulated packet has been NAT-ed to change port
number and (4) outer checksum is needed, the outer checksum for
encapsulated packet will be recalculated to make sure it is valid.
Please add a suitable fix tag.
Thanks for notifying this. We will add it in the next version of this patch.
Signed-off-by: Anqi Shen <amy.saq@xxxxxxxxxxxx>
---
net/core/utils.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/net/core/utils.c b/net/core/utils.c
index c994e95172ac..d9de60e9b347 100644
--- a/net/core/utils.c
+++ b/net/core/utils.c
@@ -435,6 +435,8 @@ void inet_proto_csum_replace4(__sum16 *sum,
struct sk_buff *skb,
*sum = ~csum_fold(csum_add(csum_sub(csum_unfold(*sum),
(__force __wsum)from),
(__force __wsum)to));
+ else if (skb->encapsulation && !!(*sum))
+ csum_replace4(sum, from, to);
This looks incorrect for a csum partial value, and AFAICS the nat
caller has already checked for !!(*sum).
Thanks,
Paolo