Re: [PATCH v2 net-next 0/2] net: allow virtual netdevs to forward UDP L4 and fraglist GSO skbs

[Jakub Kicinski]

On Sun, 01 Nov 2020 13:16:32 +0000 Alexander Lobakin wrote:
> NETIF_F_GSO_UDP_L4 and NETIF_F_GSO_FRAGLIST allow drivers to offload
> GSO UDP L4. This works well on simple setups, but when any logical
> netdev (e.g. VLAN) is present, kernel stack always performs software
> resegmentation which actually kills the performance.
> 
> The full path in such cases is like:
> 1. Our NIC driver advertises a support for fraglists, GSO UDP L4, GSO
>    fraglists.
> 2. User enables fraglisted GRO via Ethtool.
> 3. GRO subsystem receives UDP frames from driver and merges the packets
>    into fraglisted GSO skb(s).
> 4. Networking stack queues it up for xmitting.
> 5. Virtual device like VLAN doesn't advertise a support for GSO UDP L4
>    and GSO fraglists, so skb_gso_check() doesn't allow to pass this skb
>    as is to the real driver.
> 6. Kernel then has to form a bunch of regular UDP skbs from that one and
>    pass it to the driver instead. This fallback is *extremely* slow for
>    any GSO types, but especially for GSO fraglists.
> 7. All further processing performs with a series of plain UDP skbs, and
>    the driver gets it one-by-one, despite that it supports UDP L4 and
>    fraglisted GSO.
> 
> That's not OK because:
> a) logical/virtual netdevs like VLANs, bridges etc. should pass GSO skbs
>    as is;
> b) even if the final driver doesn't support such type of GSO, this
>    software resegmenting should be performed right before it, not in the
>    middle of processing -- I think I even saw that note somewhere in
>    kernel documentation, and it's totally reasonable in terms of
>    performance.
> 
> Despite the fact that no mainline drivers currently supports fraglist
> GSO, this should and can be easily fixed by adding UDP L4 and fraglist
> GSO to the list of GSO types that can be passed-through the logical
> interfaces (NETIF_F_GSO_SOFTWARE). After this change, no resegmentation
> occurs (if a particular driver supports and advertises this), and the
> performance goes on par with e.g. 1:1 forwarding.
> The only logical netdevs that seem to be unaffected to this are bridge
> interfaces, as their code uses full NETIF_F_GSO_MASK.
> 
> Tested on MIPS32 R2 router board with a WIP NIC driver in VLAN NAT:
> 20 Mbps baseline, 1 Gbps / link speed with this patch.
> 
> Since v1 [1]:
>  - handle bonding and team drivers as suggested by Willem de Bruijn;
>  - reword and expand the introduction with the particular example. 
> 
> [1] https://lore.kernel.org/netdev/Mx3BWGop6fGORN6Cpo4mHIHz2b1bb0eLxeMG8vsijnk@xxxxxxxxxxxxxxxxxxxx


Applied, thanks!