[PATCH net-next 0/7] net: devmem: size TX net_iovs from the dmabuf layout

[Stanislav Fomichev]

TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
68KB, which makes BIG TCP ineffective for devmem TX.

This series picks the TX net_iov size from the dmabuf sg table instead:
the largest power of 2, up to 2MB, which aligns every DMA segment. With
a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
no UAPI change since TX addresses are dmabuf offsets.

TCP still bounds skbs by size_goal, so frags get larger, skbs don't.
A single frag can now be as large as the skb, which matters for
NETMEM_TX drivers with tso_max_size above 64KB:
- mlx5: 32-bit length per data segment
- gve: splits frags into 16KB descriptors
- fbnic: keeps the default 64KB tso_max_size
- bnxt: 16-bit BD length, devmem TX frags are capped at 32KB by a new
  per-device limit

Tested on two mlx5 400Gb/s hosts with kperf (udmabuf, 32KB RX
net_iovs, 2MB hugepage-backed TX buffer), series backported to a 7.1
kernel on the sender only. 4 flows, MTU 4200, 6 x 60s runs, medians
(CPU over the 3 runs where every flow got its own CPUs):

                              before    after
  frags per skb (sender)       16-17      1-2
  throughput (Gb/s)            389.6    389.6
  sender sendmsg CPU (sys)     12.0%     6.7%   (-44%)

The link is the bottleneck, so the gain is CPU, not throughput.

kperf output (sender side) of runs near the median sendmsg and softirq
CPU, before:

  client: == Source <sender>
  client:   Tx 97.236 Gbps (729473351680 bytes in 60016496 usec)
  client:   Tx 97.594 Gbps (732157706240 bytes in 60016496 usec)
  client:   Tx 97.326 Gbps (730144440320 bytes in 60016496 usec)
  client:   Tx 97.505 Gbps (731486617600 bytes in 60016496 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60016496 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60016496 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60016496 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60016496 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:          2460   555 201      7    0 35035136  855
  client:          3130   664 149      2    0 34394112  956
  client:          4589   488 213      6    0 36241408  860
  client:          4778   724 145      2    0 50849792  937
  client:   net CPU  5: usr: 0.00% sys: 0.03% idle:52.74% iow: 0.00% irq: 6.20% sirq:41.01%
  client:   app CPU  9: usr: 0.05% sys:11.93% idle:84.69% iow: 0.00% irq: 0.40% sirq: 2.91%
  client:   net CPU 27: usr: 0.01% sys: 0.01% idle:53.23% iow: 0.00% irq: 5.73% sirq:40.99%
  client:   app CPU 31: usr: 0.03% sys:12.17% idle:85.90% iow: 0.00% irq: 0.25% sirq: 1.63%
  client:   net CPU 32: usr: 0.00% sys: 0.03% idle:53.57% iow: 0.00% irq: 5.63% sirq:40.75%
  client:   app CPU 36: usr: 0.03% sys:11.85% idle:84.88% iow: 0.00% irq: 0.40% sirq: 2.82%
  client:   net CPU 34: usr: 0.00% sys: 0.03% idle:54.10% iow: 0.00% irq: 5.68% sirq:40.18%
  client:   app CPU 38: usr: 0.01% sys:12.10% idle:84.84% iow: 0.00% irq: 0.34% sirq: 2.68%

after:

  client: == Source <sender>
  client:   Tx 97.569 Gbps (732157706240 bytes in 60032032 usec)
  client:   Tx 97.211 Gbps (729473351680 bytes in 60032032 usec)
  client:   Tx 97.390 Gbps (730815528960 bytes in 60032032 usec)
  client:   Tx 97.390 Gbps (730815528960 bytes in 60032032 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60032032 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60032032 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60032032 usec)
  client:   Rx  0.000 Gbps (0 bytes in 60032032 usec)
  client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
  client:          6137   316 118      1    0 63000576  775
  client:          5872   344 177      6    0 52045824  707
  client:          6597   501 121      4    0 62879744  721
  client:          5938   347 136      5    0 56500224  707
  client:   net CPU 18: usr: 0.00% sys: 0.03% idle:55.40% iow: 0.00% irq: 6.25% sirq:38.30%
  client:   app CPU 22: usr: 0.03% sys: 6.67% idle:92.34% iow: 0.00% irq: 0.11% sirq: 0.83%
  client:   net CPU 24: usr: 0.01% sys: 0.03% idle:56.98% iow: 0.00% irq: 6.07% sirq:36.89%
  client:   app CPU 28: usr: 0.03% sys: 6.75% idle:92.71% iow: 0.00% irq: 0.08% sirq: 0.41%
  client:   net CPU 34: usr: 0.01% sys: 0.03% idle:56.20% iow: 0.00% irq: 5.90% sirq:37.84%
  client:   app CPU 38: usr: 0.03% sys: 6.81% idle:89.73% iow: 0.00% irq: 0.40% sirq: 3.01%
  client:   net CPU 44: usr: 0.01% sys: 0.05% idle:57.75% iow: 0.00% irq: 5.59% sirq:36.58%
  client:   app CPU 48: usr: 0.08% sys: 6.69% idle:92.18% iow: 0.00% irq: 0.13% sirq: 0.89%

The bnxt patches (1 and 5) were tested on two 50Gb/s bnxt hosts with
CONFIG_MAX_SKB_FRAGS=45 (with 17 the check in patch 1 compiles out).
kperf devmem TX from a 2MB hugepage-backed udmabuf runs at line rate
(46.4Gb/s, payload validated) with 2-3 frags per skb, none above 32KB.
ncdevmem sending 1KB chunks builds skbs with up to 45 frags, the stack
segments them and the data arrives intact.

Patches:
- 1: bnxt: segment GSO skbs with too many frags in the stack
- 2-3: devmem TX prep (truesize, niov_shift in the TX lookup)
- 4-5: per-device netmem TX frag size limit, 32KB for bnxt
- 6: pick the TX net_iov size from the sg table
- 7: selftest for TX from a hugepage-backed dmabuf

Stanislav Fomichev (7):
  bnxt_en: let the stack segment GSO skbs with too many frags
  net: devmem: charge the frag length to TX skb truesize
  net: devmem: use niov_shift in net_devmem_get_niov_at()
  net: devmem: add a per-device netmem TX frag size limit
  bnxt_en: limit netmem TX frags to 32KB
  net: devmem: size TX net_iovs from the dmabuf sg table
  selftests: drv-net: devmem: TX from a hugepage-backed dmabuf

 .../networking/net_cachelines/net_device.rst  |  1 +
 Documentation/networking/netmem.rst           |  6 +++
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |  6 +++
 include/linux/netdevice.h                     |  2 +
 net/core/datagram.c                           |  3 +-
 net/core/dev.c                                |  1 +
 net/core/devmem.c                             | 37 +++++++++++++---
 net/core/devmem.h                             |  1 +
 net/core/netdev-genl.c                        |  2 +-
 .../selftests/drivers/net/hw/devmem.py        | 10 ++++-
 .../selftests/drivers/net/hw/devmem_lib.py    | 43 +++++++++++++++++--
 .../selftests/drivers/net/hw/ncdevmem.c       |  8 +++-
 .../selftests/drivers/net/hw/nk_devmem.py     | 11 ++++-
 13 files changed, 115 insertions(+), 16 deletions(-)

-- 
2.53.0-Meta