Re: [PATCH net-next v4 0/5] net: devmem: size TX net_iovs from the dmabuf layout

[Mina Almasry]

On Thu, Nov 12, 2026 at 4:42 PM Stanislav Fomichev <sdf.kernel@xxxxxxxxx> wrote:
>
> On 11/12, Stanislav Fomichev wrote:
> > On 11/12, Paolo Abeni wrote:
> > > Hi,
> > >
> > > A few (late) questions/comments to check I'm not missing relevant
> > > points,
> > >
> > > On 11/11/26 23:47, Stanislav Fomichev wrote:
> > > > TX bindings use PAGE_SIZE net_iovs, so a frag covers at most a page.
> > > > A 64KB skb needs 16 frags, and MAX_SKB_FRAGS caps devmem skbs at about
> > > > 68KB, which makes BIG TCP ineffective for devmem TX.
> > >
> > > ... with default .config. You could set MAX_SKB_FRAGS to 45 and reach
> > > ~180K...
> > > > This series picks the TX net_iov size from the dmabuf sg table instead:
> > > > the largest power of 2, up to 2MB, which aligns every DMA segment. With
> > > > a 2MB hugepage-backed udmabuf a BIG TCP skb takes a few frags. There's
> > > > no UAPI change since TX addresses are dmabuf offsets.
> > >
> > > ... still this option should be more effective because the limit would be
> > > higher and the skb would use less cachelines for the shared_info,
> > >
> > > I'm wondering if making a comparison vs MAX_SKB_FRAGS==45 would be
> > > complex and if you could do that, too.
> >
> > Originally I did not want to include bigtcp to keep it apples to apples:
> > same tso size, just 16 frags in skb vs 1 (in ideal case).
> >
> > I just tried running mlx5 with 45 frags configured and it fails to start with
> > the following error:
> > mlx5e_attach_netdev() warns "MLX5E: Max SQ WQEBBs firmware capability: 16, needed 23" and returns -EIO.
> >
> > (fwiw quick review with llm claims that mlx5 only supports up to 20 skb frags)
> >
> > What I also tried just now is `gso_max_size 524280` (on top of this series)
> > and it ups single flow throughput from 200Gbps to 240Gbps (presumably
> > by making the receiver go faster with 512K TSOs vs 64K TSOs) and halves
> > the sender softirq..
> >
> > I hope this gives you more data, not sure running with 20 frags is worth
> > it (vs 16)? LMK if you disagree or want me to run something else.
>
> Fun fact, also setting gro_max_size on the receiver gets line rate over
> single flow:
>
> I   client.c: Connected 1:cpu 39 | 1:cpu 118
> client: == Source 2401:db00:255a:4a:bace::b
> client:   Tx389.448 Gbps (2921248849920 bytes in 60008026 usec)
> client:   Rx  0.000 Gbps (0 bytes in 60008026 usec)
> client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> client:             2   125  86      2    0 15300608 10142
> client:   net CPU 39: usr: 0.03% sys: 0.06% idle:67.39% iow: 0.00% irq: 5.69% sirq:26.81%
> client:   app CPU 40: usr: 0.05% sys: 2.91% idle:95.67% iow: 0.00% irq: 0.21% sirq: 1.14%
> client: == Target 2401:db00:255a:44:bace::b
> client:   Tx  0.000 Gbps (0 bytes in 60009306 usec)
> client:   Rx389.438 Gbps (2921238435008 bytes in 60009306 usec)
> client:   TCP retrans reord rtt rttvar d_ce snd_wnd cwnd
> client:             0     0 210    122    0  999424   10
> client:   net CPU118: usr: 0.00% sys: 0.00% idle: 1.60% iow: 0.00% irq: 0.33% sirq:98.05%
> client:   app CPU119: usr: 2.75% sys:77.46% idle:18.95% iow: 0.00% irq: 0.47% sirq: 0.34%
>
> 4200MTU, 32K rx-buf-len, this patch series, gso/gro_mas_size of 524280,
> still the default number of frags (enough on rx to fill 512K GRO with 32K
> chunks).

With 'devmem' over udmabuf? Is this possible without devmem on your
setup? How much is devmem helping I wonder.


-- 
Thanks,
Mina