Re: [PATCH net-next 6/7] net: devmem: size TX net_iovs from the dmabuf sg table

[Mina Almasry]

On Mon, Nov 2, 2026 at 3:38 PM Stanislav Fomichev <sdf.kernel@xxxxxxxxx> wrote:
>
> On 11/02, Mina Almasry wrote:
> > On Mon, Nov 2, 2026 at 11:14 AM Stanislav Fomichev <sdf.kernel@xxxxxxxxx> wrote:
> > >
> > > PAGE_SIZE TX net_iovs cap devmem skbs at MAX_SKB_FRAGS pages (~68KB),
> > > which makes BIG TCP ineffective.
> > >
> > > TX net_iovs only need to be DMA contiguous. Use the largest power of 2,
> > > up to 2MB, which aligns every DMA segment of the dmabuf:
> > > - 2MB hugepage-backed udmabufs get 2MB net_iovs
> > > - no UAPI change, TX addresses are dmabuf offsets
> > > - skbs are still bounded by TCP's size_goal, only frags get larger
> > >
> > > Signed-off-by: Stanislav Fomichev <sdf@xxxxxxxxxxx>
> > > ---
> > >  net/core/devmem.c      | 28 ++++++++++++++++++++++++++--
> > >  net/core/netdev-genl.c |  2 +-
> > >  2 files changed, 27 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/net/core/devmem.c b/net/core/devmem.c
> > > index 19c039ed2cb2..98c9594dbf99 100644
> > > --- a/net/core/devmem.c
> > > +++ b/net/core/devmem.c
> > > @@ -181,6 +181,20 @@ int net_devmem_bind_dmabuf_to_queue(struct net_device *dev, u32 rxq_idx,
> > >         return err;
> > >  }
> > >
> > > +static bool net_devmem_sgt_aligned(struct sg_table *sgt, size_t size)
> > > +{
> > > +       struct scatterlist *sg;
> > > +       unsigned int i;
> > > +
> > > +       for_each_sgtable_dma_sg(sgt, sg, i) {
> > > +               if (!IS_ALIGNED(sg_dma_address(sg), size) ||
> > > +                   !IS_ALIGNED(sg_dma_len(sg), size))
> > > +                       return false;
> > > +       }
> > > +       return true;
> > > +}
> > > +
> > > +/* niov_shift == 0 picks the net_iov size from the dmabuf, TX only */
> > >  struct net_devmem_dmabuf_binding *
> > >  net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> > >                        struct device *dma_dev,
> > > @@ -191,11 +205,11 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> > >                        struct netlink_ext_ack *extack)
> > >  {
> > >         struct net_devmem_dmabuf_binding *binding;
> > > -       size_t niov_size = 1UL << niov_shift;
> > >         static u32 id_alloc_next;
> > >         struct scatterlist *sg;
> > >         struct dma_buf *dmabuf;
> > >         unsigned int sg_idx;
> > > +       size_t niov_size;
> > >         size_t niov_idx;
> > >         size_t i;
> > >         long err;
> > > @@ -221,7 +235,6 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> > >         binding->tx_max_frag_size = dev->netmem_tx_max_frag_size;
> > >         write_pnet(&binding->net, sock_net(nl_sk));
> > >         binding->uid = sk_uid(nl_sk);
> > > -       binding->niov_shift = niov_shift;
> > >         xa_init_flags(&binding->bound_rxqs, XA_FLAGS_ALLOC);
> > >
> > >         err = percpu_ref_init(&binding->ref,
> > > @@ -266,6 +279,17 @@ net_devmem_bind_dmabuf(struct net_device *dev, void *vdev,
> > >                 goto err_unmap;
> > >         }
> > >
> > > +       if (!niov_shift) {
> > > +               niov_size = SZ_2M;
> > > +               while (niov_size > PAGE_SIZE &&
> > > +                      (!IS_ALIGNED(dmabuf->size, niov_size) ||
> > > +                       !net_devmem_sgt_aligned(binding->sgt, niov_size)))
> > > +                       niov_size /= 2;
> > > +               niov_shift = ilog2(niov_size);
> > > +       }
> > > +       binding->niov_shift = niov_shift;
> > > +       niov_size = 1UL << niov_shift;
> > > +
> >
> > How come we're not checknig the largest supported size of the netdev here?
> >
> > Also, maybe should clarify in the cl with the largest supported size,
> > is it actually only powers of 2 up to that power of 2? Or all sizes up
> > to that (not-necassirly) power of 2 size?
>
> I think these are a bit orthogonal, no? We can have niovs larger than
> the HW can tx. If that's the case, we'll just sub-divide when we build
> the skb. If we limit to the same chunk the device can handle here,
> we might use 2 frags to send single chunk worth of data more often (offset
> doesn't have to be aligned to the chunk size).
>
>
> Assuming device can only send 16K, current patch:
>
> [       2MB        ] 2BM niov
>      [ 16K ]         sending 16K from that 2MB will use one chunk
>
>
> Limiting chunks to device's chunk size:
>
> [ 16K ][ 16K ]       16K niov
>     [ 16K ]          sending 16K from offset 8K will have to use 2 frags
>
> (obviously, the same thing can happen with 2MB chunks, but less often)
>
> LMK if it does not sound convincing.

Ah, I see, the skb gets segmented somewhere else of course. Derp

Reviewed-by: Mina Almasry <almasrymina@xxxxxxxxxx>

-- 
Thanks,
Mina