Re: [PATCH net-next v4 0/3] net: devmem: allow rx-buf-size > PAGE_SIZE per binding

[Mina Almasry]

(I'm kinda reviewing this very late here. Some suggestions/comments
but feel free to ignore if not useful).

On Wed, Jul 1, 2026 at 12:22 PM Bobby Eshleman <bobbyeshleman@xxxxxxxxx> wrote:
>
> Every devmem dmabuf binding hands the page_pool PAGE_SIZE niovs today.
> On NICs that consume one descriptor per netmem, this caps a single RX
> descriptor at PAGE_SIZE and burns CPU on buffer churn.
>
> In this series, we add a bind-time netlink attribute,
> NETDEV_A_DMABUF_RX_BUF_SIZE, that lets userspace request a larger niov size
> (power of two >= PAGE_SIZE).

FWIW we may be able to support arbitrary sizes with devmem. Because
the genpool supports byte-aligned allocations AFAIR. Also the
dma-mapping happens with the dma-buf size, so the actual niov size
doesn't matter. The only thing I can think off which may not be
flexible to arbitrary sizes is the driver itself. IDK what happens if
you ask the driver to dma into a buffer that is frag size 5023 or
something like that.

But that is something that can be relaxed in the future.

> Drivers must opt in via
> queue_mgmt_ops.QCFG_RX_PAGE_SIZE.
>

nit that probably doesn't matter: ...QCFG_RX_NETMEM_SIZE, or
(...NIOV_SIZE). This doesn't actually work with pages, right?

If you decide to extend to arbrary sizes, I would add to the
queue_mgmt ops supports_netmem_size(size_t size) function, and let the
driver enforce "it has to be power of 2" if it needs to. AFAICT core
doesn't need to.

> Selftests use udmabuf, but udmabuf sgtables were previously hardcoded to
> PAGE_SIZE. This series modifies udmabuf to respect folio sizes in its exported
> sgtable. The result is that when backing udmabuf with MFD_HUGETLB 2MB pages,
> the sgtable is populated with 2MB entries, allowing devmem's gen_pool to carve
> out large (eg. 64K) niovs.
>
> Measurements
> ------------
>
> Setup: kperf devmem RX/TX cuda, 4 flows, 64 MB messages, 60s, dctcp,
> num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov size,
> mlx5.
>
>    niov       RX dev Gbps   RX flow avg Gbps         app sys %
>   -----  ----------------  -----------------  ----------------
>      4K  300.63 +/- 53.21    75.16 +/- 13.30   54.15 +/- 10.23
>     16K  321.35 +/- 28.20    80.34 +/-  7.05   41.05 +/-  8.87
>     32K  347.63 +/-  2.20    86.91 +/-  0.55   44.54 +/-  3.51
>     64K  332.11 +/- 14.26    83.03 +/-  3.56   35.47 +/-  3.11
>
> RX app sys % drops ~19% from 4K to 64K.
>

Hard to read the columns for me but seems like good perf data. Did
performance become worse from 32K to 64K? I wonder why.

I have some devmem performance fixes that are very critical for our
production that I haven't gotten around to upstreaming yet. I wonder
if I can send them to you for upstream submission. Are you potentially
interested?

-- 
Thanks,
Mina