Re: [PATCH net-next 00/10] net/mlx5e: Add netdev support for data direct

[Mina Almasry]

On Sat, Oct 10, 2026 at 7:16 AM Dragos Tatulea <dtatulea@xxxxxxxxxx> wrote:
>
>
>
> On 10.10.26 01:54, Mina Almasry wrote:
> > On Fri, Oct 9, 2026 at 9:42 AM Mina Almasry <almasrymina@xxxxxxxxxx> wrote:
> >>
> >> On Thu, Oct 8, 2026 at 6:28 AM Tariq Toukan <tariqt@xxxxxxxxxx> wrote:
> >>>
> >>> Hi,
> >>>
> >>> This patch series by Dragos adds data direct support for netdev devices,
> >>> allowing a mlx5 netdev to issue DMA commands through multiple paths.
> >>> This feature is critical for improving performance and reaching line
> >>> rate in certain environments where issuing PCI transactions over one
> >>> path may be significantly faster than over another. These differences
> >>> can arise from various PCI generations in the system or the specific
> >>> system topology.
> >>>
> >>> Data direct for netdev will work with devmem by returning the data
> >>> direct DMA device instead of the DMA device of the NIC. Each PF netdev
> >>> registers its own data direct device and support for RX and TX datapath
> >>> is added. The feature is selected through a new data direct priv flag.
> >>>
> >>> The feature is enabled though a ethtool private flag. When this flag
> >>> is enabled, the driver will return the data direct device in the
> >>> ndo_queue_get_dma_dev op.
> >>>
> >>> During data_direct device unbind, a re-creation of the channels is
> >>> triggered.
> >>>
> >>> A note about the unbind: it does not does not tear down dmabuf bindings
> >>> attached to the data direct device; the recreated channels will still
> >>> pick up the binding via rxq->mp_params and post DMA addresses from the
> >>> data direct IOMMU domain to the PF, causing IOMMU faults. A devmem
> >>> revoke hook is needed to close those bindings before the switch. This is
> >>> out of scope for this series.
> >>>
> >>
> >> Elaborate on this please. Are you referring to the fact that on
> >> page_pool_scrub we unmap the netmems dma-mapped by the page-pool, but
> >> we don't unmap the netmems dma-mapped by the memory provider?
> >>
> >> What's the impact of leaving this unfixed? Is the machine going to
> >> crash if the device goes away but we don't unmap the dmabuf?
> >
> IOMMU faults because the dma device is invalid. Either because the
> DD dev is still in use and was unbound. Or because the non-DD
> one is used and is invalid (due to channel reopen).
>
> > Oh, you're referring to an even different bug than the one I had in
> > mind. (I'll see if I can submit a fix for the bug I had in mind).
> >
> > But in general I think not addressing the issue you're referring to is
> > way too messy. I think we do indeed need to unbind the dma-buf if the
> > dma-dev is going away. The invariant should be that the
> > binding->attachment->dev should not change at all during the entire
> > lifetime of the binding. We already 'revoke' the dma-buf binding if
> > the netdev is going away, sorta (there is a bug there I need to fix.
> > The uninstall function in the provider doesn't actually unmap the
> > dma-buf :sad face:).
> >
> > I've worked with my LLM to suggest some changes that (we think) make
> > this work, but you may disagree. I'll post the suggestions. But please
> > I think this should be handled one way or another.
> >
> My idea was to handle the unbind part in a subsequent series because
> it probably needs some more back and forth to get this cleanup path
> right.
>
> If not acceptable I'll work on adding it to this series.
>

Since these are mostly driver-local changes I don't want to be so
forceful to block it by saying it's unacceptable, but if at all
possible lets try not to exacerbate the current mess. It looks like we
already a bunch of lifetime bugs with regards to the device going
away. I've posted this series, which - I think - gives you the devmem
'revoke' mechanism that you want:

https://lore.kernel.org/netdev/20261010025532.839559-1-almasrymina@xxxxxxxxxx/

Also working on that, I realized we don't use binding->vdev and
binding->dev correctly on binding rx. binding->vdev is meant to be the
device we do the bind to, and binding->dev is meant to be the
underlying dma_device (if they're different). That fits into your
model where you have the netdev you want to bind to, but have an
underlying device that you want to dma-map to.

I've gotten some feedback on my series and I'm working on v2. If it
makes sense for you to wait on my devmem core changes so you can
integrate into them better and avoid these lifetime bugs, lets do
that.

-- 
Thanks,
Mina