Re: [RFC PATCH 0/7] vfio: selftests: Add peer-to-peer DMA selftest

From: David Matlack

Date: Mon Nov 16 2026 - 15:50:36 EST


On Mon Nov 16, 2026 at 10:24 AM PST, Samiullah Khawaja wrote:
> On Mon, Nov 16, 2026 at 10:02:56AM -0800, David Matlack wrote:
>>Looks like GPUs and NVMe can do both as well.
>
> AFAIU the memory BARs for GPUs would need initialization. The internal
> VRAM needs to be exposed (mapped) to the BARs.
>
> Do you think we should have a P2P device "setup" step that can be
> implemented for various types of devices?

Yes, I think we need something like that.

I've been prototyping a new optional op in struct vfio_pci_driver_ops:

/*
* Set up a range of device memory, accessible through a BAR,
* that other devices can target with P2P DMA and that the CPU
* can access through mem_bar->vaddr.
*/
int (*mem_bar_setup)(struct vfio_pci_device *device, u64 min_size,
struct vfio_pci_mem_bar *mem_bar);

vfio_pci_find_mem_bar() falls back to calling it when the device does
not match any entry in mem_bar_descs[], sanity checks the returned
range, and then runs the existing memory verification on it.
mem_bar_descs[] stays around for devices like pci-testdev that need no
setup.

For NVIDIA GPUs, I had an LLM work on it and the setup was simpler than
I expected, at least on Blackwell. BAR1 comes out of reset in physical
mode, where BAR1 offset N maps directly to VRAM physical address N
without going through the GPU MMU. The LLM confirmed this on by writing
VRAM through BAR1 and reading it back through the BAR0 PRAMIN window
(and vice versa) at offsets from 256MiB up to 178GiB.

The nv_falcon implementation only has to:

- Wait for FSP to finish booting, which takes a few seconds after the
reset VFIO does when the device is opened.
- Make sure BAR1 is in physical mode.
- Return the range of BAR1 that maps VRAM (the size comes from
NV_USABLE_FB_SIZE_IN_MB), staying clear of firmware reservations at
either end.

With that the P2P test passes with a ConnectX-7 (mlx5 driver) as the
initiator and B200 VRAM as the target.

A couple of open questions:

1. The prototype allows .mem_bar_setup() to be called before .init().
That is enough for GPUs, but as Jason described, mlx5 device memory
has to be allocated from firmware, which needs the command interface
and therefore init(). Supporting mlx5 as a target would mean running
.mem_bar_setup() after .init().

2. nv_falcon does not support Falcon DMA on Blackwell yet, so a B200
can only be a target for now. That requires a driver to be able to
probe a device that cannot do memcpy. I am not sure I want to go
down that road.