[PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
From: Koichiro Den
Date: Thu Mar 12 2026 - 12:51:19 EST
Hi,
This series lets an endpoint-integrated DMA engine be consumed on the RC
side through vNTB.
The initial target is DesignWare endpoint eDMA. pci-epf-vntb exports a
versioned DMA locator plus the minimum peer-visible resources,
ntb_hw_epf parses that locator and instantiates an auxiliary device after
LINK_UP, and dw-edma-aux binds to that child to expose a DMA engine
provider on the RC side. ntb_ep_dma is included both as the first
consumer and as a simple bring-up test.
Background
==========
I previously posted a broader RFC series:
https://lore.kernel.org/all/20260118135440.1958279-1-den@xxxxxxxxxxxxx/
This series is not a direct continuation of that RFC. Its scope is
narrower, and the approach has changed substantially.
That RFC had two architectural issues:
1. dw-edma-specific logic lived under drivers/ntb/hw/, even though
the exported DMA engine is not related to any NTB hardware.
2. Remote-use channel delegation relied on vendor-specific peripheral
configuration.
This series builds on the recently discussed pci_epc_aux_resource work
(see "Dependency" 3 below) and addresses both issues by:
- introducing vendor-neutral DMA-channel delegation in the PCI EPC
layer via pci_epc_delegate_dma_channels() and
- making vNTB and ntb_hw_epf aware of the remote DMA resource.
On the EP side, pci-epf-vntb describes the exported DMA resources as
part of the vNTB BAR layout. On the RC side, ntb_hw_epf detects that
export and registers an auxiliary child device. A vendor-specific
frontend can then bind to that child device and reconstruct the remote
DMA provider. This series includes such a frontend for DesignWare eDMA
in dw-edma-aux.
Architecture
============
EP kernel
pci-epf-vntb
- exports the usual vNTB control/db/MW resources
- optionally exports a versioned DMA slice
RC kernel
ntb_hw_epf
- parses the control layout
- instantiates an auxiliary child for the exported DMA ABI
dw-edma-aux
- binds to that child and registers a DMAEngine provider
Series layout
=============
01-05 prepare dw-edma and auxiliary-resource metadata
06-10 export delegated controller-owned DMA resources through vNTB
11-13 discover the exported DMA instance on the host and bind
dw-edma-aux
14 adds ntb_ep_dma as the first consumer / smoke test
15 documents the model and the configfs layout
I did not split the infrastructure patches (01-13) away from its
consumer (14). The series is meant to be reviewed as one feature:
producer, discovery, consumer, and test coverage.
Test
====
Tested on R-Car S4 Spider with the dependency below.
1. Configure and start pci_epf_vntb with DMA export enabled.
The actual commands I used for testing:
# modprobe pci_epf_vntb
# cd /sys/kernel/config/pci_ep/
# mkdir functions/pci_epf_vntb/func1
# echo 0x1912 > functions/pci_epf_vntb/func1/vendorid
# echo 0x0030 > functions/pci_epf_vntb/func1/deviceid
# echo 32 > functions/pci_epf_vntb/func1/msi_interrupts
# echo 16 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count
# echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count
# echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws
# echo 0xF9000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1
# echo 0xF9000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_offset
# echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_num_chans
# echo 0x1912 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid
# echo 0x0030 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid
# echo 0x10 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number
# echo 0 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/ctrl_bar
# echo 2 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1_bar
# echo 2 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/dma_bar
# echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_bar
# ln -s controllers/e65d0000.pcie-ep functions/pci_epf_vntb/func1/primary/
# echo 1 > controllers/e65d0000.pcie-ep/start
2. Boot or rescan the RC side and let ntb_hw_epf probe.
3. Load ntb_ep_dma on both EP and RC.
4. On the RC side, run the test as follows:
# cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/ready
# echo 1 > /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/run
# cat /sys/kernel/debug/ntb_ep_dma/0000:01:00.0/result
last_status: 0
last_len: 4096
local_buf_dma: 0xfffff000
local_buf_size: 4096
peer_ready: 1
peer_state: pass # <----(*)
peer_dma: 0x4e11e000
peer_size: 4096
peer_seq: 1
peer_xfer_len: 4096
link_up: 1
(*) The peer reports "pass" after the transfer completes successfully,
Kernel base
===========
pci.git endpoint:
Commit 0b74f7d72399 ("PCI: endpoint: Propagate error from pci_epf_create()")
Dependency
==========
1. [PATCH v4 00/10] PCI: endpoint: Differentiate between disabled and reserved BARs
https://lore.kernel.org/linux-pci/20260312130229.2282001-12-cassel@xxxxxxxxxx/
https://patchwork.kernel.org/project/linux-pci/list/?series=1065666
2. [PATCH 0/2] dmaengine: dw-edma: Interrupt-emulation doorbell support
https://lore.kernel.org/dmaengine/20260215152216.3393561-1-den@xxxxxxxxxxxxx/
https://patchwork.kernel.org/project/linux-dmaengine/list/?series=1054298
Note: already landed in dmaengine/next.
3. [PATCH v10 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
https://lore.kernel.org/all/20260302071427.534158-1-den@xxxxxxxxxxxxx/
https://patchwork.kernel.org/project/linux-pci/list/?series=1059820
4. [PATCH v2 0/3] NTB: Allow drivers to provide DMA mapping device
https://lore.kernel.org/linux-pci/20260306031443.1911860-1-den@xxxxxxxxxxxxx/
https://patchwork.kernel.org/project/linux-pci/list/?series=1062308
Note: this series uses ntb_get_dma_dev() API.
5. [PATCH v2 00/10] PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@xxxxxxxxxxxxx
https://patchwork.kernel.org/project/linux-pci/list/?series=1058871
Note: v2 title was incorrect. See my reply to the cover letter.
Additionally, for ntb_ep_dma test to pass on R-Car S4 Spider:
6. [PATCH v2] PCI: dwc: rcar-gen4-ep: Mark BAR0 and BAR2 as Resizable BARs
https://lore.kernel.org/linux-pci/20260210160315.2272930-1-den@xxxxxxxxxxxxx/
https://patchwork.kernel.org/project/linux-pci/list/?series=1052780
Note: already landed in pci/next.
7. [PATCH v2] PCI: dwc: rcar-gen4: Use 4K EPC BAR alignment
https://lore.kernel.org/linux-pci/20260305151050.1834007-1-den@xxxxxxxxxxxxx/
https://patchwork.kernel.org/project/linux-pci/list/?series=1062031
Merge plan
==========
This series touches three areas:
- PCI endpoint core and pci-epf-vntb
- DesignWare eDMA (dw-edma)
- an NTB test client
The series intentionally keeps the infrastructure changes together with
their first consumer, the ntb_ep_dma test client. Splitting them further
would leave the infrastructure patches without a consumer, so the
patches are kept together as a single series.
Mani is a maintainer for both the PCI EP and dw-edma. My initial thought
was therefore to collect acks from the relevant subsystems (PCI EP,
dw-edma, and NTB) and have the series applied through the PCI EP tree.
However, I am of course open to any suggestions regarding the preferred
merge path or series split if maintainers think another approach would
be more appropriate.
Best regards,
Koichiro
Koichiro Den (15):
dmaengine: dw-edma: Cache DMA channel IDs in dw_edma_chip
PCI: endpoint: Add DMA channel metadata to pci_epc_aux_resource
PCI: dwc: ep: Report DMA channel metadata for aux resources
dmaengine: dw-edma: Add per-channel interrupt routing control
dmaengine: dw-edma: Compose MSI messages from allocated IRQs
PCI: endpoint: pci-epf-vntb: Fold MW runtime state into a struct
PCI: endpoint: Add EPC DMA channel delegation hooks
PCI: dwc: ep: Delegate exported eDMA channels through EPC ops
PCI: endpoint: Add pci-ep-dma helper for exported DMA ABI v1
PCI: endpoint: pci-epf-vntb: Support DMA export and shared BAR layouts
NTB: hw: epf: Parse control-layout version and DMA locator
NTB: hw: epf: Enumerate auxiliary child for DMA ABI v1
dmaengine: dw-edma: Add auxiliary-bus frontend for exported eDMA
NTB: Add ntb_ep_dma test client
Documentation: PCI: endpoint: Add vNTB DMA export HOWTO
Documentation/PCI/endpoint/index.rst | 1 +
.../PCI/endpoint/pci-vntb-dma-howto.rst | 83 ++
drivers/dma/dw-edma/Kconfig | 11 +
drivers/dma/dw-edma/Makefile | 1 +
drivers/dma/dw-edma/dw-edma-aux.c | 297 +++++++
drivers/dma/dw-edma/dw-edma-core.c | 101 ++-
drivers/dma/dw-edma/dw-edma-core.h | 13 +
drivers/dma/dw-edma/dw-edma-v0-core.c | 26 +-
drivers/ntb/hw/epf/Kconfig | 1 +
drivers/ntb/hw/epf/ntb_hw_epf.c | 199 ++++-
drivers/ntb/test/Kconfig | 10 +
drivers/ntb/test/Makefile | 1 +
drivers/ntb/test/ntb_ep_dma.c | 695 +++++++++++++++
.../pci/controller/dwc/pcie-designware-ep.c | 196 +++++
drivers/pci/controller/dwc/pcie-designware.h | 11 +
drivers/pci/endpoint/Makefile | 2 +-
drivers/pci/endpoint/functions/pci-epf-vntb.c | 794 ++++++++++++++++--
drivers/pci/endpoint/pci-ep-dma.c | 342 ++++++++
drivers/pci/endpoint/pci-epc-core.c | 84 ++
include/linux/dma/edma.h | 42 +
include/linux/pci-ep-dma.h | 130 +++
include/linux/pci-epc.h | 31 +
22 files changed, 2981 insertions(+), 90 deletions(-)
create mode 100644 Documentation/PCI/endpoint/pci-vntb-dma-howto.rst
create mode 100644 drivers/dma/dw-edma/dw-edma-aux.c
create mode 100644 drivers/ntb/test/ntb_ep_dma.c
create mode 100644 drivers/pci/endpoint/pci-ep-dma.c
create mode 100644 include/linux/pci-ep-dma.h
--
2.51.0