[PATCH v9 0/7] PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
From: Koichiro Den
Date: Thu Feb 19 2026 - 03:15:36 EST
Hi,
Some endpoint platforms cannot use a GIC ITS-backed MSI domain for
EP-side doorbells. In those cases, endpoint function (EPF) drivers
cannot provide a doorbell to the root complex (RC), and features such as
vNTB may fall back to polling with significantly higher latency.
This series adds an alternate doorbell backend based on the DesignWare
PCIe controller's integrated eDMA interrupt-emulation feature. The RC
rings the doorbell by doing a single 32-bit MMIO write to an eDMA
doorbell location exposed in a BAR window. The EP side receives a Linux
IRQ that EPF drivers can use as a doorbell interrupt, without relying on
MSI message writes reaching the ITS.
To support this, the series:
- Adds an EPC auxiliary resource query API so EPF drivers can discover
controller-integrated resources (DMA MMIO, doorbell MMIO, and DMA LL
memory).
- Updates DesignWare EP controllers to report integrated eDMA
resources via the new API.
- Updates dw-edma to provide a dedicated virtual IRQ for interrupt
emulation and to perform the core-specific deassert sequence.
- Updates pci-epf-test and pci-epf-vntb to reuse a pre-exposed
BAR/offset and to honor per-doorbell IRQ flags.
Many thanks to Frank and Niklas for their continued review and valuable
feedback throughout the development of this series. The Reviewed-by tags
for the last two patches are dropped due to the additional changes
following Niklas' review in the v8 threads. Since the diff is small, I'd
appreciate it if Frank could re-check them.
Dependencies
------------
The following three series are prerequisites for this series:
(1). [PATCH v2 0/4] PCI: endpoint: Doorbell-related fixes
https://lore.kernel.org/linux-pci/20260217063856.3759713-1-den@xxxxxxxxxxxxx/
(2). [PATCH 0/2] dmaengine: dw-edma: Interrupt-emulation doorbell support
https://lore.kernel.org/dmaengine/20260215152216.3393561-1-den@xxxxxxxxxxxxx/
(3). [PATCH 0/9] PCI: endpoint differentiate between disabled and reserved BARs
https://lore.kernel.org/linux-pci/20260217212707.2450423-11-cassel@xxxxxxxxxx/
Regarding (3):
- [PATCH 2/9] and [PATCH 3/9] are strictly the prerequisites for this v9 series.
In fact, they are split out from v8 series.
- With [PATCH 6/9], this v9 series should allow the embedded doorbell fallback
path to pass on RK3588 from the beginning. Given that, picking up the whole
(3) series earlier should be the most streamlined choice.
Tested on
---------
I re-tested the embedded (DMA) doorbell fallback path (via pci-epf-test)
on R-Car Spider boards (with this v9 series):
$ ./pci_endpoint_test -t DOORBELL_TEST
TAP version 13
1..1
# Starting 1 tests from 1 test cases.
# RUN pcie_ep_doorbell.DOORBELL_TEST ...
# OK pcie_ep_doorbell.DOORBELL_TEST
ok 1 pcie_ep_doorbell.DOORBELL_TEST
# PASSED: 1 / 1 tests passed.
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
with the following message observed on the EP side:
[ 82.043715] pci_epf_test pci_epf_test.0: Can't find MSI domain for EPC
[ 82.044382] pci_epf_test pci_epf_test.0: Using embedded (DMA) doorbell fallback
(Note: for the test to pass on R-Car Spider, one of the following was required:
- echo 1048576 > functions/pci_epf_test/func1/pci_epf_test.0/bar2_size
- apply https://lore.kernel.org/linux-pci/20260210160315.2272930-1-den@xxxxxxxxxxxxx/)
Performance test: vNTB ping latency
-----------------------------------
Setup:
- configfs (R-Car Spider in EP mode):
cd /sys/kernel/config/pci_ep/
mkdir functions/pci_epf_vntb/func1
echo 0x1912 > functions/pci_epf_vntb/func1/vendorid
echo 0x0030 > functions/pci_epf_vntb/func1/deviceid
echo 32 > functions/pci_epf_vntb/func1/msi_interrupts
echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count
echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count
echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws
echo 0x100000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1
echo 0x1912 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid
echo 0x0030 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid
echo 0x10 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number
echo 0 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/ctrl_bar
echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_bar [*]
echo 2 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1_bar
ln -s controllers/e65d0000.pcie-ep functions/pci_epf_vntb/func1/primary/
echo 1 > controllers/e65d0000.pcie-ep/start
[*]: On R-Car Spider, a hack is currently needed to use BAR4 for
the doorbell. I'll consider posting a patch for that
separately.
- ensure ntb_transport/ntb_netdev are loaded on both sides
Results:
- Without this series (pci.git main)
$ ping -c 10 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=6.04 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=12.6 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=7.40 ms
64 bytes from 10.0.0.11: icmp_seq=4 ttl=64 time=5.38 ms
64 bytes from 10.0.0.11: icmp_seq=5 ttl=64 time=11.4 ms
64 bytes from 10.0.0.11: icmp_seq=6 ttl=64 time=9.42 ms
64 bytes from 10.0.0.11: icmp_seq=7 ttl=64 time=3.36 ms
64 bytes from 10.0.0.11: icmp_seq=8 ttl=64 time=9.48 ms
64 bytes from 10.0.0.11: icmp_seq=9 ttl=64 time=4.24 ms
64 bytes from 10.0.0.11: icmp_seq=10 ttl=64 time=10.4 ms
- With this series (on top of pci.git main + Dependency (1), (2) and (3))
$ ping -c 10 10.0.0.11
PING 10.0.0.11 (10.0.0.11) 56(84) bytes of data.
64 bytes from 10.0.0.11: icmp_seq=1 ttl=64 time=0.845 ms
64 bytes from 10.0.0.11: icmp_seq=2 ttl=64 time=0.742 ms
64 bytes from 10.0.0.11: icmp_seq=3 ttl=64 time=0.868 ms
64 bytes from 10.0.0.11: icmp_seq=4 ttl=64 time=0.806 ms
64 bytes from 10.0.0.11: icmp_seq=5 ttl=64 time=0.951 ms
64 bytes from 10.0.0.11: icmp_seq=6 ttl=64 time=0.965 ms
64 bytes from 10.0.0.11: icmp_seq=7 ttl=64 time=0.871 ms
64 bytes from 10.0.0.11: icmp_seq=8 ttl=64 time=0.877 ms
64 bytes from 10.0.0.11: icmp_seq=9 ttl=64 time=0.938 ms
64 bytes from 10.0.0.11: icmp_seq=10 ttl=64 time=0.960 ms
---
Changelog
---------
* v8->v9 changes:
- Add a new dependency series (3), which moved the BAR reserved-subregion
framework + the RK3588 BAR4 example out of v8 (dropping the corresponding
patches from this series).
- pci-epf-vntb: rename the duplicate-IRQ helper and invert the return value,
per Frank's review.
- pci-epf-test: drop the extra size_add() doorbell-offset check, per Niklas'
review.
- pci-ep-msi: add a DWORD alignment check for DOORBELL_MMIO, per Niklas's
review.
- Carry over Reviewed-by tags for unchanged patches + drop Reviewed-by tags
where code changed.
- Rename the last patch subject (drop 'eDMA' word).
* v7->v8 changes:
- Deduplicate request_irq()/free_irq() calls based on virq (shared
IRQ) rather than doorbell type, as suggested during review of v7
Patch #7.
- Clean up the pci_epf_alloc_doorbell() error path, as suggested
during review of v7 Patch #9.
- Use range_end_overflows_t() instead of an open-coded overflow check,
following discussion during review of v7 Patch #5.
- Add a write-data field to the DOORBELL_MMIO aux-resource metadata
and plumb it through to the embedded doorbell backend (DesignWare
uses data=0).
* v6->v7 changes:
- Split out preparatory patches to keep the series below 10 patches.
- Add support for platforms where the eDMA register block is fixed
within a reserved BAR window (e.g. RK3588 BAR4) and must be reused
as-is.
- Introduce a dedicated virtual IRQ and irq_chip (using
handle_level_irq) for interrupt-emulation doorbells instead of
reusing per-channel IRQs. This avoids delivery via different IRQs on
platforms with chip->nr_irqs > 1.
* v5->v6 changes:
- Fix a double-free in v5 Patch 8/8 caused by mixing __free(kfree) with
an explicit kfree(). This is a functional bug (detectable by KASAN),
hence the respin solely for this fix. Sorry for the noise. No other
changes.
* v4->v5 changes:
- Change the series subject now that the series has evolved into a
consumer-driven set focused on the embedded doorbell fallback and its
in-tree users (epf-test and epf-vntb).
- Drop [PATCH v4 01/09] (dw-edma per-channel interrupt routing control)
from this series for now, so the series focuses on what's needed by the
current consumer (i.e. the doorbell fallback implementation).
- Replace the v4 embedded-doorbell "test variant + host/kselftest
plumbing" with a generic embedded-doorbell fallback in
pci_epf_alloc_doorbell(), including exposing required IRQ request flags
to EPF drivers.
- Two preparatory fix patches (Patch 6/8 and 7/8) to clean up error
handling and state management ahead of Patch 8/8.
- Rename *_get_remote_resource() to *_get_aux_resources() and adjust
relevant variable namings and kernel docs. Discussion may continue.
- Rework dw-edma per-channel metadata exposure to cache the needed info
in dw_edma_chip (IRQ number + emulation doorbell offset) and consume it
from the DesignWare EPC auxiliary resource provider without calling back
to dw-edma.
* v3->v4 changes:
- Drop dma_slave_caps.hw_id and the dmaengine selfirq callback
registration API. Instead, add a dw-edma specific dw_edma_chan_info()
helper and extend the EPC remote resource metadata accordingly.
- Add explicit acking for eDMA interrupt emulation and adjust the
dw-edma IRQ path for embedded-doorbell usage.
- Replace the previous EPC API smoke test with an embedded doorbell
test variant (pci-epf-test + pci_endpoint_test/selftests).
- Rebase onto pci.git controller/dwc commit 43d324eeb08c.
* v2->v3 changes:
- Replace DWC-specific helpers with a generic EPC remote resource query API.
- Add pci-epf-test smoke test and host/kselftest support for the new API.
- Drop the dw-edma-specific notify-only channel and polling approach
([PATCH v2 4/7] and [PATCH v2 5/7]), and rework notification handling
around a generic dmaengine_(un)register_selfirq() API implemented
by dw-edma.
* v1->v2 changes:
- Combine the two previously posted series into a single set (per Frank's
suggestion). Order dmaengine/dw-edma patches first so hw_id support
lands before the PCI LL-region helper, which assumes
dma_slave_caps.hw_id availability.
v8: https://lore.kernel.org/linux-pci/20260217080601.3808847-1-den@xxxxxxxxxxxxx/
v7: https://lore.kernel.org/linux-pci/20260215163847.3522572-1-den@xxxxxxxxxxxxx/
v6: https://lore.kernel.org/all/20260209125316.2132589-1-den@xxxxxxxxxxxxx/
v5: https://lore.kernel.org/all/20260209062952.2049053-1-den@xxxxxxxxxxxxx/
v4: https://lore.kernel.org/all/20260206172646.1556847-1-den@xxxxxxxxxxxxx/
v3: https://lore.kernel.org/all/20260204145440.950609-1-den@xxxxxxxxxxxxx/
v2: https://lore.kernel.org/all/20260127033420.3460579-1-den@xxxxxxxxxxxxx/
v1: https://lore.kernel.org/dmaengine/20260126073652.3293564-1-den@xxxxxxxxxxxxx/
+
https://lore.kernel.org/linux-pci/20260126071550.3233631-1-den@xxxxxxxxxxxxx/
Thanks for reviewing.
Koichiro Den (7):
PCI: endpoint: Add auxiliary resource query API
PCI: dwc: Record integrated eDMA register window
PCI: dwc: ep: Expose integrated eDMA resources via EPC aux-resource
API
PCI: endpoint: pci-ep-msi: Refactor doorbell allocation for new
backends
PCI: endpoint: pci-epf-vntb: Reuse pre-exposed doorbells and IRQ flags
PCI: endpoint: pci-epf-test: Reuse pre-exposed doorbell targets
PCI: endpoint: pci-ep-msi: Add embedded doorbell fallback
.../pci/controller/dwc/pcie-designware-ep.c | 151 ++++++++++++++++++
drivers/pci/controller/dwc/pcie-designware.c | 4 +
drivers/pci/controller/dwc/pcie-designware.h | 2 +
drivers/pci/endpoint/functions/pci-epf-test.c | 84 ++++++----
drivers/pci/endpoint/functions/pci-epf-vntb.c | 61 ++++++-
drivers/pci/endpoint/pci-ep-msi.c | 149 +++++++++++++++--
drivers/pci/endpoint/pci-epc-core.c | 41 +++++
include/linux/pci-epc.h | 52 ++++++
include/linux/pci-epf.h | 23 ++-
9 files changed, 520 insertions(+), 47 deletions(-)
--
2.51.0