[PATCH v4 00/24] dmaengine: dw-edma: Support dynamic LL appends
From: Koichiro Den
Date: Wed Jul 29 2026 - 10:50:28 EST
Hi,
This is v4 of the dw-edma dynamic-append work originally proposed by
Frank Li in his RFT:
https://lore.kernel.org/r/20260109-edma_dymatic-v1-0-9a98c9c98536@xxxxxxx/
The driver currently treats each linked list as a finite transfer and
waits for completion before publishing more work. This series turns the
list into a circular producer/consumer ring and lets LL channels accept
new requests while running.
The motivation and hardware observations are unchanged since v2, linked
below. v4 mainly addresses feedback from Frank and Sashiko. It adds no
new patches.
With multiple DMAengine descriptors in one ring, entry reclamation and
descriptor completion need separate boundaries. The series records
IRQ-paired LLP samples, uses a conservative running boundary for both
eDMA and HDMA, and centralizes when a stopped channel needs another
doorbell.
The recovery patches add an opt-in safeguard for eDMA with direction
reset operations. On one integration, sustained multi-channel traffic
could leave a channel stopped with pending entries. Repeated doorbells
did not restart it, while toggling ENGINE_EN did. Doorbell timing
remains the leading suspect: under heavy multi-channel load,
channel-status updates visible to software appear to lag internal
start/stop transitions. That said, the exact trigger is still unknown.
Any insight from Synopsys would be appreciated. Recovery was not entered
in the fio/nvmet_pci_epf tests below.
Base
====
This series is based on dmaengine/next.
Re-testing with v4
==================
I re-tested v4 on the following three testbeds:
- eDMA: RK3588 Rock 5B, DWC PCIe v5.60a
R-Car S4, DWC PCIe v5.20a
- HDMA: SpacemiT K3, DWC PCIe v6.30a [1]
1. fio
The fio setup uses nvmet_pci_epf with a null_blk-backed namespace,
runtime=30s, ramp_time=5s. The "Before" values reuse the baseline
measurements reported with v2. The "After" values are three
consecutive v4 runs.
NOTE: The fio tests use nvmet_pci_epf, which does not keep the DMA
engine continuously busy. A workload that rarely lets the
engine stop would show the benefit of dynamic appends more
directly.
- eDMA (RK3588)
NOTE: The eDMA Before results include the separate regression
described in [2].
Summary by group (BW delta %)
all n=26 mean=+1129.2 median=+114.6 min= +38.4 max=+5346.2
read n=14 mean=+1996.5 median=+859.9 min= +54.9 max=+5346.2
write n=11 mean=+102.3 median=+107.7 min= +38.4 max=+168.2
qd32 n=16 mean=+1775.0 median=+166.5 min= +67.6 max=+5346.2
q1 n= 9 mean= +75.2 median= +66.4 min= +38.4 max=+131.2
small 4K n= 6 mean=+102.7 median= +88.7 min= +38.4 max=+168.2
large >=128K n=20 mean=+1437.1 median=+120.7 min= +52.2 max=+5346.2
Before mean -> After mean (MiB/s)
Case Before After Delta
--------------------------- ----------------- ----------------- ------
Rnd read 4KB q1 1j 19.2 (sd 7.5) 36.7 (sd 13.5) +90.6%
Rnd read 4KB q32 1j 88.7 (sd 2.1) 148.7 (sd 54.5) +67.6%
Rnd read 4KB q32 4j 88.7 (sd 1.3) 165.6 (sd 84.0) +86.7%
Rnd read 128KB q1 1j 420.0 (sd 134.6) 719.0 (sd 184.5) +71.2%
Rnd read 128KB q32 1j 142.0 (sd 37.7) 2544.3 (sd 132.2) +1691.8%
Rnd read 128KB q32 4j 63.7 (sd 10.6) 3020.0 (sd 29.7) +4643.5%
Rnd read 512KB q1 1j 572.0 (sd 35.8) 952.0 (sd 44.9) +66.4%
Rnd read 512KB q32 1j 66.8 (sd 10.8) 3007.7 (sd 38.2) +4402.5%
Rnd read 512KB q32 4j 55.4 (sd 12.0) 3019.0 (sd 39.0) +5346.2%
Rnd write 4KB q1 1j 20.7 (sd 7.0) 28.7 (sd 15.9) +38.4%
Rnd write 4KB q32 1j 42.5 (sd 0.9) 112.4 (sd 17.8) +164.7%
Rnd write 4KB q32 4j 41.7 (sd 1.6) 111.8 (sd 18.8) +168.2%
Rnd write 128KB q1 1j 330.0 (sd 90.1) 542.3 (sd 91.2) +64.3%
Rnd write 128KB q32 1j 624.0 (sd 111.5) 1309.3 (sd 60.4) +109.8%
Rnd write 128KB q32 4j 659.3 (sd 61.8) 1292.7 (sd 59.2) +96.1%
Seq read 128KB q1 1j 260.7 (sd 25.8) 602.7 (sd 85.0) +131.2%
Seq read 128KB q32 1j 149.7 (sd 30.9) 2527.3 (sd 13.0) +1588.6%
Seq read 512KB q1 1j 578.0 (sd 32.2) 895.3 (sd 24.6) +54.9%
Seq read 512KB q32 1j 65.0 (sd 10.6) 3017.0 (sd 34.9) +4541.5%
Seq read 1MB q32 1j 57.3 (sd 18.8) 3018.7 (sd 30.4) +5168.2%
Seq write 128KB q1 1j 272.3 (sd 62.0) 565.7 (sd 111.2) +107.7%
Seq write 128KB q32 1j 597.0 (sd 105.7) 1309.3 (sd 62.1) +119.3%
Seq write 512KB q1 1j 476.7 (sd 26.7) 725.7 (sd 32.7) +52.2%
Seq write 512KB q32 1j 710.7 (sd 44.0) 1297.3 (sd 57.9) +82.6%
Seq write 1MB q32 1j 546.0 (sd 100.4) 1213.0 (sd 56.3) +122.2%
Rnd rdwr 4K..1MB q8 4j 247.3 (sd 16.3) 943.3 (sd 88.5) +281.4%
- HDMA (SpacemiT K3)
Summary by group (BW delta %)
all n=26 mean= +85.6 median=+100.1 min= -26.9 max=+154.7
read n=14 mean= +76.9 median= +84.2 min= -26.9 max=+144.0
write n=11 mean= +92.0 median=+137.5 min= -24.5 max=+154.7
qd32 n=16 mean=+112.0 median=+106.1 min= +71.8 max=+145.6
q1 n= 9 mean= +33.0 median= +4.7 min= -26.9 max=+154.7
small 4K n= 6 mean= +76.9 median=+101.8 min= -19.5 max=+144.0
large >=128K n=20 mean= +88.2 median=+100.1 min= -26.9 max=+154.7
Before mean -> After mean (MiB/s)
Case Before After Delta
--------------------------- ----------------- ----------------- ------
Rnd read 4KB q1 1j 52.5 (sd 3.5) 55.3 (sd 4.4) +5.3%
Rnd read 4KB q32 1j 108.1 (sd 90.9) 263.7 (sd 7.6) +144.0%
Rnd read 4KB q32 4j 117.0 (sd 84.9) 267.0 (sd 8.9) +128.1%
Rnd read 128KB q1 1j 835.7 (sd 139.6) 611.0 (sd 171.4) -26.9%
Rnd read 128KB q32 1j 802.7 (sd 67.7) 1623.7 (sd 13.1) +102.3%
Rnd read 128KB q32 4j 931.3 (sd 262.1) 1666.3 (sd 10.4) +78.9%
Rnd read 512KB q1 1j 411.0 (sd 144.0) 895.0 (sd 90.4) +117.8%
Rnd read 512KB q32 1j 969.0 (sd 210.7) 1664.3 (sd 10.1) +71.8%
Rnd read 512KB q32 4j 889.7 (sd 298.0) 1679.7 (sd 12.5) +88.8%
Rnd write 4KB q1 1j 46.0 (sd 15.5) 37.1 (sd 18.7) -19.5%
Rnd write 4KB q32 1j 117.7 (sd 4.0) 232.3 (sd 10.1) +97.5%
Rnd write 4KB q32 4j 113.0 (sd 5.3) 233.0 (sd 9.5) +106.2%
Rnd write 128KB q1 1j 606.7 (sd 149.0) 458.3 (sd 20.2) -24.5%
Rnd write 128KB q32 1j 533.6 (sd 512.3) 1267.3 (sd 2.1) +137.5%
Rnd write 128KB q32 4j 526.0 (sd 514.9) 1266.7 (sd 2.3) +140.8%
Seq read 128KB q1 1j 626.3 (sd 247.5) 655.7 (sd 90.8) +4.7%
Seq read 128KB q32 1j 783.3 (sd 51.1) 1613.3 (sd 12.7) +106.0%
Seq read 512KB q1 1j 464.0 (sd 191.5) 918.3 (sd 18.0) +97.9%
Seq read 512KB q32 1j 918.3 (sd 241.8) 1640.0 (sd 35.5) +78.6%
Seq read 1MB q32 1j 918.7 (sd 290.5) 1649.3 (sd 42.7) +79.5%
Seq write 128KB q1 1j 611.0 (sd 79.8) 531.7 (sd 128.7) -13.0%
Seq write 128KB q32 1j 521.8 (sd 515.3) 1266.0 (sd 2.6) +142.6%
Seq write 512KB q1 1j 328.3 (sd 268.7) 836.3 (sd 34.7) +154.7%
Seq write 512KB q32 1j 519.6 (sd 516.6) 1266.7 (sd 1.5) +143.8%
Seq write 1MB q32 1j 515.1 (sd 517.8) 1265.3 (sd 2.1) +145.6%
Rnd rdwr 4K..1MB q8 4j 355.3 (sd 64.7) 840.3 (sd 3.1) +136.5%
2. pci_endpoint_test
READ_TEST and WRITE_TEST passed with v4 on all three testbeds.
3. R-Car S4 (PCIe Gen4 x2, DWC PCIe controller v5.20a, eDMA) stress
The non-upstream NTB transport/netdev saturated four HW channels in
each direction concurrently from multiple QPs. With v4, it still
reached EP->RC 18.7 Gbit/s and RC->EP 15.7 Gbit/s with iperf3 -ub0,
while keeping the DMA controller almost continuously busy.
[1] K3 was tested on an Ubuntu kernel with the prerequisite patches
backported (+ some tweaks on top), not directly on the upstream
base. The "Before" and "After" kernels differ only by this series.
[2] The current base includes commit 35de39e4511f ("dmaengine: dw-edma:
Defer channel IRQ handling to workqueue"), which defers channel IRQ
handling to an unbound workqueue. Under CPU pressure, some workloads
can see delayed completion processing and idle gaps between
descriptors. Dynamic appends largely hide those gaps by keeping the
engine fed.
A BH workqueue would improve the nvmet_pci_epf workload, but it is
not a complete replacement because BH workqueues are per-CPU.
Retaining cross-channel CPU distribution would require the driver to
choose target CPUs and account for housekeeping masks and CPU
hotplug. That is too much complexity for a quick fix, so I have not
posted a separate change.
This partly explains the unusually large eDMA gain in these tests.
Best regards,
Koichiro
---
Changes in v4:
- Use spinlock_t for LL event serialization, select the event lock
through a per-channel pointer, and use the common accessor in both
interrupt providers. (Frank)
- Put the remote LL publication read-back back in the high-level
doorbell helper, while keeping the provider callback as a raw
doorbell write. (Frank)
- Arm the stopped-tail recheck after a stopped channel is kicked,
reserve its zero sentinel across jiffies wrap, and keep the recheck
available without optional direction recovery. (Sashiko)
- Keep retrying failed direction resets with cancelable delayed work
instead of giving up after a fixed number of attempts. (Sashiko)
- Other smaller fixes, patch-ordering changes, and wording updates are
listed in the per-patch changelogs. (Frank, Sashiko)
Changes in v3:
- Main changes:
* Serialize IRQ event capture and ABORT handling with channel restart,
so deferred events cannot cross into a later hardware run. (Sashiko)
* Record each descriptor's physical LL range and validate it before
consuming IRQ-paired progress. Keep ring accounting valid across
termination, abort, and full resynchronization. (Frank, Sashiko)
* Rework stopped-tail and STOP/PAUSE handling: re-sample LLP before
restart, add one bounded eDMA recheck, and drain published entries
before completing a request. (Frank, Sashiko)
* Move LL interrupt placement into the common core and use the same
four-entry interval for eDMA and HDMA. (Frank)
* Restrict recovery to providers with usable direction-reset
operations and serialize it with IRQ/ABORT handling and teardown.
Rebuild all exposed LL contexts and flush remote LL writes before
re-enabling the engine. (Sashiko)
- Many other smaller fixes, refinements, and patch reorganizations
came from the v2 review, especially Sashiko's feedback. Each patch
has its own changelog below the --- separator, so the details are
not repeated here.
Changes in v2:
- v2 is a substantial rework; per-patch changelogs contain the details:
* Rebased onto dmaengine/next.
* Moved the remaining v1 fixes and state serialization changes into
the independent groundwork series, now merged in the base.
* Reworked ring progress around IRQ-paired LLP samples, conservative
running boundaries, and stopped-tail reconciliation.
* Reworked STOP/PAUSE handling and added full ring resynchronization
plus optional direction recovery before enabling dynamic
submission.
* Updated trace support for the reworked paths and added engine
recovery tracing.
v3: https://lore.kernel.org/r/20260727170323.2321369-1-den@xxxxxxxxxxxxx/
v2: https://lore.kernel.org/r/20260723084150.521366-1-den@xxxxxxxxxxxxx/
v1: https://lore.kernel.org/r/20260615154111.2174161-1-den@xxxxxxxxxxxxx/
Frank Li (5):
dmaengine: dw-edma: Add dw_edma_core_ll_cur_idx() to get current LL
entry index
dmaengine: dw-edma: Make DMA link list work as a circular buffer
dmaengine: dw-edma: Move callback result helper before LL helpers
dmaengine: dw-edma: Dynamically append requests while running
dmaengine: dw-edma: Add trace support
Koichiro Den (19):
dmaengine: dw-edma: Add dw_edma_core_ll_clear() to clear LL
control-word
dmaengine: dw-edma: Factor out linked-list transfer start
dmaengine: dw-edma: Dispatch DONE interrupts by channel request
dmaengine: dw-edma: Centralize LL doorbell decisions
dmaengine: dw-edma: Prepare LL progress event handling
dmaengine: dw-edma: Prepare deferred IRQ reporting for LL events
dmaengine: dw-edma: Prepare LL kicks for event serialization
dmaengine: dw-edma: Serialize LL event capture with channel kicks
dmaengine: dw-edma: Keep channels stopped while ABORT is pending
dmaengine: dw-edma: Reclaim issued descriptors from IRQ-paired LL
progress
dmaengine: dw-edma: Add LL interrupt placement policy
dmaengine: dw-edma: Recheck stopped LL channels before restart
dmaengine: dw-edma: Use HDMA watermarks as progress events
dmaengine: dw-edma: Recover stopped channels from tx_status()
dmaengine: dw-edma: Make the LL ring reset a full channel resync
dmaengine: dw-edma: Drain LL entries for STOP and PAUSE
dmaengine: dw-edma: Add engine reset and enable operations
dmaengine: dw-edma: Add engine recovery infrastructure
dmaengine: dw-edma: Detect and recover a stalled eDMA engine
drivers/dma/dw-edma/Makefile | 2 +
drivers/dma/dw-edma/dw-edma-core.c | 1574 ++++++++++++++++++++++---
drivers/dma/dw-edma/dw-edma-core.h | 159 ++-
drivers/dma/dw-edma/dw-edma-trace.c | 4 +
drivers/dma/dw-edma/dw-edma-trace.h | 169 +++
drivers/dma/dw-edma/dw-edma-v0-core.c | 201 +++-
drivers/dma/dw-edma/dw-hdma-v0-core.c | 163 ++-
drivers/dma/dw-edma/dw-hdma-v0-regs.h | 1 +
include/linux/dma/edma.h | 2 +-
9 files changed, 2010 insertions(+), 265 deletions(-)
create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h
base-commit: 0613e7934ee233d726af8d8c89f251a1c4df738d
--
2.51.0