[PATCH v7 0/7] dmaengine: dw-edma: Support dynamic LL appends
From: Koichiro Den
Date: Wed Nov 04 2026 - 11:20:17 EST
Hi,
This series lets dw-edma append requests to a running circular linked
list (LL) using PCS-CCS-CB-TCB Producer-Consumer Synchronization. This
reduces stop/start cycles between descriptors.
v7 addresses feedback from Frank and Sashiko on v6, plus termination
bugs found by testing terminate and pause/resume under DMA load.
Based on dmaengine-7.4-rc1 (aff5e6a77fc6).
v7 (Re-)testing
===============
I used three setups for testing:
Id Endpoint Link Host
--- ------------------------ --------- -----------------
(A) RK3588 ROCK 5B Gen3 x4 CIX Sky1 Orion O6
(B) SpacemiT K3 [1] Pico-ITX Gen3 x4 CIX Sky1 Orion O6
(C) R-Car S4 Spider Gen4 x2 R-Car S4 Spider
1. pci-epf-test:
For all (A)~(C), same PASS/FAIL/SKIP set with and without this series.
2. nvmet-pci-epf:
I ran fio on the host against a null_blk target, with runtime=30s,
ramp_time=5s. Each setup was tested five times without the series
(base) and five times with it (new), alternating as
base1->new1->base2->new2->...->base5->new5. The table shows median
throughput in MiB/s.
(A) RK3588 (B) K3 (C) R-Car S4
(5.60a eDMA) (6.30a HDMA) (5.20a eDMA)
---------------------- --------------- --------------- --------------
rand read 4K q1 38-> 29 -24% 49-> 56 +15% 9.2-> 7.7 -16%
rand read 4K q32 169-> 166 -2% 144-> 145 +1% 17-> 19 +16%
rand read 4K q32/4 172-> 169 -2% 145-> 151 +4% 19-> 26 +33%
rand read 128K q1 703-> 814 +16% 833-> 877 +5% 140-> 131 -6%
rand read 128K q32 74->1009 x14 1130->1496 +32% 303-> 391 +29%
rand read 128K q32/4 55-> 879 x16 1773->2251 +27% 431-> 749 +74%
rand read 512K q1 996->1097 +10% 1313->1202 -8% 138-> 141 +2%
rand read 512K q32 57-> 950 x17 1758->2094 +19% 372-> 735 +98%
rand read 512K q32/4 44-> 751 x17 1821->2326 +28% 343-> 766 x2.2
rand write 4K q1 28-> 29 +4% 55-> 50 -10% 1.9-> 2.8 +41%
rand write 4K q32 74-> 72 -2% 119-> 212 +78% 13-> 19 +44%
rand write 4K q32/4 71-> 169 x2.4 118-> 211 +79% 16-> 26 +64%
rand write 128K q1 480-> 475 -1% 751-> 810 +8% 48-> 55 +15%
rand write 128K q32 881->1326 +51% 1708->1892 +11% 290-> 525 +81%
rand write 128K q32/4 888->1306 +47% 1699->1902 +12% 314-> 756 x2.4
seq read 128K q1 796-> 693 -13% 826-> 767 -7% 147-> 132 -10%
seq read 128K q32 68->1064 x16 1141->1509 +32% 310-> 393 +27%
seq read 512K q1 1063->1131 +6% 1268->1212 -4% 136-> 145 +7%
seq read 512K q32 59-> 953 x16 1757->2126 +21% 379-> 679 +79%
seq read 1M q32 44-> 742 x17 1819->2305 +27% 384-> 712 +85%
seq write 128K q1 493-> 464 -6% 821-> 858 +5% 50-> 45 -8%
seq write 128K q32 886->1328 +50% 1704->1898 +11% 300-> 540 +80%
seq write 512K q1 567-> 737 +30% 243-> 255 +5% 109-> 132 +21%
seq write 512K q32 868->1313 +51% 1705->1890 +11% 333-> 736 x2.2
seq write 1M q32 832->1202 +44% 1703->1899 +12% 338-> 778 x2.3
rw read 4K-1M q8/4 175-> 505 x2.9 542-> 706 +30% 186-> 345 +85%
rw write 4K-1M q8/4 180-> 506 x2.8 545-> 708 +30% 190-> 348 +83%
("/4" in "q32/4" means 4 jobs.)
More briefly, geometric mean of the new/base ratios across all runs
in each group (5 runs x N cases):
(A) RK3588 (B) K3 (C) S4
--------------- ---------- ---------- ----------
QD32 (16 cases) x3.91 +12.0% +50.2%
QD1 (9 cases) -2.5% -2.5% -6.6%
rw q8 (2 cases) x2.85 +24.4% +52.0%
Note for (A) and (C):
- commit efadda401c5c ("PCI: dwc: Set DMA_BYPASS bit in outbound
IATU CTRL2 register"), which is not in dmaengine-7.4-rc1, was
applied to both the base and the new kernels. Without it:
- (A) often hit NVMe I/O timeouts in the 4K q32 write cases with
the series applied. fio still exits successfully and the
numbers look plausible, so this is easy to miss unless you
check dmesg.
- (C) was unstable in general.
3. Terminate/pause test during dma load
Following Sashiko's report:
https://lore.kernel.org/r/rbkljnmrf4ctc7rj6qlwhvhemvzvxx6pdxjxsdc2bmicez56ms@bgshyuvsjuqh/
On (A) and (B), I used a locally modified nvmet-pci-epf to
periodically terminate DMA transfers or pause/resume the channels
under fio load. Terminated transfers were retried. Baseline,
terminate-only, pause/resume-only, combined and post-injection tests
passed CRC32C write/readback verification against a memory-backed
null_blk target.
These tests exposed additional termination bugs, which are addressed
in v7.
[1] Based on Ubuntu-riscv-7.3.0-9.9.1 + some other out-of-tree patches,
especially to make EP mode work on K3.
Note
====
Much of the complexity in v1-v5 came from working around channel stalls
on R-Car S4 with the out-of-tree direct-DMA NTB transport. A channel
could remain stopped and ignore further doorbells without reporting an
error. I have not reproduced this particular behavior on other SoCs with
different DWC IP revisions, or with the in-tree DMA clients I tested on
S4. I have not obtained further information to explain it, so it remains
unresolved. More details:
https://lore.kernel.org/r/tkyu5rsxfa7i6rxxw6qxmmwke5qnlxptwx4j2s24fhftsmmf26@gamef2ec4wwm/
I substantially simplified the series since v6 because I was concerned
that keeping all those workarounds would make it too difficult to review
and leave it pending indefinitely. I continue investigating the S4
stalls separately, alongside the work to upstream the more demanding DMA
clients that expose them.
Best regards,
Koichiro Den
---
v6->v7 changes:
- Avoid premature LL reset on stale STOPPED status during termination
(new patch 2).
- Factor out descriptor completion (new patch 3, Frank).
- Handle STOP before LL reclamation to avoid scheduling callbacks
(Sashiko).
- Restart stopped LL channels to drain published entries during
termination.
- Use modulo arithmetic for the running LL boundary and add lockdep
assertions (Frank).
- Keep HDMA watermark interrupts masked for non-LL channels (Sashiko).
- Use the dw_edma_ prefix for tracepoints and the event class
(Sashiko).
- Add Frank's Reviewed-by to patch 1.
- Link to v6: https://lore.kernel.org/all/20261030150742.2586776-1-den@xxxxxxxxxxxxx/
v5->v6 changes:
- Drop the recovery code and the defensive doorbell handling added for
the R-Car S4 stall: IRQ-paired LLP snapshots and event
serialization, the delayed stopped-channel recheck, tx_status()
reclaim, the LL interrupt placement policy, engine reset and stall
recovery. Empirically, the stall so far reproduces only on S4 (5.20a
(GA)) with out-of-tree direct-DMA NTB transport.
- Read progress in the deferred handler instead. A channel counts as
stopped when its status is STOPPED and the transfer size is zero, on
both eDMA and native HDMA.
- Fold the STOP/PAUSE drain into the progress handler.
- Rebase on the circular LL groundwork in dmaengine-7.4-rc1. 14
patches -> 5.
For earlier changes and hardware observations, see v5:
https://lore.kernel.org/r/20260812155721.2807506-1-den@xxxxxxxxxxxxx/
Frank Li (2):
dmaengine: dw-edma: Dynamically append requests while running
dmaengine: dw-edma: Add trace support
Koichiro Den (5):
dmaengine: dw-edma: Add a channel transfer size operation
dmaengine: dw-edma: Avoid resetting an LL on stale stopped status
dmaengine: dw-edma: Factor out descriptor completion
dmaengine: dw-edma: Reclaim LL entries from hardware progress
dmaengine: dw-edma: Use HDMA watermark interrupts as LL progress
drivers/dma/dw-edma/Makefile | 2 +
drivers/dma/dw-edma/dw-edma-core.c | 257 ++++++++++++++++++++++----
drivers/dma/dw-edma/dw-edma-core.h | 8 +
drivers/dma/dw-edma/dw-edma-trace.c | 4 +
drivers/dma/dw-edma/dw-edma-trace.h | 117 ++++++++++++
drivers/dma/dw-edma/dw-edma-v0-core.c | 6 +
drivers/dma/dw-edma/dw-hdma-v0-core.c | 68 +++++--
drivers/dma/dw-edma/dw-hdma-v0-regs.h | 1 +
8 files changed, 413 insertions(+), 50 deletions(-)
create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h
base-commit: aff5e6a77fc69a7c490246465940d18422d7c631
--
2.54.0