[PATCH v8 0/7] dmaengine: dw-edma: Support dynamic LL appends

[Koichiro Den]

Hi,

This series lets dw-edma append requests to a running circular linked
list (LL) using PCS-CCS-CB-TCB Producer-Consumer Synchronization. This
reduces stop/start cycles between descriptors.

v8 fixes the premature LL completion reported by Sashiko on v7 patch 6
and picks up Frank's Reviewed-by tags (dropped on patches 2 and 4 due to
updates). I reran the tests, with additional tuning on K3 and S4 to
reduce run-to-run variation. See below.

Based on dmaengine-7.4-rc1 (aff5e6a77fc6).


v8 (Re-)testing
===============

I used three setups for testing:

  Id   Endpoint                  Link       Host
  ---  ------------------------  ---------  -----------------
  (A)  RK3588 ROCK 5B            Gen3 x4    CIX Sky1 Orion O6
  (B)  SpacemiT K3 [1] Pico-ITX  Gen3 x4    CIX Sky1 Orion O6
  (C)  R-Car S4 Spider           Gen4 x2    R-Car S4 Spider

1. pci-epf-test:

   For all (A)~(C), same PASS/FAIL/SKIP set with and without this series.

2. nvmet-pci-epf:

   I ran fio on the host against a null_blk target, with runtime=30s,
   ramp_time=5s. Each setup was tested five times without the series
   (base) and five times with it (new), alternating as
   base1->new1->base2->new2->...->base5->new5. The table shows median
   throughput in MiB/s.

                           (A) RK3588       (B) K3           (C) R-Car S4
                           (5.60a eDMA)     (6.30a HDMA)     (5.20a eDMA)
   ----------------------  ---------------  ---------------  -----------------
   rand read     4K q1       36->  35  -3%    62->  52 -16%  12.1->11.7  -3%
   rand read     4K q32     180-> 162 -10%   169-> 164  -3%  48.4->70.7 +46%
   rand read     4K q32/4   188-> 160 -15%   169-> 164  -3%  66.1->76.9 +16%
   rand read   128K q1      718-> 661  -8%   942-> 950  +1%   207-> 203  -2%
   rand read   128K q32      82->1199  x15  1140->1590 +39%   696-> 917 +32%
   rand read   128K q32/4    51->1248  x24  1810->2428 +34%  1243->1328  +7%
   rand read   512K q1     1039->1034   0%  1281->1287   0%   143-> 196 +37%
   rand read   512K q32      58->1312  x23  1800->2368 +32%  1217->1294  +6%
   rand read   512K q32/4    46-> 953  x21  1816->2528 +39%  1288->1371  +6%
   rand write    4K q1       34->  26 -22%    58->  48 -17%  11.7->11.3  -3%
   rand write    4K q32      70-> 148 x2.1   117-> 220 +88%  40.6->47.1 +16%
   rand write    4K q32/4    69-> 151 x2.2   115-> 219 +90%  43.3->62.0 +43%
   rand write  128K q1      466-> 474  +2%   758-> 781  +3%   142-> 144  +1%
   rand write  128K q32     888->1310 +48%  1686->1921 +14%   557-> 696 +25%
   rand write  128K q32/4   900->1287 +43%  1686->1920 +14%   733-> 860 +17%
   seq read    128K q1      663-> 621  -6%   905-> 836  -8%   203-> 207  +2%
   seq read    128K q32      69->1184  x17  1150->1590 +38%   699-> 938 +34%
   seq read    512K q1      948->1033  +9%  1304->1276  -2%   146-> 193 +32%
   seq read    512K q32      55->1335  x24  1804->2345 +30%  1072->1167  +9%
   seq read      1M q32      49->1021  x21  1841->2484 +35%  1249->1330  +6%
   seq write   128K q1      504-> 464  -8%   810-> 833  +3%   138-> 140  +1%
   seq write   128K q32     897->1307 +46%  1689->1922 +14%   565-> 704 +25%
   seq write   512K q1      569-> 715 +26%   271-> 306 +13%   113-> 172 +52%
   seq write   512K q32     899->1294 +44%  1687->1921 +14%   737-> 861 +17%
   seq write     1M q32     843->1194 +42%  1685->1918 +14%   747-> 876 +17%
   rw read    4K-1M q8/4    193-> 529 x2.7   531-> 727 +37%   453-> 499 +10%
   rw write   4K-1M q8/4    198-> 530 x2.7   532-> 728 +37%   453-> 500 +10%
   ("/4" in "q32/4" means 4 jobs.)

   More briefly, geometric mean of the new/base ratios across all runs
   in each group (5 runs x N cases):

                      (A) RK3588        (B) K3             (C) S4
     ---------------  ----------------- ------------------ ----------
     QD32 (16 cases)  x4.46             +28.2%             +20.0%
                      (x4.06 .. x4.81)  (+27.2% .. +28.7%) (+16.9% .. +22.8%)

     QD1   (9 cases)  -0.9%             +0.2%              +11.5%
                      (-10.8% .. +7.3%) (-3.1% .. +4.6%)   (+5.1% .. +16.5%)

     rw q8 (2 cases)  x2.70             +36.8%             +10.6%
                      (x2.48 .. x3.13)  (+34.9% .. +38.2%) (+9.7% .. +12.4%)

   Parentheses show the min and max of the 5 per-round geomeans, with
   each round comprising one base/new pair.

   Note for (B) and (C):
   - To reduce run-to-run variation, both base and new kernels used
     nohz=off on the endpoint, and the DMA IRQ was pinned to CPU4.

   Note for (A) and (C):
   - commit efadda401c5c ("PCI: dwc: Set DMA_BYPASS bit in outbound
     IATU CTRL2 register"), which is not in dmaengine-7.4-rc1 yet, was
     applied to both the base and the new kernels. Without it:
     - (A) often hit NVMe I/O timeouts in the 4K q32 write cases with
       the series applied. fio still exits successfully and the
       numbers look plausible, so this is easy to miss unless you
       check dmesg.
     - (C) was unstable in general.

3. Terminate/pause test during dma load

   Following Sashiko's report (against v6):
   https://lore.kernel.org/r/rbkljnmrf4ctc7rj6qlwhvhemvzvxx6pdxjxsdc2bmicez56ms@bgshyuvsjuqh/

   On (A) and (B), I used a locally modified nvmet-pci-epf to
   periodically terminate DMA transfers or pause/resume the channels
   under fio load. Terminated transfers were retried. Baseline,
   terminate-only, pause/resume-only, combined and post-injection tests
   passed CRC32C write/readback verification against a memory-backed
   null_blk target.

[1] Based on Ubuntu-riscv-7.3.0-9.9.1 + some other out-of-tree patches,
    especially to make EP mode work on K3.


Note
====

Much of the complexity in v1-v5 came from working around channel stalls
on R-Car S4 with the out-of-tree direct-DMA NTB transport. A channel
could remain stopped and ignore further doorbells without reporting an
error. I have not reproduced this particular behavior on other SoCs with
different DWC IP revisions, or with the in-tree DMA clients I tested on
S4. I have not obtained further information to explain it, so it remains
unresolved. More details:
https://lore.kernel.org/r/tkyu5rsxfa7i6rxxw6qxmmwke5qnlxptwx4j2s24fhftsmmf26@gamef2ec4wwm/

I substantially simplified the series since v6 because I was concerned
that keeping all those workarounds would make it too difficult to review
and leave it pending indefinitely. I continue investigating the S4
stalls separately, alongside the work to upstream the more demanding DMA
clients that expose them.


Best regards,
Koichiro Den
---
v7->v8 changes:
  - Read LLP before checking for a stop to avoid premature completion
    (Sashiko).
  - Resample after a stopped result, and use the same sample for
    completion and termination.
  - Reclaim delayed completions while paused without restarting the
    channel.
  - Pick up Frank's R-b on patches 3 and 5 (dropped from patches 2/4 due
    to code updates).
  - Update test numbers after rerunning with K3/S4 jitter tuning.
  - Link to v7: https://lore.kernel.org/r/20261104161227.1742005-1-den@xxxxxxxxxxxxx/

v6->v7 changes:
  - Avoid premature LL reset on stale STOPPED status during termination
    (new patch 2).
  - Factor out descriptor completion (new patch 3, Frank).
  - Handle STOP before LL reclamation to avoid scheduling callbacks
    (Sashiko).
  - Restart stopped LL channels to drain published entries during
    termination.
  - Use modulo arithmetic for the running LL boundary and add lockdep
    assertions (Frank).
  - Keep HDMA watermark interrupts masked for non-LL channels (Sashiko).
  - Use the dw_edma_ prefix for tracepoints and the event class
    (Sashiko).
  - Add Frank's Reviewed-by to patch 1.
  - Link to v6: https://lore.kernel.org/all/20261030150742.2586776-1-den@xxxxxxxxxxxxx/

v5->v6 changes:
  - Drop the recovery code and the defensive doorbell handling added for
    the R-Car S4 stall: IRQ-paired LLP snapshots and event
    serialization, the delayed stopped-channel recheck, tx_status()
    reclaim, the LL interrupt placement policy, engine reset and stall
    recovery. Empirically, the stall so far reproduces only on S4 (5.20a
    (GA)) with out-of-tree direct-DMA NTB transport.
  - Read progress in the deferred handler instead. A channel counts as
    stopped when its status is STOPPED and the transfer size is zero, on
    both eDMA and native HDMA.
  - Fold the STOP/PAUSE drain into the progress handler.
  - Rebase on the circular LL groundwork in dmaengine-7.4-rc1. 14
    patches -> 5.

For earlier changes and hardware observations, see v5:
https://lore.kernel.org/r/20260812155721.2807506-1-den@xxxxxxxxxxxxx/


Frank Li (2):
  dmaengine: dw-edma: Dynamically append requests while running
  dmaengine: dw-edma: Add trace support

Koichiro Den (5):
  dmaengine: dw-edma: Add a channel transfer size operation
  dmaengine: dw-edma: Avoid resetting an LL on stale stopped status
  dmaengine: dw-edma: Factor out descriptor completion
  dmaengine: dw-edma: Reclaim LL entries from hardware progress
  dmaengine: dw-edma: Use HDMA watermark interrupts as LL progress

 drivers/dma/dw-edma/Makefile          |   2 +
 drivers/dma/dw-edma/dw-edma-core.c    | 286 ++++++++++++++++++++++----
 drivers/dma/dw-edma/dw-edma-core.h    |   8 +
 drivers/dma/dw-edma/dw-edma-trace.c   |   4 +
 drivers/dma/dw-edma/dw-edma-trace.h   | 117 +++++++++++
 drivers/dma/dw-edma/dw-edma-v0-core.c |   6 +
 drivers/dma/dw-edma/dw-hdma-v0-core.c |  68 +++++-
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |   1 +
 8 files changed, 442 insertions(+), 50 deletions(-)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h


base-commit: aff5e6a77fc69a7c490246465940d18422d7c631
-- 
2.54.0