[PATCH v2 00/19] dmaengine: dw-edma: Support dynamic LL appends

[Koichiro Den]

The existing driver treats each linked list as a finite transfer. It
waits for completion before publishing more work, so short and concurrent
requests repeatedly stop and restart the channel.

Turn the linked list into a circular producer/consumer ring and allow
new requests to be published while a channel is running. The hardware can
then continue into newly published entries instead of waiting for
software to rebuild and restart the list for every transfer.

v2 incorporates the v1 review feedback and a substantial rework based on
failures observed under intensive multi-channel traffic.

  v1:
  https://lore.kernel.org/r/20260615154111.2174161-1-den@xxxxxxxxxxxxx/

  Frank's RFT:
  https://lore.kernel.org/r/20260109-edma_dymatic-v1-0-9a98c9c98536@xxxxxxx/

Why this is more than a ring change
===================================

Once entries from several DMAengine descriptors coexist in the ring, LL
ownership and descriptor completion no longer share one boundary.
Software must recycle entries as the hardware consumes them, but complete
a descriptor only after all of its entries have been consumed.

DMA_LLP provides that progress evidence, but its meaning depends on where
the channel was when it was sampled. A stopped sample points to the next
entry. A running sample can move ahead of payload completion, so keep it
one entry behind. Although the HDMA databook describes a watermark LLP as
a recycling boundary, testing on a DWC HDMA 6.30a integration showed that
using it directly for DMAengine completion let clients release mappings
while the hardware still accessed them. The series therefore records
IRQ-paired LLP samples and uses the same conservative running boundary for
both eDMA and HDMA.

A running channel consumes newly published entries without another
doorbell; a stopped channel with pending entries needs one. The series
keeps that decision in one helper, together with stopped-tail
reconciliation, so publication, completion, and restart use the same view
of the ring.

Recovery
========

The programming sections do not define how a second legacy-eDMA doorbell
is handled while a start or stop transition is still in flight. Register
traces point to two problematic transition windows. Channel Status can
still read STOPPED while a start from an earlier doorbell is in flight.
At the other end, DONE, STOPPED status, and zero transfer size can all be
visible before the internal stop has fully settled.

A doorbell aimed at an apparently stopped channel is suspected of
arriving while the previous start or stop transition is still in flight
inside the engine, leaving the channel stuck. The channel retains pending
entries but reports no progress or error interrupt. The exact internal
trigger is not known, and these observations should not be taken as a
rule for every integration.

Avoid doorbells while a channel reports running, and route every stopped
restart through the common helper. Since the visible registers cannot
prove that an apparent stop has fully settled, also provide an optional
recovery path. Repeated doorbell-path checks detect pending entries
without progress. Recovery gates the direction, waits for active channels
to stop, resets the engine, rebuilds configured LL contexts, and only then
enables it again. If ENGINE_EN does not clear, the direction remains
gated.

The recovery path was never entered during the fio/nvmet_pci_epf runs
used so far; whether much longer runs can reach it is unknown. It is a
safeguard for the intensive multi-channel failure, not part of normal
operation. The first stalled observation only starts the timeout and
still rings the doorbell. Recovery is queued only if a later check, after
the timeout, still sees pending work without progress, and the worker
revalidates it before reset.

Only the eDMA provider currently opts into direction recovery. Native HDMA
does not provide direction-reset operations, while HDMA compatibility
mode has no ENGINE_EN. The recovery trigger and timeout remain empirical
and need coverage on more integrations, as well as confirmation of reset
semantics from Synopsys.

Series flow
===========

- Patches 1-6 establish the circular ring and its interrupt policy.
- Patches 7-14 make progress accounting, completion, STOP/PAUSE, and ring
  reset work with multiple descriptors in flight.
- Patches 15-17 add the optional engine recovery path.
- Patch 18 enables dynamic submission; patch 19 adds tracepoints.

Base
====

This series is based on dmaengine/next: commit 0613e7934ee2 ("dmaengine:
dw-edma: Program endpoint function numbers")

Testing
=======

I retested v2 on the following three testbeds.

- eDMA: RK3588 Rock 5B, DWC PCIe v5.60a
        R-Car S4, DWC PCIe v5.20a
- HDMA: SpacemiT K3, DWC PCIe v6.30a [1]

1. fio

   The fio setup uses nvmet_pci_epf with a null_blk-backed namespace,
   runtime=30s, ramp_time=5s. Before And After were alternated three
   times (B-A-B-A-B-A).

   NOTE: The fio tests use nvmet_pci_epf, which does not keep the DMA
         engine continuously busy. A workload that rarely lets the
         engine stop would show the benefit of dynamic appends more
         directly.

   - eDMA (RK3588)

     NOTE: The eDMA Before results include the separate regression
           described in [2].

     Summary by group (BW delta %)
       all          n=26 mean=+1202.7 median=+157.0 min= +55.6 max=+5551.2
       read         n=14 mean=+2086.1 median=+893.9 min= +62.3 max=+5551.2
       write        n=11 mean=+157.4 median=+130.1 min= +55.6 max=+356.8
       qd32         n=16 mean=+1871.7 median=+345.2 min= +99.6 max=+5551.2
       q1           n= 9 mean=+110.0 median=+108.7 min= +55.6 max=+165.2
       small 4K     n= 6 mean=+220.2 median=+163.2 min=+151.4 max=+356.8
       large >=128K n=20 mean=+1497.4 median=+139.3 min= +55.6 max=+5551.2

     Before mean -> After mean (MiB/s)
       Case                         Before             After              Delta
       ---------------------------  -----------------  -----------------  ------
       Rnd read     4KB q1  1j          19.2 (sd 7.5)      51.0 (sd 9.5)  +165.2%
       Rnd read     4KB q32 1j          88.7 (sd 2.1)    231.7 (sd 17.0)  +161.2%
       Rnd read     4KB q32 4j          88.7 (sd 1.3)    223.0 (sd 38.2)  +151.4%
       Rnd read   128KB q1  1j       420.0 (sd 134.6)   876.7 (sd 325.4)  +108.7%
       Rnd read   128KB q32 1j        142.0 (sd 37.7)   2701.3 (sd 56.2)  +1802.3%
       Rnd read   128KB q32 4j         63.7 (sd 10.6)   3121.7 (sd 27.3)  +4803.1%
       Rnd read   512KB q1  1j        572.0 (sd 35.8)    963.0 (sd 48.1)   +68.4%
       Rnd read   512KB q32 1j         66.8 (sd 10.8)   3097.0 (sd 24.8)  +4536.2%
       Rnd read   512KB q32 4j         55.4 (sd 12.0)   3132.7 (sd 39.3)  +5551.2%
       Rnd write    4KB q1  1j          20.7 (sd 7.0)      52.4 (sd 8.2)  +152.9%
       Rnd write    4KB q32 1j          42.5 (sd 0.9)    194.0 (sd 24.2)  +356.8%
       Rnd write    4KB q32 4j          41.7 (sd 1.6)    180.7 (sd 12.7)  +333.6%
       Rnd write  128KB q1  1j        330.0 (sd 90.1)   600.3 (sd 133.7)   +81.9%
       Rnd write  128KB q32 1j       624.0 (sd 111.5)   1426.3 (sd 43.7)  +128.6%
       Rnd write  128KB q32 4j        659.3 (sd 61.8)   1410.0 (sd 49.5)  +113.9%
       Seq read   128KB q1  1j        260.7 (sd 25.8)   690.3 (sd 184.6)  +164.8%
       Seq read   128KB q32 1j        149.7 (sd 30.9)   2578.3 (sd 60.5)  +1622.7%
       Seq read   512KB q1  1j        578.0 (sd 32.2)    938.0 (sd 46.9)   +62.3%
       Seq read   512KB q32 1j         65.0 (sd 10.6)   3106.7 (sd 21.6)  +4679.5%
       Seq read     1MB q32 1j         57.3 (sd 18.8)   3110.7 (sd 27.1)  +5328.7%
       Seq write  128KB q1  1j        272.3 (sd 62.0)   626.7 (sd 144.1)  +130.1%
       Seq write  128KB q32 1j       597.0 (sd 105.7)   1425.0 (sd 42.5)  +138.7%
       Seq write  512KB q1  1j        476.7 (sd 26.7)    741.7 (sd 25.4)   +55.6%
       Seq write  512KB q32 1j        710.7 (sd 44.0)   1418.7 (sd 47.7)   +99.6%
       Seq write    1MB q32 1j       546.0 (sd 100.4)   1310.0 (sd 39.9)  +139.9%
       Rnd rdwr  4K..1MB q8  4j       247.3 (sd 16.3)   1068.7 (sd 13.7)  +332.1%

   - HDMA (SpacemiT K3)

     Summary by group (BW delta %)
       all          n=26 mean= +89.0 median= +97.7 min= -18.9 max=+152.9
       read         n=14 mean= +78.2 median= +84.0 min= -18.9 max=+145.5
       write        n=11 mean= +96.9 median=+121.6 min=  -0.1 max=+145.0
       qd32         n=16 mean=+114.2 median=+112.2 min= +73.5 max=+145.5
       q1           n= 9 mean= +37.1 median= +18.7 min= -18.9 max=+121.6
       small 4K     n= 6 mean= +87.2 median=+106.8 min=  +8.6 max=+145.5
       large >=128K n=20 mean= +89.5 median= +91.9 min= -18.9 max=+152.9

     Before mean -> After mean (MiB/s)
       Case                         Before             After              Delta
       ---------------------------  -----------------  -----------------  ------
       Rnd read     4KB q1  1j          52.5 (sd 3.5)     57.1 (sd 16.3)    +8.6%
       Rnd read     4KB q32 1j        108.1 (sd 90.9)    265.3 (sd 74.2)  +145.5%
       Rnd read     4KB q32 4j        117.0 (sd 84.9)    273.0 (sd 80.0)  +133.3%
       Rnd read   128KB q1  1j       835.7 (sd 139.6)    678.0 (sd 67.5)   -18.9%
       Rnd read   128KB q32 1j        802.7 (sd 67.7)   1651.3 (sd 11.0)  +105.7%
       Rnd read   128KB q32 4j       931.3 (sd 262.1)   1685.3 (sd 12.7)   +81.0%
       Rnd read   512KB q1  1j       411.0 (sd 144.0)   799.7 (sd 243.1)   +94.6%
       Rnd read   512KB q32 1j       969.0 (sd 210.7)    1681.7 (sd 9.3)   +73.5%
       Rnd read   512KB q32 4j       889.7 (sd 298.0)   1684.0 (sd 17.4)   +89.3%
       Rnd write    4KB q1  1j         46.0 (sd 15.5)     56.4 (sd 21.3)   +22.5%
       Rnd write    4KB q32 1j         117.7 (sd 4.0)    236.3 (sd 46.3)  +100.8%
       Rnd write    4KB q32 4j         113.0 (sd 5.3)    240.3 (sd 47.9)  +112.7%
       Rnd write  128KB q1  1j       606.7 (sd 149.0)   606.3 (sd 139.3)    -0.1%
       Rnd write  128KB q32 1j       533.6 (sd 512.3)   1263.0 (sd 40.4)  +136.7%
       Rnd write  128KB q32 4j       526.0 (sd 514.9)   1263.0 (sd 40.9)  +140.1%
       Seq read   128KB q1  1j       626.3 (sd 247.5)   743.7 (sd 145.5)   +18.7%
       Seq read   128KB q32 1j        783.3 (sd 51.1)   1658.0 (sd 12.5)  +111.7%
       Seq read   512KB q1  1j       464.0 (sd 191.5)   855.3 (sd 147.2)   +84.3%
       Seq read   512KB q32 1j       918.3 (sd 241.8)    1681.7 (sd 9.2)   +83.1%
       Seq read     1MB q32 1j       918.7 (sd 290.5)   1688.0 (sd 12.5)   +83.7%
       Seq write  128KB q1  1j        611.0 (sd 79.8)   624.0 (sd 152.8)    +2.1%
       Seq write  128KB q32 1j       521.8 (sd 515.3)   1261.7 (sd 41.7)  +141.8%
       Seq write  512KB q1  1j       328.3 (sd 268.7)   727.7 (sd 127.6)  +121.6%
       Seq write  512KB q32 1j       519.6 (sd 516.6)   1262.7 (sd 40.1)  +143.0%
       Seq write    1MB q32 1j       515.1 (sd 517.8)   1262.3 (sd 39.8)  +145.0%
       Rnd rdwr  4K..1MB q8  4j       355.3 (sd 64.7)    898.7 (sd 43.0)  +152.9%

2. pci_endpoint_test

   READ_TEST and WRITE_TEST passed on all three testbeds.

3. R-Car S4 (PCIe Gen4 x2, DWC PCIe controller v5.20a, eDMA) stress

   The non-upstream NTB transport/netdev saturated four HW channels in
   each direction concurrently from multiple QPs. It reached EP->RC 18.7
   Gbit/s and RC->EP 15.7 Gbit/s with iperf3 -ub0, while keeping the DMA
   controller almost continuously busy.

[1] K3 was tested on an Ubuntu kernel with the prerequisite patches
    backported, not directly on the upstream base. The "Before" and
    "After" kernels differ only by this series.

[2] During final testing, I found a separate performance regression
    associated with commit 35de39e4511f ("dmaengine: dw-edma: Defer
    channel IRQ handling to workqueue"). In a spot check, reverting it
    from the eDMA Before kernel raised 512 KiB random-read QD32
    throughput from about 67 MiB/s to 2370 MiB/s.

    The large eDMA deltas above are therefore not the standalone gain
    from dynamic appends. They also reflect that queued transfers can
    continue without waiting for the deferred IRQ path to restart the
    channel. A separate fix for that regression is under investigation.

Best regards,
Koichiro
---
Changes in v2:
  - v2 is a substantial rework; per-patch changelogs contain the details:
    * Rebased onto dmaengine/next.
    * Moved the remaining v1 fixes and state serialization changes into
      the independent groundwork series, now merged in the base.
    * Reworked ring progress around IRQ-paired LLP samples, conservative
      running boundaries, and stopped-tail reconciliation.
    * Reworked STOP/PAUSE handling and added full ring resynchronization
      plus optional direction recovery before enabling dynamic
      submission.
    * Updated trace support for the reworked paths and added engine
      recovery tracing.

v1: https://lore.kernel.org/r/20260615154111.2174161-1-den@xxxxxxxxxxxxx/


Frank Li (5):
  dmaengine: dw-edma: Add dw_edma_core_ll_cur_idx() to get current LL
    entry index
  dmaengine: dw-edma: Make DMA link list work as a circular buffer
  dmaengine: dw-edma: Move callback result helper before LL helpers
  dmaengine: dw-edma: Dynamically append requests while running
  dmaengine: dw-edma: Add trace support

Koichiro Den (14):
  dmaengine: dw-edma: Add dw_edma_core_ll_clear() to clear LL
    control-word
  dmaengine: dw-edma: Factor out linked-list transfer start
  dmaengine: dw-edma: Add LL interrupt placement policy
  dmaengine: dw-edma: Dispatch DONE interrupts by channel request
  dmaengine: dw-edma: Centralize LL doorbell decisions
  dmaengine: dw-edma: Reclaim issued descriptors from IRQ-paired LL
    progress
  dmaengine: dw-edma: Use HDMA watermarks as progress events
  dmaengine: dw-edma: Reconcile lost completions from a stopped LLP
    re-sample
  dmaengine: dw-edma: Recover stopped channels from tx_status()
  dmaengine: dw-edma: Make the LL ring reset a full channel resync
  dmaengine: dw-edma: Reset LL state after terminate and abort
  dmaengine: dw-edma: Add engine reset and enable operations
  dmaengine: dw-edma: Add engine recovery infrastructure
  dmaengine: dw-edma: Detect and recover a stalled eDMA engine

 drivers/dma/dw-edma/Makefile          |    2 +
 drivers/dma/dw-edma/dw-edma-core.c    | 1069 ++++++++++++++++++++++---
 drivers/dma/dw-edma/dw-edma-core.h    |  104 ++-
 drivers/dma/dw-edma/dw-edma-trace.c   |    4 +
 drivers/dma/dw-edma/dw-edma-trace.h   |  187 +++++
 drivers/dma/dw-edma/dw-edma-v0-core.c |  122 ++-
 drivers/dma/dw-edma/dw-hdma-v0-core.c |  141 +++-
 drivers/dma/dw-edma/dw-hdma-v0-regs.h |    1 +
 8 files changed, 1497 insertions(+), 133 deletions(-)
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.c
 create mode 100644 drivers/dma/dw-edma/dw-edma-trace.h

base-commit: 0613e7934ee233d726af8d8c89f251a1c4df738d
-- 
2.51.0