[RFC PATCH 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

[Denis Pisarev]

Hi,

This RFC follows up on bugzilla 219492 ("failed to write reg 28b4 wait
reg 28c6" after hibernation) and on the resume_gpu_stable proposal
rejected in January:

  https://lore.kernel.org/amd-gfx/20260106125929.25214-6-sunlightlinux@xxxxxxxxx/

The rejection argued that (a) kiq sched.ready is managed correctly -
set false in suspend, set true again only after ring tests pass on
resume - and (b) the proposed flag would have disabled the KIQ flush
path permanently. Both points are correct, and this RFC takes a
different route: nothing is disabled proactively; the driver reacts to
observed failures.

MEASURED FAILURE MODE (Cezanne iGPU, Ryzen 7 PRO 5850U, 7.1.8, KDE
Wayland, no ROCm):

- S4 resume -> "failed to write reg 28b4/28c6" + "1a6f4/1a706" pairs
  begin in the same second as the thaw and continue at 80-140/hour
  for 9+ hours until reboot. kiq sched.ready is true throughout (the
  ring test at resume passed).
- Two user-visible consequences: each failed flush burns the ~5 s KIQ
  retry window (desktop-wide sluggishness), and the invalidation is
  then silently dropped (stale TLBs - correctness).

TRIGGER ISOLATION (single-variable tests, one hibernate cycle each):

- stock (GFXOFF allowed across S4):    ~70-140 errors / 30 min of use
- GFXOFF held off across S4:                    0 errors (30 min)
- GFXOFF re-enabled 40 min after the clean
  resume:                                      0 errors (15 min;
  gating resumption not independently verified)

So the wedge forms in the S4 resume window while GFXOFF is allowed,
and is consistent with the existing semaphore workaround comment in
gmc_v9_0.c about losing invalidate-acknowledge state across
power-gating cycles. A boot-time amdgpu.pg_mask=0 does not even
initialize on this part (probe -ETIMEDOUT), so runtime debugfs
control was used.

THE PATCH (1/1):

- amdgpu_gmc_fw_reg_write_reg_wait() returns -ETIME on failure and
  counts consecutive failures
- gmc_v9_0_flush_gpu_tlb() uses its existing MMIO path as fallback
  when the KIQ submit fails (invalidations no longer dropped), and
  after 3 consecutive failures skips the KIQ path entirely until the
  counter resets - so wedged systems stop paying 5 s per flush
- counter resets on every success and in gmc_v9_0_hw_fini(); each
  suspend/resume cycle re-arms KIQ. No proactive disabling anywhere.
- gmc_v10/v11/v12 call sites unchanged and compile-safe (int return
  used as statement); extending the fallback to them is trivial once
  the approach is agreed.

This mirrors what the sibling PASID path already does (KIQ submit,
-ETIME on timeout, MMIO variant as fallback), applied to the per-VMID
path.

OPEN QUESTIONS:

1. Root cause in the S4 window: RLC/ME not processing KIQ invalidates
   while GFXOFF is allowed? Happy to run tracing (amdgpu.vm_debug,
   VM_INVALIDATE ack registers) on the affected hardware on request.
2. Should the latch be per-xcc-inst?
3. Should amdgpu_gmc_flush_gpu_tlb_pasid()'s KIQ branch share the
   latch?

Patch 1/1 follows. Comments very welcome - especially on whether the
failure-driven fallback is the right shape vs. fixing the S4 resume
ordering directly.

Denis Pisarev

Denis Pisarev (1):
  drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
    unresponsive

 drivers/gpu/drm/amd/amdgpu/amdgpu.h     |  2 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 15 +++++++++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  4 +++-
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 18 ++++++++++++++----
 4 files changed, 30 insertions(+), 9 deletions(-)

-- 
2.55.0