Re: [RFC PATCH 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

[Alex Deucher]

On Wed, Aug 19, 2026 at 3:30 PM Denis Pisarev <pisarevden@xxxxxxxxx> wrote:
>
> Hi,
>
> This RFC follows up on bugzilla 219492 ("failed to write reg 28b4 wait
> reg 28c6" after hibernation) and on the resume_gpu_stable proposal
> rejected in January:
>
>   https://lore.kernel.org/amd-gfx/20260106125929.25214-6-sunlightlinux@xxxxxxxxx/
>
> The rejection argued that (a) kiq sched.ready is managed correctly -
> set false in suspend, set true again only after ring tests pass on
> resume - and (b) the proposed flag would have disabled the KIQ flush
> path permanently. Both points are correct, and this RFC takes a
> different route: nothing is disabled proactively; the driver reacts to
> observed failures.
>
> MEASURED FAILURE MODE (Cezanne iGPU, Ryzen 7 PRO 5850U, 7.1.8, KDE
> Wayland, no ROCm):
>
> - S4 resume -> "failed to write reg 28b4/28c6" + "1a6f4/1a706" pairs
>   begin in the same second as the thaw and continue at 80-140/hour
>   for 9+ hours until reboot. kiq sched.ready is true throughout (the
>   ring test at resume passed).
> - Two user-visible consequences: each failed flush burns the ~5 s KIQ
>   retry window (desktop-wide sluggishness), and the invalidation is
>   then silently dropped (stale TLBs - correctness).
>
> TRIGGER ISOLATION (single-variable tests, one hibernate cycle each):
>
> - stock (GFXOFF allowed across S4):    ~70-140 errors / 30 min of use
> - GFXOFF held off across S4:                    0 errors (30 min)
> - GFXOFF re-enabled 40 min after the clean
>   resume:                                      0 errors (15 min;
>   gating resumption not independently verified)
>
> So the wedge forms in the S4 resume window while GFXOFF is allowed,
> and is consistent with the existing semaphore workaround comment in
> gmc_v9_0.c about losing invalidate-acknowledge state across
> power-gating cycles. A boot-time amdgpu.pg_mask=0 does not even
> initialize on this part (probe -ETIMEDOUT), so runtime debugfs
> control was used.
>
> THE PATCH (1/1):
>
> - amdgpu_gmc_fw_reg_write_reg_wait() returns -ETIME on failure and
>   counts consecutive failures
> - gmc_v9_0_flush_gpu_tlb() uses its existing MMIO path as fallback
>   when the KIQ submit fails (invalidations no longer dropped), and
>   after 3 consecutive failures skips the KIQ path entirely until the
>   counter resets - so wedged systems stop paying 5 s per flush
> - counter resets on every success and in gmc_v9_0_hw_fini(); each
>   suspend/resume cycle re-arms KIQ. No proactive disabling anywhere.
> - gmc_v10/v11/v12 call sites unchanged and compile-safe (int return
>   used as statement); extending the fallback to them is trivial once
>   the approach is agreed.

I think it would be better to get KIQ and MES out of the TLB path.  If
they get hung, we have bigger problems.  Can you give this branch a
spin?
https://gitlab.freedesktop.org/agd5f/linux/-/commits/tlb_inv_rework?ref_type=heads
It switches over to using SDMA and only falls back to MMIO if SDMA is
not available.  If you use MMIO, you need to disallow gfxoff before
you touch the registers over MMIO otherwise you'll hang GFX.

Alex

>
> This mirrors what the sibling PASID path already does (KIQ submit,
> -ETIME on timeout, MMIO variant as fallback), applied to the per-VMID
> path.
>
> OPEN QUESTIONS:
>
> 1. Root cause in the S4 window: RLC/ME not processing KIQ invalidates
>    while GFXOFF is allowed? Happy to run tracing (amdgpu.vm_debug,
>    VM_INVALIDATE ack registers) on the affected hardware on request.
> 2. Should the latch be per-xcc-inst?
> 3. Should amdgpu_gmc_flush_gpu_tlb_pasid()'s KIQ branch share the
>    latch?
>
> Patch 1/1 follows. Comments very welcome - especially on whether the
> failure-driven fallback is the right shape vs. fixing the S4 resume
> ordering directly.
>
> Denis Pisarev
>
> Denis Pisarev (1):
>   drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
>     unresponsive
>
>  drivers/gpu/drm/amd/amdgpu/amdgpu.h     |  2 ++
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 15 +++++++++++----
>  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |  4 +++-
>  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 18 ++++++++++++++----
>  4 files changed, 30 insertions(+), 9 deletions(-)
>
> --
> 2.55.0
>