Re: [RFC PATCH v3 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

[Denis Pisarev]

Hi Christian, Alex,

Understood regarding the fallback - dropping that approach. A working
KIQ after S4 resume is the real fix.

Regarding GFXOFF, I agree that a gated-off CP/ME/KIQ would also fail
the ring test, which aligns with the data. The ring test passes upon
resuming and sched.ready is set; however, the write-reg/wait-ack
handshake begins to fail during runtime, starting in the same second
as the thaw and persisting for hours (80-140 errors per hour, measured
over 9+ hours). So the KIQ initialises fine, but then stops completing
those packets.

The GFXOFF attribution comes from a single-variable experiment only:
same workload, one hibernation cycle each. GFXOFF held off across the
whole S4 cycle (disabled before hibernating and verified to be still
off after resuming): zero errors. Also zero errors over 15 minutes
after re-enabling it post-resuming. Plain S3 suspend never triggers
it. This is merely a correlation; I cannot prove that it is a
mechanism. The existing semaphore workaround comment in gmc_v9_0.c
about losing invalidate acknowledge state across power-gating cycles
appears to be relevant, but I cannot prove that. I am happy to run
whatever tracing is required to identify the mechanism from the
affected machine (RLC/ME state, gfx_off_state sampled at the time of
failure, KIQ fence timestamps and register dumps).

Thanks,
Denis

On Mon, 24 Aug 2026 at 20:31, Alex Deucher <alexdeucher@xxxxxxxxx> wrote:
>
> On Mon, Aug 24, 2026 at 12:30 PM Denis Pisarev <pisarevden@xxxxxxxxx> wrote:
> >
> > Hi,
> >
> > v3 addresses the two findings from the sashiko-bot review of v2.
> > Failure data and trigger isolation are unchanged (bugzilla 219492):
> > S4 resume on Cezanne (gmc_v9, GFXOFF) wedges KIQ TLB flushes at
> > 80-140/hour for 9+ hours with sched.ready true throughout; holding
> > GFXOFF off across the S4 cycle produces zero errors.
> >
> > 1. [High] "VFs and interrupt contexts silently drop TLB flushes once
> >    the threshold is reached" - correct, and fixed. In v3 the latch
> >    only reroutes bare metal process context to MMIO. VFs and IRQ
> >    contexts keep submitting to KIQ exactly as before this patch, with
> >    per-failure logging, because they have no MMIO alternative; there
> >    is no longer any code path that drops a flush without attempting
> >    and logging.
> >
> > 2. [High] "KIQ and MMIO race on the same invalidation engine if KIQ
> >    recovers" - this remains the documented open question; no code
> >    change in v3. Our analysis: once latched, this path submits no new
> >    KIQ commands, so the exposure is limited to already-queued stale
> >    commands and the recovery transition window. The engine serializes
> >    requests internally, so the realistic worst case is a lost flush
> >    request caught by the existing ACK timeout ("Timeout waiting for VM
> >    flush ACK!"), not silent state corruption. If maintainers consider
> >    a fence necessary (or a dedicated invalidate engine for the MMIO
> >    path), guidance on the preferred mechanism would be welcome.
> >
> > Full patch history: v1 (initial fallback+counter), v2 (GFXOFF hold,
> > per-instance counter, irqsave, VF/IRQ restrictions, MES error
> > propagation) - all from bot review; v3 (this one) fixes the VF/IRQ
> > drop regression the bot found in v2.
> >
> > Also still open from the cover letters: the alternative direction of
> > fixing the S4 resume ordering itself (RLC/ME vs GFXOFF) instead of a
> > runtime fallback. Happy to run tracing on the affected hardware.
> >
>
> If that's a problem, that should be fixed.  That said, did you have
> any success with the branch I proposed?
>
> Alex
>
> > Patch 1/1 follows.
> >
> > Denis Pisarev
> >
> > Denis Pisarev (1):
> >   drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
> >     unresponsive
> >
> >  drivers/gpu/drm/amd/amdgpu/amdgpu.h     |   2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h |   2 +
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c |  18 ++--
> >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |   2 +-
> >  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 123 +++++++++++++++++++-----
> >  5 files changed, 114 insertions(+), 33 deletions(-)
> >
> > --
> > 2.55.0
> >