Re: [RFC PATCH v3 0/1] drm/amdgpu: MMIO TLB invalidation fallback when KIQ is wedged after S4 resume

[Alex Deucher]

On Tue, Aug 25, 2026 at 9:19 AM Denis Pisarev <pisarevden@xxxxxxxxx> wrote:
>
> Hi Christian, Alex,
>
> Understood regarding the fallback - dropping that approach. A working
> KIQ after S4 resume is the real fix.
>
> Regarding GFXOFF, I agree that a gated-off CP/ME/KIQ would also fail
> the ring test, which aligns with the data. The ring test passes upon
> resuming and sched.ready is set; however, the write-reg/wait-ack
> handshake begins to fail during runtime, starting in the same second
> as the thaw and persisting for hours (80-140 errors per hour, measured
> over 9+ hours). So the KIQ initialises fine, but then stops completing
> those packets.
>
> The GFXOFF attribution comes from a single-variable experiment only:
> same workload, one hibernation cycle each. GFXOFF held off across the
> whole S4 cycle (disabled before hibernating and verified to be still
> off after resuming): zero errors. Also zero errors over 15 minutes
> after re-enabling it post-resuming. Plain S3 suspend never triggers
> it. This is merely a correlation; I cannot prove that it is a
> mechanism. The existing semaphore workaround comment in gmc_v9_0.c
> about losing invalidate acknowledge state across power-gating cycles
> appears to be relevant, but I cannot prove that. I am happy to run
> whatever tracing is required to identify the mechanism from the
> affected machine (RLC/ME state, gfx_off_state sampled at the time of
> failure, KIQ fence timestamps and register dumps).

Can you try this branch?
https://gitlab.freedesktop.org/agd5f/linux/-/commits/tlb_inv_rework?ref_type=heads

Alex

>
> Thanks,
> Denis
>
> On Mon, 24 Aug 2026 at 20:31, Alex Deucher <alexdeucher@xxxxxxxxx> wrote:
> >
> > On Mon, Aug 24, 2026 at 12:30 PM Denis Pisarev <pisarevden@xxxxxxxxx> wrote:
> > >
> > > Hi,
> > >
> > > v3 addresses the two findings from the sashiko-bot review of v2.
> > > Failure data and trigger isolation are unchanged (bugzilla 219492):
> > > S4 resume on Cezanne (gmc_v9, GFXOFF) wedges KIQ TLB flushes at
> > > 80-140/hour for 9+ hours with sched.ready true throughout; holding
> > > GFXOFF off across the S4 cycle produces zero errors.
> > >
> > > 1. [High] "VFs and interrupt contexts silently drop TLB flushes once
> > >    the threshold is reached" - correct, and fixed. In v3 the latch
> > >    only reroutes bare metal process context to MMIO. VFs and IRQ
> > >    contexts keep submitting to KIQ exactly as before this patch, with
> > >    per-failure logging, because they have no MMIO alternative; there
> > >    is no longer any code path that drops a flush without attempting
> > >    and logging.
> > >
> > > 2. [High] "KIQ and MMIO race on the same invalidation engine if KIQ
> > >    recovers" - this remains the documented open question; no code
> > >    change in v3. Our analysis: once latched, this path submits no new
> > >    KIQ commands, so the exposure is limited to already-queued stale
> > >    commands and the recovery transition window. The engine serializes
> > >    requests internally, so the realistic worst case is a lost flush
> > >    request caught by the existing ACK timeout ("Timeout waiting for VM
> > >    flush ACK!"), not silent state corruption. If maintainers consider
> > >    a fence necessary (or a dedicated invalidate engine for the MMIO
> > >    path), guidance on the preferred mechanism would be welcome.
> > >
> > > Full patch history: v1 (initial fallback+counter), v2 (GFXOFF hold,
> > > per-instance counter, irqsave, VF/IRQ restrictions, MES error
> > > propagation) - all from bot review; v3 (this one) fixes the VF/IRQ
> > > drop regression the bot found in v2.
> > >
> > > Also still open from the cover letters: the alternative direction of
> > > fixing the S4 resume ordering itself (RLC/ME vs GFXOFF) instead of a
> > > runtime fallback. Happy to run tracing on the affected hardware.
> > >
> >
> > If that's a problem, that should be fixed.  That said, did you have
> > any success with the branch I proposed?
> >
> > Alex
> >
> > > Patch 1/1 follows.
> > >
> > > Denis Pisarev
> > >
> > > Denis Pisarev (1):
> > >   drm/amdgpu: fall back to MMIO TLB invalidation when KIQ is
> > >     unresponsive
> > >
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu.h     |   2 +
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h |   2 +
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c |  18 ++--
> > >  drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.h |   2 +-
> > >  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c   | 123 +++++++++++++++++++-----
> > >  5 files changed, 114 insertions(+), 33 deletions(-)
> > >
> > > --
> > > 2.55.0
> > >