Re: [REGRESSION] drm/sched: FAIR policy causes serious performance degradation at max GPU load on 9070XT

[Luke . Wildhardt]

I will test this later tonight when I get home and report my findings.

If all is well, I do have a 6900XT I can dig out and cross validate.


Also, I attempted to find a synthetic way to induce the condition; the only slight way I found to do so was running furmark tail, but the desktop never froze, only the desktop seemed to get mildly choppy. I have not yet checked against the FIFO scheduler.  I was trying to find a more objective way to trigger the issue, but it appears in my case I got rather (un)lucky with the specific game I was playing the reproduces it so well.

No problem for helping, I reported the issue and since I am capable of helping at the very least test a fix and reporting on it, I feel I have an obligation to, especially since you haven't been able to reproduce the issue on your end.

-------- Original Message --------
On Wednesday, 08/12/26 at 04:38 Tvrtko Ursulin <tvrtko.ursulin@xxxxxxxxxx> wrote:

On 11/08/2026 18:59, Luke.Wildhardt@xxxxxxxxx wrote:
> I want to be very clear here, I was working with Claude Opus on debugging this issue. I am not a good programmer and work in on the hardware side of things, so this is only code I've tested and not of my creation, just hoping to be helpful.

It is really helpful and appreciated! Both that you found the regression
and that you are helping debug and fix it.

> It appears the freezing (and maybe what I'll call micro freezing instead of stuttering) was resolved after an issue was potentially identified in drivers/gpu/drm/scheduler/sched_rq.c
>
> I did a 20m play session attempting to reproduce the issue but was not able to. Previously, I could reproduce this on-demand.
>
> I will copy the writeup it created and the testing we did below. I am only claiming that the freezing had been seemingly resolved, not a particular fitness or correct identification of the issue, that is for developers to judge.
>
> Quotations are an output of a summery I had Claude create.
>
> "
> THE INVARIANT
>
> drm_sched_entity_stats::vruntime is held absolute while the entity is linked
> in the run queue tree, and relative to min_vruntime while it is idle.
> save_vruntime() converts one way on the way out, restore_vruntime() the other
> on the way back in. drm_sched_rq_add_entity() calls restore unconditionally,
> which is only correct if the entity has definitely been through save.
>
>
> THE WINDOW
>
> drm_sched_entity_pop_job():
>
>      spsc_queue_pop(&entity->job_queue);      /* queue now empty */
>
>      drm_sched_rq_pop_entity(entity);         /* takes entity->lock inside */
>
> No lock held across those two lines. A concurrent push seeing the queue empty
> gets first == true, calls add_entity, and restores a vruntime that has not yet
> been saved -- adding min_vruntime to an already-absolute value. If the entity
> is leftmost, get_min_vruntime() returns its own vruntime and it doubles.
>
> It persists: pop_entity then finds the pushed job, takes the next_job branch,
> and update_vruntime() carries the inflated value forward. The entity sits far
> right in the tree and is not selected until min_vruntime catches up, which
> under sustained load may be effectively never.
>
>
> EVIDENCE
>
> WARN_ON_ONCE(!RB_EMPTY_NODE(&entity->rb_tree_node)) at the top of
> add_entity(), after the rq lock, fires reproducibly.
>
> That condition should be unreachable. add_entity() is reached only from the
> "first job" branch of push_job(), and first is true only when the SPSC queue
> was empty. A linked entity implies its last pop_entity() peeked non-empty.
> Empty queue and still linked cannot both hold in any ordered execution. Fresh
> entities are excluded -- entity_init() calls RB_CLEAR_NODE() unconditionally.
>
> Instrumented as a counter instead, with ring and comm logged:
>
>    idle 10s:      0 races /  5662 restores
>    gameplay 60s:  8 races / 43315 restores
>
>    [ 192.754604] vruntime race on ring gfx_0.0.0 (comm kwin_wayla:cs0)
>    [ 226.882135] vruntime race on ring gfx_0.0.0 (comm kwin_wayla:cs0)
>    [ 292.226032] vruntime race on ring gfx_0.0.0 (comm kwin_wayla:cs0)
>
> plus Xwayland:cs0 and vkd3d_queue on gfx, and many more on sdma0/sdma1. The
> gfx_0.0.0 / kwin_wayla:cs0 entries are the compositor submission thread on the
> graphics ring -- the process that dies when the desktop freezes.
>
>
> CHANGE
>
> The principled fix is presumably to close the window by holding entity->lock
> across the pop, which needs a _locked variant of drm_sched_rq_pop_entity().
> I did not attempt that. What I tested checks the invariant instead: a still-
> linked entity never left, so its vruntime is already absolute and its tree
> position valid.
>
> --- a/drivers/gpu/drm/scheduler/sched_rq.c
> +++ b/drivers/gpu/drm/scheduler/sched_rq.c
> @@ drm_sched_rq_add_entity
> -	ts = drm_sched_rq_get_min_vruntime(rq);
> -	ts = drm_sched_entity_restore_vruntime(entity, ts, rq->head_prio);
> -	drm_sched_rq_update_tree_locked(entity, rq, ts);
> +	if (RB_EMPTY_NODE(&entity->rb_tree_node)) {
> +		ts = drm_sched_rq_get_min_vruntime(rq);
> +		ts = drm_sched_entity_restore_vruntime(entity, ts,
> +						       rq->head_prio);
> +		drm_sched_rq_update_tree_locked(entity, rq, ts);
> +	}
>
> On stock 7.2-rc7 with this applied, freezes are gone against a trigger that
> previously reproduced on demand, and perceptible hitching appears gone too.
> The race counter keeps incrementing (35 in the last session against 295744
> restores), so the window still opens at the same rate and is now absorbed --
> the timing has not merely shifted. Extended soak in progress.
> "

Good find! I almost feel obsolete. Perhaps our new AI overlords should
make a pension fund out of the proceeds obtained by training their
models on the decades of our work so us old programmers can safely retire.

Jokes aside, I agree with the above analysis that a better fix would be
to pull things under the lock, and interestingly, Philipp had a lock
widening patch not so long ago but I don't remember what happened it or
how exactly did it look. It would possibly have fixed this problem.

Anyway, I have prepared a branch with the two fixes if you would be kind
enough to give it a spin:

https://cgit.freedesktop.org/~tursulin/drm-intel/log/?h=drm-sched-fair-fixes

In my testing it all looks good. Fingers crossed.

Regards,

Tvrtko

> On Monday, August 10th, 2026 at 9:40 AM, Luke.Wildhardt@xxxxxxxxx <Luke.Wildhardt@xxxxxxxxx> wrote:
>
>> Oops, forgot to reply all in my last message.
>>
>> I did a quick test of the patch before I left this morning, the freezing seems to gone, but when the GPU is under the same condition, the stuttering is more uniform if that makes sense. Granted this was only a 5 minute test but the freeze was rather reliable before, I can do more testing later.
>>
>> I know these things can be hard to describe, if that helps. If there's any profiling I could do or detailed logging that would help don't hesitate to ask.
>>
>>
>>
>> -------- Original Message --------
>> On Monday, 08/10/26 at 06:48 Tvrtko Ursulin <tvrtko.ursulin@xxxxxxxxxx> wrote:
>>
>> On 10/08/2026 13:45, Danilo Krummrich wrote:
>>> On Mon Aug 10, 2026 at 10:28 AM CEST, Philipp Stanner wrote:
>>>>> Reverted Commits:
>>>>>
>>>>> d09339388b77 drm/sched: Remove drm_sched_init_args->num_rqs
>>>>> 2833a0512b4c drm/sched: Remove drm_sched_init_args->num_rqs usage
>>>>> 16e7698bc04d drm/sched: Embed run queue singleton into the scheduler
>>>>> 77a6809f1dc3 drm/sched: Remove FIFO and RR and simplify to a single run queue
>>>>> 45c211ddf92a drm/sched: Switch default policy to fair
>>>>> 2462a0ce23b0 drm/amdgpu: Remove drm_sched_init_args->num_rqs usage
>>>
>>> [...]
>>>
>>>> That might be the wisest thing to do; but let's hear if Tvrtko has an idea for
>>>> a hotfix first.
>>>
>>> -rc7 was released yesterday; even with a working hotfix today it'd be tricky to
>>> ensure the hotfix does not regress other drivers or workloads with 7.2 being
>>> just a few days ahead.
>>>
>>> I suggest to not wait and get the reverts ready.
>>
>> Yes reverts would be safer. I have them in a branch at
>> people.freedesktop.org/~tursulin/drm-intel drm-sched-fair-reverts,
>> mostly straightforward apart from one easy conflict in
>> amdgpu_xcp_release_sched. Smoke tested on Steam Deck looks fine.
>>
>> Having said that, the fix for min_vruntime handling I provided earlier
>> in the thread also looks fine in my testing and is simple. No
>> regressions found with synthetic unit test workloads or messing around
>> on the Steam Deck. But we need to wait to hear from Luke since I haven't
>> been able to repro his report locally yet.
>>
>> Regards,
>>
>> Tvrtko
>>
>>