Re: [RFC PATCH sched_ext/for-7.4 0/5] sched_ext: Make ops.cpu_acquire/release() reliable

[Kuba Piecuch]

On Sun, Oct 18, 2026 at 12:14:44PM +0000, Kuba Piecuch wrote:
> - ops.cpu_acquire() moves from before ops.dispatch() to the point where
>   the CPU actually switches to an SCX task or idle, so ops.dispatch()
>   may now run on a released CPU. A speculative acquire before dispatch
>   can't be made reliable without hooks in every class.

I'd like to reconsider this part of the approach, for two reasons.

First, it changes the semantics of ops.cpu_acquire() relative to the
original implementation. Schedulers could so far assume that the CPU
is acquired whenever ops.dispatch() runs. With this patchset they can't.

Second, it loses a case ops.cpu_release() should cover. Say an RT task
blocks, ops.dispatch() inserts T into the local DSQ, the rq lock gets
dropped, another RT task wakes up on the CPU, the SCX pick returns
RETRY_TASK and the RT task wins. That's an RT -> RT switch, so the CPU
never becomes available, no ops.cpu_release() is invoked and T sits on
the local DSQ until the CPU picks from SCX again.

The standard answer to that would be SCX_ENQ_IMMED, but IMMED isn't
bulletproof either. It relies on rq->next_class being raised through
wakeup_preempt() whenever a higher-class task is about to take the CPU,
and there are several ways around that:

- sched_setscheduler() or a PI boost promoting a queued task to RT/DL:
  sched_change_end() doesn't call wakeup_preempt() for non-running
  tasks, switched_to_rt() only does resched_curr() and switched_to_dl()
  calls wakeup_preempt_dl() directly.

- DL replenishment (dl_task_timer()), CFS bandwidth unthrottling
  (partial mode) and RT group unthrottling enqueue and resched_curr()
  without going through wakeup_preempt().

- When the ext dl_server picks an SCX task, next_class becomes ext
  although RT tasks are queued, so rq_is_open() accepts IMMED inserts.
  Once the server runs out of budget, RT takes over without any wakeup.

- With core scheduling, a forced idle pick resets next_class to idle
  while tasks remain on the local DSQ, so a later RT wakeup goes to
  wakeup_preempt_idle() rather than wakeup_preempt_scx().

In all of these, IMMED tasks end up stranded on the local DSQ behind
the higher-class task.

Making next_class exact would mean touching rt, dl, fair and core, and
still wouldn't cover the dl_server and forced idle cases. What I think
would fix it is a single SCX hook in __schedule(), right after the pick:

	next = pick_next_task(rq, &rf);
	rq->next_class = next->sched_class;
	scx_settle_pick(rq, next);

gated on scx_enabled() and next's class being above ext. At that point
the pick is final, the hook runs on the CPU that's going to run next
(core-sched siblings settle in their own __schedule()), and a preempted
prev as well as anything ops.dispatch() inserted are already on the
local DSQ. If rq->scx.nr_immed is non-zero, the hook schedules a
re-enqueue of the IMMED tasks, whatever made the higher-class task
runnable.

The same hook also allows fixing ops.cpu_acquire/release() while
preserving the original semantics. ops.cpu_acquire() stays in
dispatch_one(), before ops.dispatch(), and the hook releases the CPU
if it's still acquired when the pick lands above ext, with the task
actually picked in args->task. That correctly handles every abandoned
pick (RETRY_TASK, the ext dl_server returning NULL, core-sched
restarts), and in the example with an RT task above ops.cpu_release()
gets to re-enqueue T. The release calls v1 adds to put_prev_task_scx()
and put_prev_task_idle() go away. The rest of v1 (class changes,
hotplug, enable/disable) stays.

Tejun, Peter, would a hook like this be acceptable? The core.c
footprint is one line, it's a static branch when SCX isn't enabled, and
the out-of-line call only happens when the pick goes to a class above
SCX. If you're OK with it, I'd send the IMMED fix first, separately
(Fixes: 98d709cba319 ("sched_ext: Implement SCX_ENQ_IMMED")), and
rebase v2 of this series on top of it.

Thanks,
Kuba