Re: [ANNOUNCE] linux-7.2-ck1, MuQSS CPU scheduler for linux-7.2

[Con Kolivas]

On Mon, 24 Aug 2026 at 00:28, Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>
> On Sun, 23 Aug 2026 12:08:19 +1000
> Con Kolivas <con@xxxxxxxxxxx> wrote:
>
> > Hi again Steve et. al
> >
> > > > [1] https://lwn.net/Articles/1020596/
> > > > [2] https://docs.google.com/presentation/d/1dtm0AiiTI30gTFeKj95vmSyirYmk5_QiR_lh17l_Moo/edit?usp=sharing
> >
> > I'm now caught up with your presentation as presented in the youtube
> > video linked in the lwn article. Thanks, very informative and
> > thoughtful. I was unable to access the google doc but have requested
> > read access - though I believe it was all presented on the video.
> >
> > Your governor idea is not as dissimilar to plugsched as may appear on
> > the surface. Plugsched built in all the schedulers into the kernel and
> > allowed you to boot the scheduler of your choice at boot time; it was
> > not to just build one scheduler into the kernel. Making it switch on
> > the fly was a pipe-dream goal but since it got shot down in
> > spectacular fashion I did not pursue it further. Its code is also so
> > outdated that literally nothing is of relevance in the current kernel
> > tree.
>
> Yeah, the main reason I call it a "governor" and not "plugable" is
> because I wanted to stress that it's not for any scheduler. It is for
> an environment. Similar to the power management governors. Saying
> "plugable" gives a "plug and play" feel that I wanted to avoid. Ingo's
> complaint back then was that we would have hundreds of schedulers. Now
> with sched_ext, that's exactly what we have. Thus, the governor was an
> idea to bring back collaboration between folks that work in the same
> environment.

Fair enough.
>
> >
> > If you do pursue the governor idea there are a few things worth noting
> > about how high up and broad the hooks need to be.
> > One overhead problem with plugsched was it added a layer of
> > indirection to every single scheduler function call that was shared
> > between different schedulers. The cost of this may be considered
> > either trivially irrelevant or not remotely worth it depending on your
> > viewpoint. A the time I wrote plugsched, Itanic[sic] was still an
> > active architecture and the indirection was considered a huge
> > downside.
>
> I'm not sure you are aware that the Linux kernel today has something
> called a "static_call"[1]. It's a direct function call that can be
> switched at runtime with run-time code modification. It is something
> that I was planning on using.

No, I was not aware, that takes the worst aspect off the table.

>
> > There are four broad aspects to achieving low latency with muqss which
> > all need to be adopted to reproduce its behaviour, in order of
> > decreasing importance.
> > 1. Policy - the simple ordering aspect based on deadline, timeslice
> > interval etc. based on a shared monotonically increasing nanosecond
> > time counter.
> > 2. Shared access to a global queue - BFS did this by having only one
> > queue. MuQSS was created as a way to address scalability concerns by
> > reintroducing separate runqueues. It became clear very quickly that
> > policy alone did not reproduce the behaviour of BFS and that's where
> > the idea for having shared runqueues came about. The more the
> > runqueues were shared, the closer the latency approximated BFS'. The
> > default configuration chooses MC - Multicore. For virtually all
> > desktops and mobile devices that means they all end up with one
> > runqueue anyway. It is pre-configurable in kconfig, but also boot-time
> > selectable.
>
> > 3. Busy and idle load balancing. In MuQSS' case the busy balancing
> > happens by proxy through the next task selection, but idle balancing
> > is handled separately. Mainline handles both of these separately from
> > policy.
>
> For my idea, I would have the governors only affect SCHED_OTHER and
> SCHED_IDLE. The RT and DL schedulers would not be affected. But having
> the governor take over the idle and other tasks, it would have more
> control of what it could do.

Sounds good. MuQSS' multiple runqueues would be better without having
to handle RT tasks as well - there's a corner case where checking
another runqueue for its best task won't find the best SCHED_OTHER
task if there's an RT task ahead of it (though in the desktop it would
probably be one runqueue.)
Switching to a different number of runqueues on the fly would require
some special handling, something like the CPU hotplug paths used for
suspend/resume.

>
> > 4. Highres timer based scheduling to effect the nanosecond timers.
> > This is to disentangle the scheduler's latency dependency on the
> > chosen jiffy Hz which ties all other subsystem components to that
> > resolution and/or overhead.
>
> I believe mainline is going in this direction too.

Sounds good.
>
> >
> > None of these are insurmountable endpoints with enough LLM tokens, but
> > I suspect there will at least be one/some indirection somewhere in the
> > implementation.
>
> Again, indirection is solved by the static calls. When spectre and
> meltdown mitigations were introduced into the kernel, indirect calls
> took a hugh hit. Tracepoints used them quite extensively and it slowed
> down hackbench with the sched_switch tracepoint active by 10%!
>
> I started working on a way to replace indirect calls with a direct call
> that could be modified. Then Josh Poimbeouf took over and Linus didn't
> like the implementation.Finally, Peter Zijlstra got it into the kernel.
>
> They are expensive to change, but for things that do not change often
> (like enabling a tracepoint, picking a KVM implementation, or picking a
> specific scheduler governor) it is really useful. The hackbench
> slowdown disappeared and tracepoints are actually even faster than what
> they were with the indirect calls.

Very good. Looking forward to seeing your progress.
I've split out the muqss patch into two separate patches on a new git
branch, leaving the IO time code as a separate patch, making it
possible to concentrate purely on the scheduler components.
https://github.com/ckolivas/linux/tree/7.2-muqss

>
> -- Steve
>
> [1] https://lwn.net/Articles/815908/

Thanks,
-ck