Re: [RESEND RFC] sched_ext: Support RT task within sched_ext
From: Tengfei Fan
Date: Fri Oct 23 2026 - 06:42:26 EST
On 10/16/2026 11:42 AM, K Prateek Nayak wrote:
Hello Tengfei,
On 10/15/2026 6:17 PM, Tengfei Fan wrote:
Sorry for resending this as plain text. The previous email was rejectedThis doing everything in sched-ext and eBPF attitude needs to change.
by the sched-ext mailing list because it contained HTML.
Hi all,
We would like to discuss how RT tasks can best coordinate with sched_ext
for cpu placement, cpu balance, and sched load reporting to schedutil
governor. This RFC is intended to continue the discussion in email list
from last LPC[1].
One of the options we would like to explore is allowing sched_ext to
schedule some explicitly selected SCHED_FIFO/SCHED_RR tasks.
We received feedback from Changwoo that the issue is valid, and in
LAVD sched-ext, they tried affining RT tasks to a limited set of CPUs as a
workaround for the scheduler placement, but we do know that it is not a
flexible enough solution.
We also had a discussion with Andrea Righi, that he acknowledged that
he understands the scenario and current problem, and he would suggest
not changing the current core scheduler, but instead using bpf ops to
bring the RT tasks under the sched-ext control. We are trying to explore
more on this path, and welcome to suggest or comment here.
Sure it is good and all to ship quickly but you'll be left with a
sub optimal configuration forever and will always be workign around
that.
If you have some magic sauce to detect which scx threads are high
priority and matters for the KPI, why not just send a signal to a
userspace to demote the task temporarily to SCHED_OTHER and then
later move it back to RT when the situation allows?
You can always use a BPF Map to communicate the pids that need
promotion / demotion right? What is stopping you from doing this?
I thought that was the feedback from folks at LPC.
And if that is not possible, maybe you can convince the folks who
originally used the RT (most likely to get around some old issue)
that the custom scheduler is good enough to not hamper performance
and everything can now be a -20 nice task in sched-ext for
scheduler to know that it must be prioritized.
We are using sched_ext for the current experiments because it gives us a convenient way to test different ideas. If an idea works well, we would be happy to bring the results back to the community and discuss whether a more general solution makes sense.
Simply demoting these tasks to SCHED_OTHER does not fully capture what we need, since we still want these selected RT application threads to retain their RT priority information and behavior. For example, when there is a choice of CPUs, we would prefer them to preempt a less important SCX task rather than an important one, while still allowing them to preempt an important SCX task when there is no better choice.
Following Andrea's suggestion, we will first explore preserving the requested RT policy/priority of these selected application threads as metadata, making them non-RT, and using this metadata for scheduling decisions within sched_ext. We will evaluate whether this can provide the behavior we need, while keeping threads that require native RT behavior in the RT class.
LPC topic:Seems like a generic problem, no?
[1] https://lpc.events/event/20/contributions/2561/
Slides:
[2]
https://lpc.events/event/20/contributions/2561/attachments/2025/4612/2026_LPC_Support_RT_tasks_within_sched-ext_.pdf
Here is a text-based description of the main idea presented at LPC.
If you already have that context, you can skip the section below.
Background
----------
In most of the Android gaming, video playback, and livestreaming test
scenarios, non-RT threads work together with RT threads to construct and
consume a frame. Finishing the frame on time without frame jank is a
key performance KPI. A balanced workload across CPUs with fewer
unnecessary CPU frequency bumps is another key power KPI.
More specifically, taking a gaming scenario for example, UnityMain and
UnityGfxDeviceW are the critical normal threads that have significantly
higher load and usually run on the most performance-efficient cpus in
gold/prime cluster. Other than those high load tasks, the other normal
tasks mostly have similar workloads.
It has RT threads like the display HAL tasks, SurfaceFlinger, and the
GPU related tasks. Those RT tasks have workloads comparable to those
of other non-high-load normal tasks.
In our tests, sched_ext runs in switch-all mode, with
SCX_OPS_SWITCH_PARTIAL unset. It schedules the non-RT tasks
covered by this mode. SCHED_FIFO/SCHED_RR tasks still run in
the native RT class.
Below, we call these non-RT tasks "SCX tasks".
SCHED_DEADLINE tasks are not part of this proposal.
Observed problems
-----------------
cpu placement:
In our tests, RT tasks can run on CPUs used by important SCX tasks.
This can cause the SCX tasks to be preempted, waiting longer to run,
or lose cache locality.
Here is a diagram for easier understanding:
RT task + important SCX task
|
v
same CPU
|
v
CPU contention
|
+-----------+-----------+
| | |
preemption longer wait cache impact
\ | /
\ | /
+---------+---------+
|
v
SCX work may finish later
RT tasks have higher priority, so the preemption itself is
expected. However, from the whole scenario perspective, better CPU
placement could reduce this interference when another suitable CPU
is available.
This is a generic placement problem. We are currently exploring it in the sched_ext context, as sched_ext provides a convenient way to experiment with different ideas. If we learn something useful from these experiments, we would be happy to share the results with the community and further discuss how this could be developed into a more general solution.
Load balance:This is also a problem with fair but fair scheduler seems to have
RT tasks can be scheduled in the same cluster of non-high load scx
tasks. This can also be a preferred placement decision. While the
load balance within the cluster is also hard to maintain.
We sometimes see RT and SCX tasks on the same CPU while other CPUs
can become idle very shortly. We need to check whether those newly
idle CPUs can pull and run the scx tasks. While if RT tasks are also
in the same scx scheduler, it is easier to make decisions from a load
balance perspective.
figured it out well - we have newidle balance to pull tasks and as
far as I know, SCX has ops.cpu_release() callaback that can do a
re-enqueue to a different.
Why is this not enough to move this high priority task away and
preempt something else on another CPU?
For newidle balance, pulling a preempted SCX task to another CPU may add waiting and migration overhead and hurt cache locality. We would prefer to avoid this when better initial placement is possible.
We have not looked closely at ops.cpu_release() yet. We need to evaluate whether it is sufficient for this case.
sched util calculation:Wasn't UCLAMP specifically designed to avoid those bumps if they
We also see a difference in load tracking. Our implementation
uses a frame-window model for SCX tasks, while the RT load signal
in our setup uses PELT.
The SCX task load is flat and steady, while the RT task load
sometimes spikes. An unnecessary CPU frequency bump up can be triggered
by RT task load spikes.
are unnecessary? RT can make sure you preempt first and UCLAMP
can ensure you don't boost frequency unnecessarily.
I think sched-util will request the highest frequency when an RT
task is running but that seems like a more generic problem to
solve.
In our experiments, we observed cases where the SCX workload utilization was already close to a frequency transition point, and some additional RT load could trigger a frequency bump.
We have not yet verified whether an appropriate uclamp configuration can avoid this unnecessary bump while still providing the performance needed by the RT task.
The RT/schedutil behavior itself is a generic issue. We are currently exploring it in the sched_ext context, as sched_ext provides a convenient way to experiment with different ideas. If we learn something useful from these experiments, we would be happy to share the results with the community and further discuss how this could be developed into a more general solution.
Why CPU affinity is not enoughIf you can consider cpu affinities an option, you can also consider
------------------------------
We could use CPU affinity to keep RT and important SCX tasks
on different CPUs. This could reduce the time they compete
for the same CPU.
However, CPU affinity is fixed and strictly limits how tasks can move
between CPUs. If the number of background tasks increases, CPUs perferred
for critical SCX tasks may still have spare capacity. With fixed CPU
affinity, non-critical SCX tasks cannot use those CPUs even when doing
so would help balance the system load.
Ideally, the scheduler should be able to choose CPUs dynamically as
the load changes, while still keeping tasks within their allowed CPU
masks.
CPU affinity may still help, but it does not provide the
flexibility we want here.
demoting them to SCHED_OTHER as an option no?
CPU affinity was not intended as the solution here.
Simply demoting these tasks to SCHED_OTHER would not fully capture what we need, since we still want to retain their requested RT policy/priority as information for scheduling decisions. Following Andrea's suggestion, we will explore keeping these attributes as metadata, making the selected application threads non-RT, and using the metadata within sched_ext to provide the placement behavior we need.
Possible optionsWho gets to decide this selection of threads?
----------------
One option is to let sched_ext schedule some explicitly selected RT
threads together with SCX tasks.
We don't have a final selection mechanism yet. For the initial evaluation of Andrea's suggestion, the selected application threads will be explicitly specified by userspace through a simple configuration, e.g. by PID. This is only intended for validating the approach; we have not settled on how the selection should be made in the final design.
This could let one scheduler choose CPUs for both selected RTI fail to see how "RT task's priority" is respected if you are
threads and SCX tasks.
For example, if other cpu is idle, or running a low priority scx task,
or running a low load scx task, an RT task could use that CPU instead
of a CPU running an important high load SCX task. This could reduce CPU
contention for the frame important KPI and help balance RT and SCX tasks
across CPUs.
Such placement should remain a preference rather than a fixed
rule. It must respect the RT task's priority, timing needs, and
allowed CPU mask. The other CPU must also have enough capacity.
suggesting a SCHED_OTHER task can be prioritized over it.
Also how do you infer a timing needs of RT? From a traditional
theory perspective, if the task is of a higher priority, it
always preempts.
We are not suggesting that a SCHED_OTHER task should have higher priority than an RT task. What we are describing is a placement preference when selecting a CPU for the RT task. For example, when multiple suitable CPUs are available, we would prefer the RT task to preempt a less important SCX task rather than an important one. If there is no better choice, however, the RT task should still be able to preempt the important SCX task.
The term "timing needs" was not well defined here. For SCHED_FIFO/SCHED_RR tasks, we do not have a separate timing requirement beyond the scheduling policy/priority, so this wording was misleading.
It could also allow RT and SCX tasks to use the same utilizationThis can be a generic optimization too right? If RT task has a
model. We would like to study whether this can make the load
signal more stable and reduce late CPU frequency changes.
Another option is to keep all RT tasks in the native RT
class, but provide enough information for the two schedulers
to better coordinate CPU placement. However, this would also require
placement and load synchronization between the RT scheduling class
and the sched_ext class.
choice to be scheduled on a bunch of CPU, maybe there are
heuristics that can be used using nice value, CPU load to better
place them.
This can also be a generic placement optimization. As mentioned above, we are currently exploring it in the sched_ext context, as sched_ext provides a convenient way to experiment with different ideas. If we learn something useful from these experiments, we would be happy to share the results with the community and further discuss how this could be developed into a more general solution.
After the LPC discussion, we'd like to explore option 1 and useI don't like it one bit but what exactly are you planning to do
BPF-only hooks to achieve current needed RT tasks within scx class.
with this BPF op?
Direct RT task placement to another CPU? How is the RT scheduler
supposed to verify if that alternate placement doesn't violate the
RT guarantees? (And please don't say you'll use the ext's dispatch
path to do this)
The suggestion at LPC came from Andrea. We don't consider to directly place native RT tasks on another CPU through the sched_ext dispatch path. As discussed above, we will instead explore preserving the requested RT policy/priority of selected application threads as BPF metadata, making those threads non-RT, and using the metadata for scheduling decisions within sched_ext.
One option is to have all RT tasks with a priority lower than the SCXYou can always demote these tasks from the userspace bit of
helper thread go into sched_ext.
Migration thread, sched-ext enable/disable threads and IRQ threads need
to remain outside sched_ext. With this option, we will need to verify
more common scenarios. We can explore the first option first.
Open questions
--------------
* Can sched deadline can used to optimize those scenarios?
It is possible, of course, so we put it here, welcome to share the
possible configurations of which tasks goes to deadline. However, here
we discuss only the topic of plugging RT tasks into and out of
sched_ext for these scenarios. For different scenarios optimization,
there is of course no single solution. It still can have different
scheduler optimization to make a specific tasks a different sched-class.
* Other ideas?
sched-ext and there is no explanation in this entire write-up
that says why that is bad / insufficient.
As mentioned above, simply demoting these tasks to SCHED_OTHER does not fully capture what we need. We still want to retain the RT priority information of these selected application threads and use it for scheduling decisions. For example, when multiple suitable CPUs are available, we would prefer these tasks to preempt a less important SCX task rather than an important one. If there is no better choice, however, they should still be able to preempt an important SCX task.
We would appreciate feedback on the problem, the possible
directions, and what we should try first.