[PATCH] sched/rt: push a queued task when its priority is boosted
From: Chaosheng Qu
Date: Sat Nov 21 2026 - 06:33:06 EST
A PI boost can raise a queued task above everything that is runnable on
other CPUs while it still cannot run on its own rq, because the task
currently running there has equal or higher priority. prio_changed_rt()
only calls resched_curr() in that case. The task then waits for the
current task to schedule out.
The pull side does not necessarily rescue it: pull_rt_task() only picks a
task whose affinity includes the pulling CPU, so a task whose mask
excludes the CPUs that just became idle is unreachable until the running
task leaves on its own. With RT_PUSH_IPI off this is the difference
between a millisecond migration and a wait for the hog to finish.
Queue a push when the priority of a queued, non-current RT task goes up.
push_rt_task() already declines to move a task that is not pushable and
skips the reschedule path when the task is higher priority than the
current one, so the extra callback only acts when a migration target
exists.
Signed-off-by: Chaosheng Qu <chaosheng.qu.linux@xxxxxxxxx>
---
Steve,
This is the fix we discussed in October. The case that matters is the
queued task that the pull side cannot reach because its mask excludes the
CPUs that just became idle.
Numbers from a 4-vCPU QEMU guest (TCG; KVM is not available here), non-RT
kernel, RT_PUSH_IPI off. L is FIFO 10 with mask {cpu1,cpu3}, X is FIFO 99
hogging cpu1, W is FIFO 90 on cpu2 and blocks on L's PI mutex, R is FIFO 60
on cpu3:
baseline: 1/5 rescued, 802.8 / 1161.0 / 1311.8 / 30.2 / 1280.0 ms
patched: 5/5 rescued, 13.8 / 1.1 / 0.2 / 0.2 / 0.8 ms
The reproducer is a small static C program; I can post it if useful.
kernel/sched/rt.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 85303add7..db93bfff7 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -2499,6 +2499,19 @@ prio_changed_rt(struct rq *rq, struct task_struct *p, u64 oldprio)
*/
if (p->prio < rq->donor->prio)
resched_curr(rq);
+
+ /*
+ * A (PI) boost can make @p the highest priority
+ * pushable task on this rq while it still cannot run
+ * here, because the running task is of equal or higher
+ * priority. If @p's affinity excludes the CPUs that
+ * become idle, the pull side cannot pick it and it may
+ * stay queued until the current task schedules out.
+ * Trigger a push for the new priority.
+ */
+ if (oldprio > p->prio && p->nr_cpus_allowed > 1 &&
+ p->prio >= rq->donor->prio)
+ rt_queue_push_tasks(rq);
}
}
--
2.43.0