[PATCH v2 2/3] sched/core: Don't take the idle fast path while the core is forced idle
From: Ibrahim Elshahed
Date: Fri Nov 20 2026 - 05:58:54 EST
Since v6.12, with commit 3dcac251b066 ("sched/core: Introduce SM_IDLE
and an idle re-entry fast-path in __schedule()"), an idle CPU with an
empty runqueue skips pick_next_task() when it is rescheduled. Under
core scheduling that pick also ends or re-examines the core's forced
idle.
A core-wide selection chooses a task for an idle sibling and reschedules
it. If the chosen task is migrated away before the sibling schedules
and it was the sibling's only task, the sibling goes back to idle
through the fast path. When that selection also forced the other
sibling idle, because its runnable tasks did not match the chosen
task's cookie, the core stays forced idle with nothing running on it,
and two things go wrong:
- Without the previous patch, the idle sibling's rq->core_pick keeps
pointing at the migrated task, and with schedstats enabled the forced
idle accounting keeps writing through it, even after the task and its
cgroup are freed:
BUG: unable to handle page fault for address: ffff8d6b205f7018
#PF: supervisor write access in kernel mode
#PF: error_code(0x0002) - not-present page
RIP: 0010:__cgroup_account_cputime_field+0x3c/0x70
Call Trace:
__sched_core_account_forceidle+0xd8/0x140
__schedule+0x1390/0x18d0
schedule_idle+0x1a/0x30
cpu_startup_entry+0x29/0x30
start_secondary+0x11e/0x140
- The forced-idle sibling keeps waiting with runnable tasks until some
unrelated event reschedules the core. Likewise, when the forced-idle
CPU itself loses its last task, the reschedule that
sched_core_dequeue() sends to end the forced idle also ends in the
fast path.
Take the SM_IDLE fast path only when the core is not forced idle. A
pending rq->core_pick needs no check of its own: the previous patch
clears it when the picked task is dequeued, so a CPU with an empty
runqueue has none.
Fixes: 3dcac251b066 ("sched/core: Introduce SM_IDLE and an idle re-entry fast-path in __schedule()")
Cc: stable@xxxxxxxxxxxxxxx
Co-developed-by: Roman Kagan <rkagan@xxxxxxxxx>
Signed-off-by: Roman Kagan <rkagan@xxxxxxxxx>
Signed-off-by: Ibrahim Elshahed <lshahed@xxxxxxxxx>
---
kernel/sched/core.c | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index b1718ba5a4d1..d2f7532fc7cf 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -347,6 +347,15 @@ void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags)
resched_curr(rq);
}
+/*
+ * An idle CPU with an empty runqueue must still call pick_next_task() while
+ * the core is forced idle. Only that pick ends or re-examines the forced idle.
+ */
+static inline bool sched_core_idle_needs_pick(struct rq *rq)
+{
+ return sched_core_enabled(rq) && rq->core->core_forceidle_count;
+}
+
static int sched_task_is_throttled(struct task_struct *p, int cpu)
{
if (p->sched_class->task_is_throttled)
@@ -550,6 +559,7 @@ void sched_core_put(void)
static inline void sched_core_enqueue(struct rq *rq, struct task_struct *p) { }
static inline void
sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags) { }
+static inline bool sched_core_idle_needs_pick(struct rq *rq) { return false; }
#endif /* !CONFIG_SCHED_CORE */
@@ -7285,8 +7295,13 @@ static void __sched notrace __schedule(int sched_mode)
*/
prev_state = READ_ONCE(prev->__state);
if (sched_mode == SM_IDLE) {
- /* SCX must consult the BPF scheduler to tell if rq is empty */
- if (!rq->nr_running && !scx_enabled()) {
+ /*
+ * SCX must consult the BPF scheduler to tell if rq is empty,
+ * and core scheduling may need pick_next_task() to end forced
+ * idle.
+ */
+ if (!rq->nr_running && !scx_enabled() &&
+ !sched_core_idle_needs_pick(rq)) {
next = prev;
rq->next_class = &idle_sched_class;
goto picked;
--
2.47.3