Re: [BUG] Apparent lost wakeups

[K Prateek Nayak]

Hello Paul,

On 10/13/2026 12:11 AM, Paul E. McKenney wrote:
> The scheduler appears to lose wakeups on both x86 and arm64 systems
> when running the rcutorture TREE01 scenario with RCU priority boosting
> enabled, but without also enabling testing of RCU priority boosting.
> The following pair of commits in my -rcu tree [1] make these two changes:
> 
> c7684673c6ca ("EXP rcutorture: Enable RCU_BOOST where supported")
> 4b08261f7bff ("EXP rcutorture: Disable RCU priority boost testing on non-TREE03 scenarios")
> 
> These two changes were motivated by CI systems reporting RCU CPU stall
> warnings due to low-priority tasks being preempted for extended periods
> of time.  The hope is that undisciplined RCU priority boosting would
> eliminate these issues.  (For an example of a disciplined RCU priority
> boosting configuration, see the TREE03 scenario.)
> 
> The following command reproduces the lost-wakeup problem, which shows
> up as RCU CPU stall warnings caused by rcu_torture_reader() kthreads:
> 
> tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 20m --configs '19*TREE01' --bootargs "nohz=off rcutorture.onoff_interval=0 rcutorture.nocbs_nthreads=0 rcutorture.shuffle_interval=0" --trust-make

tl;dr

I could hit that WARN_ON() from your debug patch with the above test
even under rq_lock. Seemed like a missing resched_curr() but it
turned out to also be related to rq->next_class interaction.

I haven't been able to hit the warnings with the below diff. Could
you please try this:

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0f63c30b4324..f2f8af3cdfb5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -11239,6 +11245,11 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		if (p->sched_class->switched_to)
 			p->sched_class->switched_to(rq, p);
 
+		if (ctx->queued && sched_class_above(p->sched_class, rq->next_class)) {
+			rq->next_class = p->sched_class;
+			resched_curr(rq);
+		}
+
 		if (ctx->running) {
 			/*
 			 * If this was a class promotion; let the old class
---

Larger explanation
==================

I added more debug data and I see what happens at time of last wakeup:

    Last wakeup comm: rcu_torture_rea

    rq->next_class: 000000008d901c4b
    p->sched_class: 000000008d901c4b

    next_class:     idle?(0) fair?(1) rt?(0) dl?(0) stop?(0)
    p->sched class: idle?(0) fair?(1) rt?(0) dl?(0) stop?(0)

    need_resched_set_after_wakeup: 1


And then at the time when warning hits, we have:

    h_nr_runnable: 0
    h_nr_queued: 0
    rt_nr_running: 1
    rr_nr_running: 0
    dl_nr_running: 0

Looks like the task that is enqueued is RT but the enqueue context saw
it as a fair task which pointed to __sched_setscheduler().

When pick_task_fair() drops rq_lock() for newidle balance, it is
possible to queue a task and switch it to a higher class before the
CPU doing newidle balance grabs the rq_lock() again.

It then sees rq->next_class is still fair and only checks
"rq->cfs.h_nr_queued" which is now 0 because that task was moved to
RT. The task was not running so we neither upgraded the next_class,
nor did we do a resched.

sched_balance_newidle() returns 0 and since we are already at
pick_task_fair(), we think nothing has queued above us and bail out.
The above diff fixes rq->next_class and calls a resched_curr()
when the task in upgraded but not yet running.

Lightly tested with TREE01 which passes without complains after
adding the above diff. Your mileage may vary :-)

-- 
Thanks and Regards,
Prateek

=========================
Artifact: Full debug diff
=========================

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 0f63c30b4324..f2f8af3cdfb5 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2305,6 +2305,10 @@ void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags)
 {
 	struct task_struct *donor = rq->donor;
 
+	rq->last_comm = p->comm;
+	rq->last_class = rq->next_class;
+	rq->wakeup_class = p->sched_class;
+
 	if (p->sched_class == rq->next_class) {
 		rq->next_class->wakeup_preempt(rq, p, flags);
 
@@ -2314,6 +2318,8 @@ void wakeup_preempt(struct rq *rq, struct task_struct *p, int flags)
 		rq->next_class = p->sched_class;
 	}
 
+	rq->nr_set_after_wakeup = test_tsk_need_resched(rq->curr);
+
 	/*
 	 * A queue event has occurred, and we're going to schedule.  In
 	 * this case, we can save a useless back to back clock update.
@@ -11239,6 +11245,11 @@ void sched_change_end(struct sched_change_ctx *ctx)
 		if (p->sched_class->switched_to)
 			p->sched_class->switched_to(rq, p);
 
+		if (ctx->queued && sched_class_above(p->sched_class, rq->next_class)) {
+			rq->next_class = p->sched_class;
+			resched_curr(rq);
+		}
+
 		if (ctx->running) {
 			/*
 			 * If this was a class promotion; let the old class
diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 76f3c84ca684..7a0f2f701d56 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -279,6 +279,7 @@ static void do_idle(void)
 {
 	int cpu = smp_processor_id();
 	bool got_tick = false;
+	struct rq *rq;
 
 	if (cpu_is_offline(cpu)) {
 		local_irq_disable();
@@ -306,6 +307,7 @@ static void do_idle(void)
 	tick_nohz_idle_enter();
 
 	while (!need_resched()) {
+		bool redo = false;
 
 		/*
 		 * Interrupts shouldn't be re-enabled from that point on until
@@ -339,6 +341,69 @@ static void do_idle(void)
 		 */
 		local_irq_disable();
 
+		/*
+		 * Lost-kick safety net: never sleep with genuinely runnable
+		 * work queued.  Exclude delayed-dequeue tasks, which are
+		 * counted in nr_running but not pickable (as in
+		 * wake_affine_idle()).  Lockless reads are fine: a spurious
+		 * hit costs one extra schedule pass, a miss is re-checked
+		 * on the next tick/interrupt.
+		 */
+		rq = this_rq();
+
+		scoped_guard (rq_lock, rq) {
+			struct rq_flags *rf = &scope.rf;
+
+			if ((READ_ONCE(rq->nr_running) >
+			    READ_ONCE(rq->cfs.h_nr_queued) - READ_ONCE(rq->cfs.h_nr_runnable)) &&
+			    !need_resched()) {
+				WARN_ONCE(1, "CPU %d entering idle with %u runnable tasks queued\n",
+					  cpu, READ_ONCE(rq->nr_running));
+
+				pr_warn("Last comm: %s\n", rq->last_comm);
+				pr_warn("Last class: %p\n", rq->last_class);
+				pr_warn("Wakeup class: %p\n", rq->wakeup_class);
+
+				pr_warn("Last class: idle?(%d) fair?(%d) rt?(%d) dl?(%d) stop?(%d)\n",
+						rq->last_class == &idle_sched_class,
+						rq->last_class == &fair_sched_class,
+						rq->last_class == &rt_sched_class,
+						rq->last_class == &dl_sched_class,
+						rq->last_class == &stop_sched_class);
+
+				pr_warn("Wake class: idle?(%d) fair?(%d) rt?(%d) dl?(%d) stop?(%d)\n",
+						rq->wakeup_class == &idle_sched_class,
+						rq->wakeup_class == &fair_sched_class,
+						rq->wakeup_class == &rt_sched_class,
+						rq->wakeup_class == &dl_sched_class,
+						rq->wakeup_class == &stop_sched_class);
+
+				pr_warn("nr_set_after_wakeup: %d\n", rq->nr_set_after_wakeup);
+				pr_warn("h_nr_runnable: %u\n", rq->cfs.h_nr_runnable);
+				pr_warn("h_nr_queued: %u\n", rq->cfs.h_nr_queued);
+
+				pr_warn("rt_nr_running: %u\n",  rq->rt.rt_nr_running);
+				pr_warn("rr_nr_running: %u\n",  rq->rt.rr_nr_running);
+				pr_warn("dl_nr_running: %u\n",  rq->dl.dl_nr_running);
+
+
+				if (rq->cfs.h_nr_queued) {
+					update_rq_clock(rq);
+					pr_warn("Pick on fair: %p\n", fair_sched_class.pick_task(rq, rf));
+				}
+
+				// set_tsk_need_resched(current);
+				// redo = true;
+				break;
+			}
+		}
+
+		if (redo) {
+			local_irq_enable();
+			continue;
+		}
+
+
 		arch_cpu_idle_enter();
 		rcu_nocb_flush_deferred_wakeup();
 
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 7d2ec527b8a2..7e685cc51e43 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1253,6 +1253,11 @@ struct rq {
 	unsigned long		next_balance;
 	struct mm_struct	*prev_mm;
 
+	char			*last_comm;
+	const struct sched_class	*last_class;
+	const struct sched_class	*wakeup_class;
+	int			nr_set_after_wakeup;
+
 	/*
 	 * The following fields of clock data are frequently referenced
 	 * and updated together, and should go on their own cache line.
---