[BUG] Apparent lost wakeups

[Paul E. McKenney]

The scheduler appears to lose wakeups on both x86 and arm64 systems
when running the rcutorture TREE01 scenario with RCU priority boosting
enabled, but without also enabling testing of RCU priority boosting.
The following pair of commits in my -rcu tree [1] make these two changes:

c7684673c6ca ("EXP rcutorture: Enable RCU_BOOST where supported")
4b08261f7bff ("EXP rcutorture: Disable RCU priority boost testing on non-TREE03 scenarios")

These two changes were motivated by CI systems reporting RCU CPU stall
warnings due to low-priority tasks being preempted for extended periods
of time.  The hope is that undisciplined RCU priority boosting would
eliminate these issues.  (For an example of a disciplined RCU priority
boosting configuration, see the TREE03 scenario.)

The following command reproduces the lost-wakeup problem, which shows
up as RCU CPU stall warnings caused by rcu_torture_reader() kthreads:

tools/testing/selftests/rcutorture/bin/kvm.sh --allcpus --duration 20m --configs '19*TREE01' --bootargs "nohz=off rcutorture.onoff_interval=0 rcutorture.nocbs_nthreads=0 rcutorture.shuffle_interval=0" --trust-make

Replace the "19" in the "19*TREE01" above with the number of CPUs on your
system divided by 8, rounded down.

This issue occurs on both x86 and arm64 systems.

This wakeup failure occurs in other scenarios, but more rarely.  The thing
that makes the above reproducer do better (or worse, depending on your
perspective) is that it is providing only eight CPUs for each guest OS's
17 vCPUs.  This could be considered abusive, but if so, it is abuse that
provides orders-of-magnitude acceleration of a real failure that also
happens without vCPU overcommit.

The following pair of commits add diagnostics that show that the stalled
kthread is consistently preempted on an idle CPU, which is a perfectly
reasonable transient state (for example, during delivery of a resched
IPI), but not a state that should persist for many tens of seconds:

49be2ba9f0fa ("rcu: Add running and boosted indications to RCU task stall dump")
e3bbe32c1fd9 ("EXP rcu: Print CPU that preempted stalled reader wants to run on")

It appears that the cause of this issue is the wakeup racing with the
target CPU entering the idle loop.  The AI-slop patch shown below does
a recheck of flags to detect this condition and exit the idle loop when
detected.  Note that this is *not* a complete fix in the general case
because the wakeup might happen just after this check, and to its credit
the AI agrees that this is not a complete fix.  However, this "fix"
does appear to prevent (or at least severely decrease the probability
of) the problem for the above reproducer because of the nohz=off kernel
boot parameter.

This patch also splats when the recheck detects a racing wakeup, which
demonstrates that this patch is actually doing something.

An alternative and (apparently) equally effective workaround,
involving resched_cpu() executed by the RCU grace-period kthread's
force-quiescent-state processing, may be found in my -rcu tree:

34bace6b7fab ("EXP rcu: Send resched_cpu() to idle CPUs having preempted RCU readers")

So why does is it necessary to disable rcutorture's testing of RCU
priority boosting to reproduce this lost-waitup issue?  Because enabling
this testing means that there are per-CPU kthreads that wake up once
per second, which forces each CPU to notice any runnable tasks.

As always, please let me know how I can help with further debugging.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

Assisted-by: LLM
Not-signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxx>

diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index eb73b65ce6c47..90f2dfc67e2d0 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -279,6 +279,7 @@ static void do_idle(void)
 {
 	int cpu = smp_processor_id();
 	bool got_tick = false;
+	struct rq *rq;
 
 	if (cpu_is_offline(cpu)) {
 		local_irq_disable();
@@ -339,6 +340,24 @@ static void do_idle(void)
 		 */
 		local_irq_disable();
 
+		/*
+		 * Lost-kick safety net: never sleep with genuinely runnable
+		 * work queued.  Exclude delayed-dequeue tasks, which are
+		 * counted in nr_running but not pickable (as in
+		 * wake_affine_idle()).  Lockless reads are fine: a spurious
+		 * hit costs one extra schedule pass, a miss is re-checked
+		 * on the next tick/interrupt.
+		 */
+		rq = this_rq();
+		if (READ_ONCE(rq->nr_running) >
+		    READ_ONCE(rq->cfs.h_nr_queued) - READ_ONCE(rq->cfs.h_nr_runnable)) {
+			WARN_ONCE(1, "CPU %d entering idle with %u runnable tasks queued\n",
+				  cpu, READ_ONCE(rq->nr_running));
+			set_tsk_need_resched(current);
+			local_irq_enable();
+			continue;
+		}
+
 		arch_cpu_idle_enter();
 		rcu_nocb_flush_deferred_wakeup();