Re: [PATCH tip/core/rcu 13/22] rcu: Fix grace-period hangs due to race with CPU offline

[Peter Zijlstra]

On Tue, Jun 26, 2018 at 10:10:39AM -0700, Paul E. McKenney wrote:
> Without special fail-safe quiescent-state-propagation checks, grace-period
> hangs can result from the following scenario:
> 
> 1.	CPU 1 goes offline.
> 
> 2.	Because CPU 1 is the only CPU in the system blocking the current
> 	grace period, as soon as rcu_cleanup_dying_idle_cpu()'s call to
> 	rcu_report_qs_rnp() returns.
> 
> 3.	At this point, the leaf rcu_node structure's ->lock is no longer
> 	held: rcu_report_qs_rnp() has released it, as it must in order
> 	to awaken the RCU grace-period kthread.
> 
> 4.	At this point, that same leaf rcu_node structure's ->qsmaskinitnext
> 	field still records CPU 1 as being online.  This is absolutely
> 	necessary because the scheduler uses RCU, and ->qsmaskinitnext

Can you expand a bit on this, where does the scheduler care about the
online state of the CPU that's about to call into arch_cpu_idle_dead()?

> 	contains RCU's idea as to which CPUs are online.  Therefore,
> 	invoking rcu_report_qs_rnp() after clearing CPU 1's bit from
> 	->qsmaskinitnext would result in a lockdep-RCU splat due to
> 	RCU being used from an offline CPU.
> 
> 5.	RCU's grace-period kthread awakens, sees that the old grace period
> 	has completed and that a new one is needed.  It therefore starts
> 	a new grace period, but because CPU 1's leaf rcu_node structure's
> 	->qsmaskinitnext field still shows CPU 1 as being online, this new
> 	grace period is initialized to wait for a quiescent state from the
> 	now-offline CPU 1.

If we're past cpuhp_report_idle_cpu() -> rcu_report_dead(), then
cpu_offline() is true. Is that not sufficient state to avoid this?

> 6.	Without the fail-safe force-quiescent-state checks, there would
> 	be no quiescent state from the now-offline CPU 1, which would
> 	eventually result in RCU CPU stall warnings and memory exhaustion.