Re: [PATCH RFC tip/core/rcu 14/14] rcu/nohz: Make multi_cpu_stop() enable tick on all online CPUs

[Paul E. McKenney]

On Mon, Aug 12, 2019 at 11:02:33PM +0200, Frederic Weisbecker wrote:
> On Fri, Aug 02, 2019 at 08:15:01AM -0700, Paul E. McKenney wrote:
> > The multi_cpu_stop() function relies on the scheduler to gain control from
> > whatever is running on the various online CPUs, including any nohz_full
> > CPUs running long loops in kernel-mode code.  Lack of the scheduler-clock
> > interrupt on such CPUs can delay multi_cpu_stop() for several minutes
> > and can also result in RCU CPU stall warnings.  This commit therefore
> > causes multi_cpu_stop() to enable the scheduler-clock interrupt on all
> > online CPUs.
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxx>
> > ---
> >  kernel/stop_machine.c | 9 ++++++++-
> >  1 file changed, 8 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> > index b4f83f7bdf86..a2659f61ed92 100644
> > --- a/kernel/stop_machine.c
> > +++ b/kernel/stop_machine.c
> > @@ -20,6 +20,7 @@
> >  #include <linux/smpboot.h>
> >  #include <linux/atomic.h>
> >  #include <linux/nmi.h>
> > +#include <linux/tick.h>
> >  #include <linux/sched/wake_q.h>
> >  
> >  /*
> > @@ -187,15 +188,19 @@ static int multi_cpu_stop(void *data)
> >  {
> >  	struct multi_stop_data *msdata = data;
> >  	enum multi_stop_state curstate = MULTI_STOP_NONE;
> > -	int cpu = smp_processor_id(), err = 0;
> > +	int cpu, err = 0;
> >  	const struct cpumask *cpumask;
> >  	unsigned long flags;
> >  	bool is_active;
> >  
> > +	for_each_online_cpu(cpu)
> > +		tick_nohz_dep_set_cpu(cpu, TICK_DEP_MASK_RCU);
> 
> Looks like it's not the right fix but, should you ever need to set an
> all-CPUs (system wide) tick dependency in the future, you can use tick_set_dep().

Indeed, I have dropped this patch, but I now do something similar in
RCU's CPU-hotplug notifiers.  Which does have an effect, especially on
the system that isn't subject to the insane-latency cpu_relax().

Plus I am having to put a similar workaround into RCU's quiescent-state
forcing logic.

But how should this really be done?

Isn't there some sort of monitoring of nohz_full CPUs for accounting
purposes?  If so, would it make sense for this monitoring to check for
long-duration kernel execution and enable the tick in this case?  The
RCU dyntick machinery can be used to remotely detect the long-duration
kernel execution using something like the following:

	int nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);

	...

	if (rcu_dynticks_in_eqs_cpu(cpu, nohz_in_kernel_snap)
		nohz_in_kernel_snap = rcu_dynticks_snap_cpu(cpu);
	else
		/* Turn on the tick! */

I would supply rcu_dynticks_snap_cpu() and rcu_dynticks_in_eqs_cpu(),
which would be simple wrappers around RCU's private rcu_dynticks_snap()
and rcu_dynticks_in_eqs() functions.

Would this make sense as a general solution, or am I missing a corner
case or three?

							Thanx, Paul