Re: NOHZ tick-stop error: Non-RCU local softirq work is pending

[Frederic Weisbecker]

On Thu, Dec 10, 2020 at 04:46:38PM -0800, Paul E. McKenney wrote:
> > diff --git a/kernel/softirq.c b/kernel/softirq.c
> > index 09229ad82209..7d558cb7a037 100644
> > --- a/kernel/softirq.c
> > +++ b/kernel/softirq.c
> > @@ -650,7 +650,9 @@ static void run_ksoftirqd(unsigned int cpu)
> >  		 * We can safely run softirq on inline stack, as we are not deep
> >  		 * in the task stack here.
> >  		 */
> > -		__do_softirq();
> > +		do {
> > +			__do_softirq();
> > +		} while (kthread_should_park() && local_softirq_pending());
> >  		local_irq_enable();
> >  		cond_resched();
> >  		return;
> 
> Huh.  I guess that self-propagating timers, RCU callbacks, and the
> like are non-problems because they cannot retrigger while interrupts
> are disabled?  But can these things reappear just after the
> local_irq_enable()?

Exactly, unless those things have been disabled in CPU hotplug callbacks
before smpboot_park_threads() (and we are not too deep in CPU hotplug at that
stage, cpu_online() is still true), they should be able to raise again
the softirqs. Yeah I'm pretty sure the above won't be enough.

> In the case of RCU, softirq would need to run on this CPU, which it won't,
> so we are good in that case.  (Any stranded callbacks will be requeued
> onto some other CPU later in the CPU-hotplug offline processing.)

Ah that's good, so at least we shouldn't worry about pending RCU softirqs
after ksoftirqd has parked. But I also see at least SCHED_SOFTIRQ and
TIMER_SOFTIRQ in your list.

Perhaps we should have some flush_softirq() somewhere late in cpu hotplug,
assuming this is actually necessary and none of these will requeue themselves...

Thanks.

> 
> 							Thanx, Paul
> 
> > Thanks!