Re: [PATCH -rcu dev 3/3] RFC: rcu/tree: Read dynticks_nmi_nesting in advance

[Paul E. McKenney]

On Fri, Aug 16, 2019 at 12:24:04PM -0400, Joel Fernandes wrote:
> On Thu, Aug 15, 2019 at 10:53:11PM -0400, Joel Fernandes (Google) wrote:
> > I really cannot explain this patch, but without it, the "else if" block
> > just doesn't execute thus causing the tick's dep mask to not be set and
> > causes the tick to be turned off.
> > 
> > I tried various _ONCE() macros but the only thing that works is this
> > patch.
> > 
> > Signed-off-by: Joel Fernandes (Google) <joel@xxxxxxxxxxxxxxxxx>
> > ---
> >  kernel/rcu/tree.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > index 856d3c9f1955..ac6bcf7614d7 100644
> > --- a/kernel/rcu/tree.c
> > +++ b/kernel/rcu/tree.c
> > @@ -802,6 +802,7 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
> >  {
> >  	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
> >  	long incby = 2;
> > +	int dnn = rdp->dynticks_nmi_nesting;
> 
> I believe the accidental sign extension / conversion from long to int was
> giving me an illusion since things started working well. Changing the 'int
> dnn' to 'long dnn' gives similar behavior as without this patch! At least I
> know now. Please feel free to ignore this particular RFC patch while I debug
> this more (over the weekend or early next week). The first 2 patches are
> good, just ignore this one.

Ah, good point on the type!  So you were ending up with zero due to the
low-order 32 bits of DYNTICK_IRQ_NONIDLE being zero, correct?  If so,
the "!rdp->dynticks_nmi_nesting" instead needs to be something like
"rdp->dynticks_nmi_nesting == DYNTICK_IRQ_NONIDLE", which sounds like
it is actually worse then the earlier comparison against the constant 2.

Sounds like I should revert the -rcu commit 805a16eaefc3 ("rcu: Force
nohz_full tick on upon irq enter instead of exit").

Or would that once again cause RCU to fail to enable the tick?

								Thanx, Paul

> thanks,
> 
>  - Joel
> 
> 
> >  
> >  	/* Complain about underflow. */
> >  	WARN_ON_ONCE(rdp->dynticks_nmi_nesting < 0);
> > @@ -826,7 +827,7 @@ static __always_inline void rcu_nmi_enter_common(bool irq)
> >  
> >  		incby = 1;
> >  	} else if (tick_nohz_full_cpu(rdp->cpu) &&
> > -		   !rdp->dynticks_nmi_nesting &&
> > +		   !dnn &&
> >  		   rdp->rcu_urgent_qs && !rdp->rcu_forced_tick) {
> >  		rdp->rcu_forced_tick = true;
> >  		tick_dep_set_cpu(rdp->cpu, TICK_DEP_BIT_RCU);
> > -- 
> > 2.23.0.rc1.153.gdeed80330f-goog
> > 
>