Re: Possible use of RCU while in extended QS: idle vs RCU read-side in interrupt vs rcu_eqs_exit

[Paul E. McKenney]

On Thu, Jan 10, 2019 at 01:30:05AM -0500, Mathieu Desnoyers wrote:
> ----- On Jan 9, 2019, at 8:13 PM, paulmck paulmck@xxxxxxxxxxxxx wrote:
> 
> > On Wed, Jan 09, 2019 at 08:38:51PM -0500, Mathieu Desnoyers wrote:
> >> Hi Paul,
> >> 
> >> I've had a user report that trace_sched_waking() appears to be
> >> invoked while !rcu_is_watching() in some situation, so I started
> >> digging into the scheduler idle code.
> >> 
> >> It appears that interrupts are re-enabled before rcu_eqs_exit() is
> >> invoked when exiting idle code from the scheduler.
> >> 
> >> I wonder what happens if an interrupt handler (including scheduler code)
> >> happens to issue a RCU read-side critical section before rcu_eqs_exit()
> >> is called ? Is there some code on interrupt entry that ensures rcu eqs
> >> state is exited in such scenario ?
> > 
> > Interrupt handlers are supposed to invoke irq_enter(), which will in
> > turn invoke rcu_irq_enter(), which should take care of things.
> > 
> > However, there are cases where a given architecture knows that a given
> > interrupt handler does not contain RCU readers, and in this case, the
> > architecture might omit the rcu_irq_enter() or maybe even the whole
> > irq_enter().  And then it is all fun and games until someone adds an
> > RCU read-side critical section.  ;-)
> 
> Even if an irq handler does not contain any RCU read-side critical
> section, won't it end by possibly invoking the scheduler before
> returning ? Considering that the scheduler has tracepoints which
> use RCU, this might be related to the issue that has been brought
> to my attention.

Most interrupt handlers just return, but yes, scheduler state is often
checked during return from interrupt.  But in that case, the interrupt
handler needs to have invoked irq_enter().

> Do you have examples of such interrupt handlers which do not invoke
> rcu_irq_enter() ?

Mostly examples of lightweight interrupts handlers that used to not invoke
irq_enter() and thus not rcu_irq_enter(), but which later started using
RCU readers.  Which means that they are no longer examples that do not
invoke rcu_irq_enter().  ;-)

Some of them just invoked rcu_irq_enter(), others had to do the full
irq_enter() call (which in turn invokes rcu_irq_enter()).

These interrupt handlers were very light-weight.  Page-table walkers,
hardware events, and the like.  Take an interrupt, look at a hardware
register, update a data structure, maybe write to a hardware register,
return from interrupt.

If there is only one such tracepoint, one approach is to use _rcuidle,
that is, instead of trace_blarvitz(), trace_blarvitz_rcuidle().  This can
add overhead, so this might not be appropriate for any of the scheduler's
fastpaths.  Which brings me back to the interrupt handler invoking
either irq_enter() or rcu_irq_enter().  Or moving the tracepoints to
a nearby region of code that RCU is already watching.

So, is it reasonably to add the rcu_irq_enter()?  If you do change this,
please test with CONFIG_RCU_EQS_DEBUG=y.

							Thanx, Paul