Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct rcu_dynticks

[Paul E. McKenney]

On Fri, Jun 22, 2018 at 11:19:16AM -0700, Joel Fernandes wrote:
> On Fri, Jun 22, 2018 at 06:28:43AM -0700, Paul E. McKenney wrote:
> > On Thu, Jun 21, 2018 at 10:56:59PM -0700, Joel Fernandes wrote:
> > > Hi Paul,
> > > 
> > > On Wed, Jun 20, 2018 at 09:49:02AM -0700, Paul E. McKenney wrote:
> > > > On Thu, Jun 21, 2018 at 01:05:22AM +0900, Byungchul Park wrote:
> > > > > On Wed, Jun 20, 2018 at 11:58 PM, Paul E. McKenney
> > > > > <paulmck@xxxxxxxxxxxxxxxxxx> wrote:
> > > > > > On Wed, Jun 20, 2018 at 05:47:20PM +0900, Byungchul Park wrote:
> > > > > >> Hello folks,
> > > > > >>
> > > > > >> I'm careful in saying that ->dynticks_nmi_nesting can be removed but I
> > > > > >> think it's possible since the only thing we are interested in with
> > > > > >> regard to ->dynticks_nesting or ->dynticks_nmi_nesting is whether rcu is
> > > > > >> idle or not.
> > > > > >
> > > > > > Please keep in mind that NMIs cannot be masked, which means that the
> > > > > > rcu_nmi_enter() and rcu_nmi_exit() pair can be invoked at any point in
> > > > > > the process, between any consecutive pair of instructions.  The saving
> > > > 
> > > > And yes, I should have looked at this patch more closely before replying.
> > > > But please see below.
> > > > 
> > > > > I believe I understand what NMI is and why you introduced
> > > > > ->dynticks_nmi_nesting. Or am I missing something?
> > > > 
> > > > Perhaps the fact that there are architectures that can enter interrupt
> > > > handlers and never leave them when the CPU is non-idle.  One example of
> > > > this is the usermode upcalls in the comment that you removed.
> > > 
> > > I spent some time tonight and last night trying to understand this concept of
> > > never leaving an interrupt, I hope you don't mind me asking this dumb
> > > question... perhaps I will learn something : Could you let me know how is it
> > > possible that an interrupt never exits?
> > > 
> > > Typically an interrupt never exiting sounds like a hard-lockup. This is how
> > > hardlock detector works: Since regular interrupts in linux can't nest, the
> > > hardlockup detector checks if hrtimer interrupts are being handled and if
> > > not, then it throws a splat, panics the kernel etc. So I am a bit troubled by
> > > this interrupt never exiting concept..
> > > 
> > > Further since an interrupt is an atomic context, it cannot sleep or schedule
> > > into usermode so how are these upcalls handled from the interrupt?
> > 
> > It has been some years since I traced the code flow, but what happened
> 
> No problem, thanks for the discussion. :)
> 
> > back then is that it switches itself from an interrupt handler to not
> > without actually returning from the interrupt.  This can only happen when
> > interrupting a non-idle process, thankfully, and RCU's dyntick-idle code
> > relies on this restriction.  If I remember correctly, the code ends up
> > executing in the context of the interrupted process, but it has been some
> > years, so please apply appropriate skepticism.
> 
> Sure. So in a later thread you mentioned "usermode helpers". I took a closer
> look at that subsystem, and it seems you can execute usermode helpers from
> atomic sections with help of UMH_NO_WAIT flag.
> 
> Then I checked where this flag is used and it turns out its from the
> mce_work_trigger function in x86/kernel/cpu/mcheck/dev-mcelog.c which can be
> called infact from an interrupt context (mce_notify_irq).
> 
> Is this the usecase you remember causing this weird transitions to userspace?

It was on Power, so it was not this one.

> > Please take a look at the "Interrupts and NMIs" section of the file
> > Documentation/RCU/Design/Requirements/Requirements.html for a bit
> > more information.
> 
> Sure, thanks for the pointer.
> 
> > > Lastly, can you point me to an example how the rcu_nmi_enter/exit() pair can go
> > > out sync? That is they aren't paired and nested properly? In my mind they
> > > always should be but I may be missing the usecase. I'm happy to try and
> > > reproduce and trace this if you can let me know how to so that I can study
> > > it better. 
> > 
> > I have never seen NMIs be unpaired or improperly nested.  However,
> > given that rcu_irq_enter() invokes rcu_nmi_enter() and rcu_irq_exit()
> > invokes rcu_nmi_exit(), it is definitely the case that rcu_nmi_enter()
> > and rcu_nmi_exit() need to deal with unpaired and improperly nested
> > invocations.
> 
> Just wondering how would the fact that rcu_irq_enter calls into rcu_nmi_enter
> cause an improper nesting?
> 
> Just to define what "improper nesting" means, if we can go through an
> example. Do you mean a scenario like?
> 
> rcu_nmi_enter (called because of NMI)
> rcu_nmi_enter (called because of IRQ)
> rcu_nmi_exit (called because of NMI)
> rcu_nmi_exit (called because of IRQ)
> 
> Such scenario seems impossible to me because the IRQ can't be entered after
> the NMI entered.

That is OK.  NMIs really can nest.  Or more accurately, things that
appear to be NMIs from RCU's viewpoint can nest within other things
that appear to be NMIs from RCU's viewpoint.

> On the other hand, if you meant that when IRQ is being handled, an NMI fires
> just before the rcu_irq_enter calls rcu_nmi_enter, then the worst that could
> happen seems to be that the rcu_nmi_enter/exit pairs will not be nested
> within the outer rcu_nmi_enter/exit pair even though the NMI interrupted the
> IRQ. So it'll be something like:
> 
> rcu_nmi_enter (called because of NMI)
> rcu_nmi_exit (called because of NMI)
> rcu_nmi_enter (called because of IRQ)
> rcu_nmi_exit (called because of IRQ)
> 
> Even though what really happened in the real world is:
> 
> IRQ entered
> NMI entered
> NMI exited
> IRQ exited
> 
> This also seems reasonable to me, but is this what you meant by improper
> nesting of the rcu_nmi_enter/exit? If yes, what makes it unreasonable?

Something like this:

	IRQ entered

And never exited.  Ever.  I actually saw this in 2011.

Or something like this:

	IRQ exited

Without a corresponding IRQ enter.

The current code handles both of these situations, at least assuming
that the interrupt entry/exit happens during a non-idle period.

> > So why this function-call structure?  Well, you see, NMI handlers can
> > take what appear to RCU to be normal interrupts...
> > 
> > (And I just added that fun fact to Requirements.html.)
> 
> Yes, I'll definitely go through all the interrupt requirements in the doc and
> thanks for referring me to it.

My concern may well be obsolete.  It would be good if it was!  ;-)

								Thanx, Paul