Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct rcu_dynticks

[Joel Fernandes]

On Fri, Jun 22, 2018 at 06:28:43AM -0700, Paul E. McKenney wrote:
> On Thu, Jun 21, 2018 at 10:56:59PM -0700, Joel Fernandes wrote:
> > Hi Paul,
> > 
> > On Wed, Jun 20, 2018 at 09:49:02AM -0700, Paul E. McKenney wrote:
> > > On Thu, Jun 21, 2018 at 01:05:22AM +0900, Byungchul Park wrote:
> > > > On Wed, Jun 20, 2018 at 11:58 PM, Paul E. McKenney
> > > > <paulmck@xxxxxxxxxxxxxxxxxx> wrote:
> > > > > On Wed, Jun 20, 2018 at 05:47:20PM +0900, Byungchul Park wrote:
> > > > >> Hello folks,
> > > > >>
> > > > >> I'm careful in saying that ->dynticks_nmi_nesting can be removed but I
> > > > >> think it's possible since the only thing we are interested in with
> > > > >> regard to ->dynticks_nesting or ->dynticks_nmi_nesting is whether rcu is
> > > > >> idle or not.
> > > > >
> > > > > Please keep in mind that NMIs cannot be masked, which means that the
> > > > > rcu_nmi_enter() and rcu_nmi_exit() pair can be invoked at any point in
> > > > > the process, between any consecutive pair of instructions.  The saving
> > > 
> > > And yes, I should have looked at this patch more closely before replying.
> > > But please see below.
> > > 
> > > > I believe I understand what NMI is and why you introduced
> > > > ->dynticks_nmi_nesting. Or am I missing something?
> > > 
> > > Perhaps the fact that there are architectures that can enter interrupt
> > > handlers and never leave them when the CPU is non-idle.  One example of
> > > this is the usermode upcalls in the comment that you removed.
> > 
> > I spent some time tonight and last night trying to understand this concept of
> > never leaving an interrupt, I hope you don't mind me asking this dumb
> > question... perhaps I will learn something : Could you let me know how is it
> > possible that an interrupt never exits?
> > 
> > Typically an interrupt never exiting sounds like a hard-lockup. This is how
> > hardlock detector works: Since regular interrupts in linux can't nest, the
> > hardlockup detector checks if hrtimer interrupts are being handled and if
> > not, then it throws a splat, panics the kernel etc. So I am a bit troubled by
> > this interrupt never exiting concept..
> > 
> > Further since an interrupt is an atomic context, it cannot sleep or schedule
> > into usermode so how are these upcalls handled from the interrupt?
> 
> It has been some years since I traced the code flow, but what happened

No problem, thanks for the discussion. :)

> back then is that it switches itself from an interrupt handler to not
> without actually returning from the interrupt.  This can only happen when
> interrupting a non-idle process, thankfully, and RCU's dyntick-idle code
> relies on this restriction.  If I remember correctly, the code ends up
> executing in the context of the interrupted process, but it has been some
> years, so please apply appropriate skepticism.

Sure. So in a later thread you mentioned "usermode helpers". I took a closer
look at that subsystem, and it seems you can execute usermode helpers from
atomic sections with help of UMH_NO_WAIT flag.

Then I checked where this flag is used and it turns out its from the
mce_work_trigger function in x86/kernel/cpu/mcheck/dev-mcelog.c which can be
called infact from an interrupt context (mce_notify_irq).

Is this the usecase you remember causing this weird transitions to userspace?

> Please take a look at the "Interrupts and NMIs" section of the file
> Documentation/RCU/Design/Requirements/Requirements.html for a bit
> more information.

Sure, thanks for the pointer.

> > Lastly, can you point me to an example how the rcu_nmi_enter/exit() pair can go
> > out sync? That is they aren't paired and nested properly? In my mind they
> > always should be but I may be missing the usecase. I'm happy to try and
> > reproduce and trace this if you can let me know how to so that I can study
> > it better. 
> 
> I have never seen NMIs be unpaired or improperly nested.  However,
> given that rcu_irq_enter() invokes rcu_nmi_enter() and rcu_irq_exit()
> invokes rcu_nmi_exit(), it is definitely the case that rcu_nmi_enter()
> and rcu_nmi_exit() need to deal with unpaired and improperly nested
> invocations.

Just wondering how would the fact that rcu_irq_enter calls into rcu_nmi_enter
cause an improper nesting?

Just to define what "improper nesting" means, if we can go through an
example. Do you mean a scenario like?

rcu_nmi_enter (called because of NMI)
rcu_nmi_enter (called because of IRQ)
rcu_nmi_exit (called because of NMI)
rcu_nmi_exit (called because of IRQ)

Such scenario seems impossible to me because the IRQ can't be entered after
the NMI entered.

On the other hand, if you meant that when IRQ is being handled, an NMI fires
just before the rcu_irq_enter calls rcu_nmi_enter, then the worst that could
happen seems to be that the rcu_nmi_enter/exit pairs will not be nested
within the outer rcu_nmi_enter/exit pair even though the NMI interrupted the
IRQ. So it'll be something like:

rcu_nmi_enter (called because of NMI)
rcu_nmi_exit (called because of NMI)
rcu_nmi_enter (called because of IRQ)
rcu_nmi_exit (called because of IRQ)

Even though what really happened in the real world is:

IRQ entered
NMI entered
NMI exited
IRQ exited

This also seems reasonable to me, but is this what you meant by improper
nesting of the rcu_nmi_enter/exit? If yes, what makes it unreasonable?

> So why this function-call structure?  Well, you see, NMI handlers can
> take what appear to RCU to be normal interrupts...
> 
> (And I just added that fun fact to Requirements.html.)

Yes, I'll definitely go through all the interrupt requirements in the doc and
thanks for referring me to it.

thanks,

 - Joel