Re: [RFC 2/2] rcu: Remove ->dynticks_nmi_nesting from struct rcu_dynticks

[Paul E. McKenney]

On Sat, Jun 23, 2018 at 10:53:56AM -0700, Paul E. McKenney wrote:
> On Fri, Jun 22, 2018 at 03:03:35PM -0700, Andy Lutomirski wrote:
> > On Fri, Jun 22, 2018 at 2:14 PM Paul E. McKenney
> > <paulmck@xxxxxxxxxxxxxxxxxx> wrote:
> > >
> > > On Fri, Jun 22, 2018 at 05:00:42PM -0400, Steven Rostedt wrote:
> > > > On Fri, 22 Jun 2018 13:58:13 -0700
> > > > "Paul E. McKenney" <paulmck@xxxxxxxxxxxxxxxxxx> wrote:
> > > >
> > > > > Something like this:
> > > > >
> > > > >     IRQ entered
> > > > >
> > > > > And never exited.  Ever.  I actually saw this in 2011.
> > > >
> > > > I still believe this was actually a bug. And perhaps you made the RCU
> > > > code robust enough to handle this bug ;-)
> > >
> > > Welcome to my world!
> > >
> > > But I recall it being used in several places, so if it was a bug, it
> > > was an intentional bug.  Probably the worst kind.
> > >
> > > Sort of like nested NMIs and interrupts within NMI handlers.  ;-)
> > >
> > > > > Or something like this:
> > > > >
> > > > >     IRQ exited
> > > > >
> > > > > Without a corresponding IRQ enter.
> > > > >
> > > > > The current code handles both of these situations, at least assuming
> > > > > that the interrupt entry/exit happens during a non-idle period.
> > > > >
> > > > > > > So why this function-call structure?  Well, you see, NMI handlers can
> > > > > > > take what appear to RCU to be normal interrupts...
> > > > > > >
> > > > > > > (And I just added that fun fact to Requirements.html.)
> > > > > >
> > > > > > Yes, I'll definitely go through all the interrupt requirements in the doc and
> > > > > > thanks for referring me to it.
> > > > >
> > > > > My concern may well be obsolete.  It would be good if it was!  ;-)
> > > >
> > > > I'd love to mandate that irq_enter() must be paired with irq_exit(). I
> > > > don't really see any rationale for it to be otherwise. If there is a
> > > > case, perhaps it needs to be fixed.
> > >
> > > Given that the usermode helpers now look to be common code using
> > > workqueues, kthreads, and calls to do_execve(), it might well be that
> > > the days of half-interrupts are behind us.
> > >
> > > But how to actually validate this?  My offer of adding a WARN_ON_ONCE()
> > > and waiting a few years still stands, but perhaps you have a better
> > > approach.
> > 
> > I think you should add a WARN_ON_ONCE().  Let's get the bugs fixed.
> 
> Or the obscure features identified, as the case may be.  ;-)
> 
> Either way, will do!

And here is a prototype patch.

							Thanx, Paul

------------------------------------------------------------------------

commit ef544593a7bcad74628fa0537badc49dce1f2d95
Author: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>
Date:   Thu Jun 28 12:45:23 2018 -0700

    rcu: Add warning to detect half-interrupts
    
    RCU's dyntick-idle code is written to tolerate half-interrupts, that it,
    either an interrupt that invokes rcu_irq_enter() but never invokes the
    corresponding rcu_irq_exit() on the one hand, or an interrupt that never
    invokes rcu_irq_enter() but does invoke the "corresponding" rcu_irq_exit()
    on the other.  These things really did happen at one time, as evidenced
    by this ca-2011 LKML post:
    
    http://lkml.kernel.org/r/20111014170019.GE2428@xxxxxxxxxxxxxxxxxx
    
    The reason why RCU tolerates half-interrupts is that usermode helpers
    used exceptions to invoke a system call from within the kernel such that
    the system call did a normal return (not a return from exception) to
    the calling context.  This caused rcu_irq_enter() to be invoked without
    a matching rcu_irq_exit().  However, usermode helpers have since been
    rewritten to make much more housebroken use of workqueues, kernel threads,
    and do_execve(), and therefore should no longer produce half-interrupts.
    No one knows of any other source of half-interrupts, but then again,
    no one seems insane enough to go audit the entire kernel to verify that
    half-interrupts really are a relic of the past.
    
    This commit therefore adds a pair of WARN_ON_ONCE() calls that will
    trigger in the presence of half interrupts, which the code will continue
    to handle correctly.  If neither of these WARN_ON_ONCE() trigger by
    mid-2021, then perhaps RCU can stop handling half-interrupts, which
    would be a considerable simplification.
    
    Reported-by: Steven Rostedt <rostedt@xxxxxxxxxxx>
    Reported-by: Joel Fernandes <joel@xxxxxxxxxxxxxxxxx>
    Reported-by: Andy Lutomirski <luto@xxxxxxxxxx>
    Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>

diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
index 6c5a7f0daadc..37ae0d77854d 100644
--- a/kernel/rcu/tree.c
+++ b/kernel/rcu/tree.c
@@ -714,6 +714,7 @@ static void rcu_eqs_enter(bool user)
 	struct rcu_dynticks *rdtp;
 
 	rdtp = this_cpu_ptr(&rcu_dynticks);
+	WARN_ON_ONCE(rdtp->dynticks_nmi_nesting != DYNTICK_IRQ_NONIDLE);
 	WRITE_ONCE(rdtp->dynticks_nmi_nesting, 0);
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) &&
 		     rdtp->dynticks_nesting == 0);
@@ -895,6 +896,7 @@ static void rcu_eqs_exit(bool user)
 	trace_rcu_dyntick(TPS("End"), rdtp->dynticks_nesting, 1, rdtp->dynticks);
 	WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current));
 	WRITE_ONCE(rdtp->dynticks_nesting, 1);
+	WARN_ON_ONCE(rdtp->dynticks_nmi_nesting);
 	WRITE_ONCE(rdtp->dynticks_nmi_nesting, DYNTICK_IRQ_NONIDLE);
 }