Re: [PATCH 08/11 v2] nohz: Allow rcu extended quiescent statehandling seperately from tick stop

[Paul E. McKenney]

On Thu, Oct 13, 2011 at 02:50:20PM +0200, Frederic Weisbecker wrote:
> On Thu, Oct 13, 2011 at 12:03:57AM -0700, Paul E. McKenney wrote:
> > On Wed, Oct 12, 2011 at 11:57:52PM -0700, Paul E. McKenney wrote:
> > > On Sat, Oct 08, 2011 at 04:01:00PM +0200, Frederic Weisbecker wrote:
> > > > It is assumed that rcu won't be used once we switch to tickless
> > > > mode and until we restart the tick. However this is not always
> > > > true, as in x86-64 where we dereference the idle notifiers after
> > > > the tick is stopped.
> > > > 
> > > > To prepare for fixing this, add two new APIs:
> > > > tick_nohz_idle_enter_norcu() and tick_nohz_idle_exit_norcu().
> > > > 
> > > > If no use of RCU is made in the idle loop between
> > > > tick_nohz_enter_idle() and tick_nohz_exit_idle() calls, the arch
> > > > must instead call the new *_norcu() version such that the arch doesn't
> > > > need to call rcu_idle_enter() and rcu_idle_exit().
> > > > 
> > > > Otherwise the arch must call tick_nohz_enter_idle() and
> > > > tick_nohz_exit_idle() and also call explicitly:
> > > > 
> > > > - rcu_idle_enter() after its last use of RCU before the CPU is put
> > > > to sleep.
> > > > - rcu_idle_exit() before the first use of RCU after the CPU is woken
> > > > up.
> > > 
> > > Thank you, Frederic!  I have queued this to replace the earlier
> > > version.  The set is available on branch rcu/dyntick of
> > > 
> > > 	https://github.com/paulmckrcu/linux
> > 
> > Which reminds me...  About the ultimate objective, getting tick-free
> > operation.  (Or, for the guys who want to eliminate the tick entirely,
> > shutting up the hrtimer stuff that they want to replace it with.)
> > 
> > I believe that you will then need to have two levels of not-in-dynticks
> > for processes, one for idle vs. not and another for when a process
> > switches from user-space to kernel execution.  Correct, or am I
> > confused?
> > 
> > The reason I ask is that commit e11f5981 currently only allows one
> > level of not-in-dynticks for processes.  It is easy to add another
> > level, but thought I should check beforehand.
> 
> Hmm, yeah looking at that patch, it's going to be hard to have a nesting
> that looks like:
> 
> rcu_irq_enter();
> rcu_user_enter();
> rcu_irq_exit(); <-- with effective extended quiescent state starting there

OK, so the idea here is that there has been two runnable processes on
the current CPU, but during the irq handler one of them moves or some
such?  If so, how about a rcu_user_enter_fromirq() that sets the counter
to 1 so that the rcu_irq_exit() cleans up properly?  If need be, I could
of course provide an argument to allow you to specify the count offset.

Or am I misunderstanding the problem?

> I also need to be able to call rcu_user_enter() from non-irq path.

Then rcu_user_enter_fromirq() would be for the irq path and
rcu_user_enter() from the non-irq path.

Would that work for you?

> I don't truly understand the problem of the usermode helpers that
> mess up the dynticks counts. May be we can somewhow fix it differently
> from the offending callsite?

I tried a few approaches along these lines, but there were way too
many opportunities for interruption and preemption along the way.
The problem is that unless the fixup happens under a no-preempt
region of code that includes the rcu_irq_enter() or rcu_irq_exit()
call (as the case may be), then you end up messing up the idle-depth
count of two CPUs rather than just one.  :-(

But maybe I am missing something -- suggestions more than welcome!

							Thanx, Paul
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/