Re: Query regarding synchronize_sched_expedited and resched_cpu

[Paul E. McKenney]

On Tue, Sep 19, 2017 at 08:11:53AM +0200, Mike Galbraith wrote:
> On Tue, 2017-09-19 at 13:37 +0800, Boqun Feng wrote:
> > On Mon, Sep 18, 2017 at 09:04:56PM -0700, Paul E. McKenney wrote:
> > > On Tue, Sep 19, 2017 at 11:48:22AM +0900, Byungchul Park wrote:
> > > > On Mon, Sep 18, 2017 at 07:33:29PM -0700, Paul E. McKenney wrote:
> > > > > > > Hello Paul and Steven,
> > > > > > > 
> > 
> > So I think this is another false positive, and the reason is we use
> > st->done for multiple purposes.
> > 
> > > > > > > This is saying:
> > > > > > > 
> > > > > > > Thread A
> > > > > > > --------
> > > > > > > takedown_cpu()
> > > > > > >    irq_lock_sparse()
> > > > > > >    wait_for_completion(&st->done) // Wait for completion of B
> > 
> > Thread A wait for the idle task on the outgoing to set the st->state to 
> > CPUHP_AP_IDLE_DEAD(i.e. the corresponding complete() is the one in
> > cpuhp_complete_idle_dead()), and it happens when we try to _offline_ a
> > cpu.
> > 
> > > > > > >    irq_unlock_sparse()
> > > > > > > 
> > > > > > > Thread B
> > > > > > > --------
> > > > > > > cpuhp_invoke_callback()
> > > > > > >    irq_lock_sparse() // Wait for A to irq_unlock_sparse()
> > 
> > irq_affinity_online_cpu() is called here, so it happens when we try to
> > _online_ a cpu.
> > 
> > > > > > >    (on the way going to complete(&st->done))
> > 
> > and we are going to complete(&st->done) in a hotplug thread context to
> > indicate the hotplug thread has finished its job(i.e. this complete() is
> > the one in cpuhp_thread_fun()).
> > 
> > 
> > So even though the &st->done are the same instance, the deadlock could
> > not happen, I think, as we could not up/down a same cpu at the same
> > time?
> > 
> > If I'm not missing something subtle. To fix this we can either
> > 
> > 1)	have dedicated completion instances for different wait purposes
> > 	in cpuhp_cpu_state.
> > 
> > or
> > 
> > 2)	extend crossrelease to have the "subclass" concept, so that
> > 	callsite of complete() and wait_for_completion() for the same
> > 	completion instance but with different purposes could be
> > 	differed by lockdep.
> > 
> > Thoughts?
> 
> https://lkml.org/lkml/2017/9/5/184
> 
> Peter's patches worked for me, but per tglx, additional (non-
> grasshopper level) hotplug-fu is required.

Thank you, I will give these a go!

							Thanx, Paul