Re: [patch 10/29] lockup_detector/perf: Prevent cpu hotplug deadlock

[Don Zickus]

On Fri, Sep 01, 2017 at 09:29:07PM +0200, Thomas Gleixner wrote:
> On Fri, 1 Sep 2017, Don Zickus wrote:
> > On Thu, Aug 31, 2017 at 09:16:08AM +0200, Thomas Gleixner wrote:
> > > The following deadlock is possible in the watchdog hotplug code:
> > > 
> > >   cpus_write_lock()
> > >     ...
> > >       takedown_cpu()
> > >         smpboot_park_threads()
> > >           smpboot_park_thread()
> > >             kthread_park()
> > >               ->park() := watchdog_disable()
> > >                 watchdog_nmi_disable()
> > >                   perf_event_release_kernel();
> > >                     put_event()
> > >                       _free_event()
> > >                         ->destroy() := hw_perf_event_destroy()
> > >                           x86_release_hardware()
> > >                             release_ds_buffers()
> > >                               get_online_cpus()
> > > 
> > > when a per cpu watchdog perf event is destroyed which drops the last
> > > reference to the PMU hardware. The cleanup code there invokes
> > > get_online_cpus() which instantly deadlocks because the hotplug percpu
> > > rwsem is write locked.
> > 
> > The main reason perf_event_release_kernel is in this path is because the
> > oprofile folks complained they couldn't use the perf counters when the
> > nmi_watchdog was disabled on the command line.
> 
> If the nmi watchdog is disabled on the command line then there are no
> counters claimed at all.

Ah, I see it now.  When you park all the threads, you clear the cpumask
which then calls the lockup detector_cleanup to release all perf_counters
that are no longer used by the cpumask.

Further reading this code, I see how the code covers various race conditions
we tried solving in the past.

Thanks!

Cheers,
Don