Re: [RFC PATCH 0/6] CPU hotplug: Reverse invocation of notifiersduring CPU hotplug

[Alan Stern]

On Wed, 25 Jul 2012, Srivatsa S. Bhat wrote:

> On 07/25/2012 08:27 PM, Alan Stern wrote:
> > On Wed, 25 Jul 2012, Srivatsa S. Bhat wrote:
> > 
> >> Hi,
> >>
> >> This patchset implements the approach of invoking the CPU hotplug callbacks
> >> (notifiers) in one order during CPU online and in the reverse order during CPU
> >> offline. The rationale behind this is that services for a CPU are started in a
> >> particular order (perhaps, with implicit dependencies between them) while
> >> bringing up the CPU, and hence, it makes sense to tear down the services in
> >> the opposite order, thereby honoring most of the dependencies automatically
> >> (and also correctly). This is explained in more detail in Patch 6.
> > 
> > This strongly suggests that a notifier chain may be the wrong mechanism
> > to use here.  Notifiers provide only limited guarantees about ordering,
> > and it's hard to say much about the services a particular chain will
> > provide since callbacks can be added from anywhere.
> > 
> 
> True, the ability to register any random callback from anywhere is still a
> problem that we are fighting... The zillions of callbacks that we have today
> makes the hotplug process quite entangled.. we can't even roll-back from a
> failure easily!
> 
> > Instead of adding all this complication to the notifier mechanism, how 
> > about using something else for CPU hotplug?
> > 
> 
> The problem is that today, many different subsystems need to know about CPUs coming
> up or going down.. And CPU hotplug is not atomic, it happens in stages, and the
> coordination between those subsystems is what actually drives CPU hotplug, in a way.

All this reinforces the idea that notifiers are the wrong mechanism for 
CPU hotplug.

> At present, I think that the best we can do is to redesign the hotplug code such that
> the number of callbacks that are needed can be reduced to a minimum amount and then
> have good control over what those callbacks do. For example, Thomas Gleixner posted
> the park/unpark patchset[1], which not only speeds-up CPU hotplug by avoiding destruction
> and creation of per-cpu kthreads on every hotplug operation, but also gets rid of quite
> a few notifiers by providing a framework to manage those per-cpu kthreads...

I think the best you can do is stop using notifiers and use something 
else instead.  For example, a simple set of function calls (assuming 
you know beforehand what callbacks need to be invoked).

> One of the other ideas to improve the hotplug notifier stuff that came up during some
> of the discussions was to implement explicit dependency tracking between the notifiers
> and perhaps get rid of the priority numbers that are currently being used to provide
> some sort of ordering between the callbacks. Links to some of the related discussions
> are provided below.

This seems like misplaced over-engineering.

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/