Re: [RFC PATCH 12/12] housekeeping: Reimplement isolcpus on housekeeping

[Frederic Weisbecker]

On Mon, Aug 28, 2017 at 03:31:16PM +0200, Peter Zijlstra wrote:
> On Mon, Aug 28, 2017 at 03:23:06PM +0200, Frederic Weisbecker wrote:
> > On Mon, Aug 28, 2017 at 12:09:57PM +0200, Peter Zijlstra wrote:
> > > On Wed, Aug 23, 2017 at 03:51:11AM +0200, Frederic Weisbecker wrote:
> > > > We want to centralize the isolation features on the housekeeping
> > > > subsystem and scheduler isolation is a significant part of it.
> > > > 
> > > > While at it, this is a proposition for a reimplementation of isolcpus=
> > > > that doesn't involve  scheduler domain isolation. Therefore this
> > > > brings a behaviour change: all user tasks inherit init/1 affinity which
> > > > avoid the isolcpus= range. But if a task later overrides its affinity
> > > > which turns out to intersect an isolated CPU, load balancing may occur
> > > > on it.
> > > > 
> > > > OTOH such a reimplementation that doesn't shortcut scheduler internals
> > > > makes a better candidate for an interface extension to cpuset.
> > > 
> > > Not sure we can do this. It'll break users that rely on the no
> > > scheduling thing, that's a well documented part of isolcpus.
> > 
> > That was my worry :-s  That NULL domain was probably a design mistake and
> > I fear we now have to maintain it.
> 
> I'm fairly sure that was very intentional. If you want to isolate stuff
> you don't want load-balancing.

Yes I guess that was intentional. In fact having NULL domains is convenient
as it also isolates from many things: tasks, workqueues, timers.

Although for example I guess (IIUC) that if you create an unbound timer on a NULL
domain, it will be stuck on it for ever as we can't walk any hierarchy from the
current CPU domain. I'm not sure how much that can apply to unbound workqueues
as well. But the thing is with NULL domains: things can not migrate in and neither
can them migrate out, which is not exactly what CPU isolation wants.

> You get the same NULL domain with cpusets if you disable balancing for a set of CPUs.

Ok, I didn't know that.

> 
> Now, I completely hate the isolcpus feature and wish is a speedy death,
> but replacing it with something sensible is difficult because cgroups
> :-(

Ah, that would break cgroup somehow?