Re: [PATCH] sched: Fix numabalancing to work with isolated cpus

[Peter Zijlstra]

On Thu, Apr 06, 2017 at 09:34:36AM +0200, Michal Hocko wrote:
> On Thu 06-04-17 12:49:50, Srikar Dronamraju wrote:

> > Similar example that I gave in my reply to Mel.
> > 
> > Lets consider 2 node, 24 core with 12 cores in each node.
> > Cores 0-11 in Node 1 and cores 12-23 in the other node.
> > Lets also disable smt/hyperthreading, enable isolcpus from core
> > 6-11,12-17.  Lets run 48 thread ebizzy workload and give it a cpu list
> > of say 11,12-17 using taskset.
> > 
> > Now all the 48 ebizzy threads will only run on core 11. It will never
> > spread to other cores even in the same node(or in the same node/but
> > isolated cpus) or to the different nodes. i.e even if numabalancing is
> > running or not, even if my fix is around or not, all threads will be
> > confined to core 11, even though the cpus_allowed is 11,12-17.

Argh, why such a convoluted example :-(

> Isn't that a bug in isolcpus implementation? It is certainly an
> unexpected behavior I would say. Is this documented anywhere?

Without engaging the brain too much to decipher the example; it does
look right. isolcpus will have no balancing.

> Isn't sched_setaffinity the only way how to actually make it possible to
> run on isolcpus?

Think so.. Personally I hate isolcpus and never use it.

> I would really like to see it confirmed by the scheduler maintainers and
> documented properly as well. What you are claiming here is rather
> surprising to my understanding of what isolcpus acutally is.

isolcpus gets you a set of fully partitioned CPUs. What's surprising
about that?