Re: [PATCH v2] sched/core: Don't mix isolcpus and housekeeping CPUs

From: Peter Zijlstra
Date: Fri Oct 26 2018 - 04:41:31 EST


On Thu, Oct 25, 2018 at 11:00:58PM +0530, Srikar Dronamraju wrote:
> > But it doesn't solve the problem.
> >
> > You can create multiple partitions with cpusets but still have an
> > unbound task in the root cgroup. That would suffer the exact same
> > problems.
> >
> > Thing is, load-balancing, of any kind, should respect sched_domains, and
> > currently numa balancing barely looks at it.
>
> Agreed that we should have looked at sched_domains. However I still believe
> we can't have task->cpus_allowed with a mix of isolcpus and non-isolcpus.
> won't it lead to inconsistent behaviour?

We currently can; and it depends on what you call inconsistent.

> > The proposed patch puts the minimal constraints on the numa balancer to
> > respect sched_domains; but doesn't yet correctly deal with hotplug.
>
> I was also thinking about hotplug. Also your proposed patch and even my
> proposed patch don't seem to work well with the below scenario.
>
> # cat /sys/devices/system/cpu/possible
> 0-31
> # cat /sys/devices/system/cpu/isolated
> 1,5,9,13
> # cat hist.sh
> echo 0 > /proc/sys/kernel/numa_balancing
> cd /sys/fs/cgroup/cpuset
> mkdir -p student
> cp cpuset.mems student/
> cd student
> echo "0-31" > cpuset.cpus
> echo $$ > cgroup.procs
> echo "1-8" > cpuset.cpus
> /home/srikar/work/ebizzy-0.3/ebizzy -S 1000 &
> PID=$!
> sleep 10
> pidstat -p $! -t |tail -n +3 |head -n 10
> pidstat -p $$ -t |tail -n +3
> pkill ebizzy
> #
> # ./hist.sh
> 10:35:21 IST UID TGID TID %usr %system %guest %CPU CPU Command
> 10:35:21 IST 0 2645 - 8.70 0.01 0.00 8.71 1 ebizzy
> 10:35:21 IST 0 - 2645 0.01 0.00 0.00 0.01 1 |__ebizzy
> 10:35:21 IST 0 - 2647 0.14 0.00 0.00 0.14 1 |__ebizzy
> 10:35:21 IST 0 - 2648 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:21 IST 0 - 2649 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:21 IST 0 - 2650 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:21 IST 0 - 2651 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:21 IST 0 - 2652 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:21 IST 0 - 2653 0.13 0.00 0.00 0.13 1 |__ebizzy
> 10:35:23 IST UID TGID TID %usr %system %guest %CPU CPU Command
> 10:35:23 IST 0 2642 - 0.00 0.00 0.00 0.00 1 hist.sh
> 10:35:23 IST 0 - 2642 0.00 0.00 0.00 0.00 1 |__hist.sh
> #

That's correct and specified behaviour.

cpu1 has no sched domain (or the singleton domaon, which is the same)
and thus its tasks will not be migrated.

If there is a problem, it is with cpuset and its interaction with
isolcpus. I still have to look at the last version of cpuset-v2, but
iirc that was better in this regard.

I'm stil saying we should kill isolcpus entirely, its rubbish, has
always been.