[PATCH v2 2/3] cgroup/cpuset: Keep the top sched domain non-empty on CPU hotplug

[Guopeng Zhang]

From: Guopeng Zhang <zhangguopeng@xxxxxxxxxx>

Configuration updates must leave an active boot domain housekeeping CPU
in the top cpuset, but CPU hotplug can still empty its scheduling domain.

For a local partition with child partitions, cpuset_hotplug_update_tasks()
passes the partition's own effective CPUs to tasks_nocpu_error(). The
embedded top_cpuset_sd_nocpu() check credits the old exclusive CPUs not
in that mask as CPUs being returned to the top cpuset. Those CPUs may
still belong to child partitions, so no such return takes place.
For example, with isolcpus=domain,10-31 on a 32-CPU system:

  echo 0,2-9 > A/cpuset.cpus
  echo root > A/cpuset.cpus.partition
  echo 5-9 > A/B/cpuset.cpus
  echo root > A/B/cpuset.cpus.partition
  echo 0 > /sys/devices/system/cpu/cpu1/online
  Oops: general protection fault
  RIP: 0010:build_perf_domains+...

Remote partition hotplug checks also miss the empty top scheduling
domain because they only examine the remote partition's own CPUs.

Check the local partition's full exclusive mask without crediting any
returned CPUs. For remote partitions, check the top scheduling domain
with all partition grants still excluded. Invalidate a partition through
its existing path to return its CPUs. Once an active boot domain
housekeeping CPU is returned, subsequent partitions no longer fail this
check.

Add hotplug test cases to test_cpuset_prs.sh to verify that the top
cpuset sched domain stays non-empty when the last housekeeping CPU
goes away.

Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
Signed-off-by: Guopeng Zhang <zhangguopeng@xxxxxxxxxx>
---
 kernel/cgroup/cpuset.c                            | 4 ++++
 tools/testing/selftests/cgroup/test_cpuset_prs.sh | 8 ++++++++
 2 files changed, 12 insertions(+)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index c405719ccd12..fc5454c2740f 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4053,6 +4053,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 		goto update_tasks;
 
 	if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) ||
+				       top_cpuset_sd_nocpu(subpartitions_cpus, NULL) ||
 				       (cpumask_empty(&new_cpus) &&
 					partition_is_populated(cs, NULL)))) {
 		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
@@ -4067,9 +4068,12 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
 	 * 2) parent is invalid or doesn't grant any cpus to child
 	 *    partitions.
 	 * 3) subpartitions_cpus is empty.
+	 * 4) top_cpuset has no active boot domain housekeeping CPU left.
 	 */
 	if (is_local_partition(cs) &&
 	    (!is_partition_valid(parent) ||
+	     (parent == &top_cpuset &&
+	      top_cpuset_sd_nocpu(cs->effective_xcpus, NULL)) ||
 	     tasks_nocpu_error(parent, cs, &new_cpus) ||
 	     cpumask_empty(subpartitions_cpus)))
 		partcmd = partcmd_invalidate;
diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
index e813ea9959f4..f03fffeab034 100755
--- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
+++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
@@ -646,6 +646,14 @@ ISOLCPUS_DOMAIN_TEST_MATRIX=(
 
 	# An invalid partition can recover once it leaves a top-level HK CPU.
 	" C0-9:P1   .      .      .    C0-8     .      .      .     0 A1:0-8 A1:P1 ."
+
+	# Offlining CPU 1 leaves only boot-isolated CPUs in the top cpuset.
+	# A local root must not count its child's CPUs as already returned.
+	# In 1|, offline CPU 1 has no sched domain but is not cpuset-isolated.
+	" C0,2-9:P1 C5-9:P1 . . O1=0 . . . 0 A1:0,2-9|A2:5-9 A1:P-1|A2:P-1 1|"
+
+	# A remote root must return CPUs when the top domain loses its last HK CPU.
+	" X0,2-9 CX0,2-9:P1 . . O1=0 . . . 0 A2:0,2-9 A2:P-1 1|"
 )
 
 # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7
-- 
2.43.0