Re: [BUG] cpuset: empty sched domain with isolcpus=domain, panic at CPU offline
From: Waiman Long
Date: Sun Nov 08 2026 - 13:37:07 EST
On 11/8/26 9:57 AM, Mathieu Desnoyers wrote:
Hi,
With isolcpus=domain, cpuset partitions can take all the housekeeping
CPUs. The top cpuset then gets an empty sched domain. In a VM, this
gave a WARNING in build_sched_domains(), then a panic at the next CPU
offline. I hit it once, while testing a SCHED_DEADLINE series [1]. The
analysis below is from reading the code after that panic. A proposed
fix follows the scissors line.
The code involved is the same in v7.4-rc1 and in tip sched/urgent
(006991aa8997). The series does not change it.
Trigger
=======
Boot 5 CPUs with isolcpus=domain,0,1. Then, with cgroup v2:
cd /sys/fs/cgroup
echo +cpuset > cgroup.subtree_control
mkdir P Q
echo 4 > P/cpuset.cpus
echo root > P/cpuset.cpus.partition
echo 2-3 > Q/cpuset.cpus
echo root > Q/cpuset.cpus.partition # WARNING
echo 0 > /sys/devices/system/cpu/cpu2/online # panic
The test that hit it made these writes. It also ran a deadline task in
Q, which the analysis does not need. After the writes, the top cpuset
keeps only CPUs 0 and 1, which are isolated.
Log
===
WARNING: kernel/sched/topology.c:3111 at build_sched_domains+0x1321/0x1830, CPU#0: orig/91
CPU: 0 UID: 0 PID: 91 Comm: orig Not tainted 7.4.0-rc1-00300-gc36e8c1d51c9 #98 PREEMPT(lazy)
Call Trace:
partition_sched_domains+0x331/0x990
cpuset_update_sd_hk_unlock+0x76/0x80
cpuset_partition_write+0x9e/0xe0
kernfs_fop_write_iter+0x14d/0x210
vfs_write+0x252/0x550
ksys_write+0x6c/0xe0
do_syscall_64+0x109/0x5d0
[...]
BUG: unable to handle page fault for address: 0000000083cf6513
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
Oops: Oops: 0000 [#1] SMP NOPTI
CPU: 2 UID: 0 PID: 26 Comm: cpuhp/2 Tainted: G W 7.4.0-rc1-00300-gc36e8c1d51c9 #98 PREEMPT(lazy)
RIP: 0010:partition_sched_domains+0x5ee/0x990
RAX: 0000000083cf58c3 RBX: 0000000000000003 RCX: ff11000004f60b58
R10: 0000000000000200 R11: 0000000000000004 R12: 0000000000000040
Call Trace:
cpuset_update_active_cpus+0x512/0xb00
sched_cpu_deactivate+0x1f3/0x300
cpuhp_invoke_callback+0x1b4/0x780
cpuhp_thread_fun+0x183/0x260
smpboot_thread_fn+0x143/0x300
kthread+0xfe/0x130
ret_from_fork+0x22a/0x330
Kernel panic - not syncing: Fatal exception
Analysis
========
The top cpuset may hold CPUs isolated at boot. generate_sched_domains()
keeps a partition root if its effective_cpus is not empty. For the top
cpuset, it then builds the domain from effective_cpus without these
CPUs:
if ((cp->partition_root_state == PRS_ROOT) &&
!cpumask_empty(cp->effective_cpus))
csa[ndoms++] = cp;
[...]
if (!csa || csa[i] == &top_cpuset)
cpumask_and(doms[i], top_cpuset.effective_cpus,
housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
Here effective_cpus is 0-1, so the domain is empty.
build_sched_domains() warns and builds nothing. But
partition_sched_domains_locked() still records the empty mask in
doms_cur.
This is a known problem and we are working to address this in the cpuset code to make sure that there is at least one online domain housekeeping CPUs for the top_cpuset.
At the CPU offline, cpuset passes the empty domain again. It matches
the empty one in doms_cur. The reuse branch then reads the runqueue of
the first CPU of the mask:
int cpu = cpumask_first(doms_cur[j]);
guard(rcu)();
sd = rcu_dereference(cpu_rq(cpu)->sd);
On an empty mask, cpumask_first() returns NR_CPUS when NR_CPUS is at
most BITS_PER_LONG. This kernel has NR_CPUS=64. R12 = 0x40 is that
index into __per_cpu_offset[], one entry past its end. The load of
rq->sd then faults.
The partition checks do not refuse Q. tasks_nocpu_error() tests the
effective_cpus of the top cpuset, which still holds CPUs 0 and 1.
prstate_housekeeping_conflict() keeps boot time isolated CPUs out of
root partitions. Nothing asks the top cpuset to keep a housekeeping
CPU.
Versions:
- The empty domain comes from 9b496a8bbed9 ("cgroup/cpuset: Prevent
leakage of isolated CPUs into sched domains"), in v6.13.
- The reuse branch comes from a7660ce1590f ("sched/cache: Fix
has_multi_llcs iff at least one partition has multiple LLCs"), in
v7.2.
From v6.13 to v7.1, a similar read would happen elsewhere. This is from
reading only:
- detach_destroy_domains() reads per_cpu() of cpumask_any() of the
mask, when the empty domain goes away.
- With CONFIG_ENERGY_MODEL, build_perf_domains() reads
cpu_rq(cpumask_first()) of the mask at its first build.
Where to fix
============
Commit cd1cb3350561 ("sched/topology: Don't try to build empty sched
domains") fixed a crash of this kind in 2019. It made
generate_sched_domains() skip empty masks. Its changelog says that the
scheduler should not have to check cpumask_first() against nr_cpu_ids.
So the patch below fixes cpuset. It keeps a partition root only if its
effective_cpus holds a housekeeping CPU.
Two questions:
- Should cpuset refuse a partition that leaves the top cpuset without
a housekeeping CPU? The patch keeps the current behaviour. The
partition is valid. The tasks of the top cpuset run on the isolated
CPUs.
Yes, we are going to add the check to make sure that there will be at least one housekeeping CPU left for the top_cpuset sched domain.
See https://lore.kernel.org/lkml/20261105130344.664336-1-guopeng.zhang@xxxxxxxxx/
- Should the scheduler also skip empty masks? The reuse branch, the
perf domain code and detach_destroy_domains() all take a CPU from
the mask.
Testing1
=======
With the patch on tip sched/urgent (006991aa8997), in a VM with
5 CPUs:
- isolcpus=domain,0,1 and the sequence above. There is no WARNING.
All CPU offlines and onlines complete. Q gets the domain 2-3, and the
top cpuset gets none. The top cpuset gets the domain 2-3 back when Q
becomes a member again.
- The same boot, with P as an isolated partition. Q is a root
partition first, then isolated too. With both isolated, no partition
root has a housekeeping CPU. The scheduler then gets no domain, and
CPU hotplug works. Making Q isolated also warns in
cpuset_update_sd_hk_unlock(). That is a separate problem, which I
report separately. The patch does not touch that path.
- No isolcpus, and the same sequence. There is no WARNING. The top
cpuset keeps its domain 0-1.
test_cpuset_prs.sh passes on 9 CPUs. It also passes on 10 CPUs with
isolcpus=domain,9. I did not run the crash sequence again on a kernel
without the fix.
The analysis and the patch were written by an LLM coding assistant
under my direction and review.
Thanks,
Mathieu
[1] https://lore.kernel.org/all/20261107014451.1487825-1-mathieu.desnoyers@xxxxxxxxxxxx/
-- >8 --
From: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
Subject: cgroup/cpuset: Do not build an empty sched domain for the top cpuset
The scheduler expects each sched domain from cpuset to hold at least
one CPU. Commit cd1cb3350561 ("sched/topology: Don't try to build empty
sched domains") made generate_sched_domains() skip a cpuset with an
empty effective_cpus. It also made build_sched_domains() warn on an
empty mask.
The top cpuset may hold CPUs isolated at boot. Commit 9b496a8bbed9
("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
removes them from its sched domain. The check for an empty domain was
not updated. It still tests effective_cpus, which includes the isolated
CPUs.
So partitions that take all the housekeeping CPUs leave the top cpuset
with an empty domain. For example, boot 5 CPUs with isolcpus=domain,0,1.
Then make CPUs 2-3 and 4 root partitions. build_sched_domains() warns.
The scheduler still records the empty mask as a current domain. At the
next rebuild, the new empty domain matches it.
partition_sched_domains_locked() then calls cpu_rq() on cpumask_first()
of the empty mask, which is not a valid CPU. In a VM, the next CPU
offline panicked there.
Fix this by keeping a partition root only if its effective_cpus holds a
housekeeping CPU. This tests the mask that the top cpuset gets.
prstate_housekeeping_conflict() keeps boot time isolated CPUs out of
the other partition roots. So nothing changes for them.
This change was developed with assistance from an LLM.
Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
---
kernel/cgroup/cpuset.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 3219f927c35e..426c98cf4a08 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -894,10 +894,13 @@ static int generate_sched_domains(cpumask_var_t **domains,
cpuset_for_each_descendant_pre(cp, pos_css, &top_cpuset) {
/*
* Only valid partition roots that are not isolated and with
- * non-empty effective_cpus will be saved into csa[].
+ * housekeeping CPUs in effective_cpus will be saved into csa[].
+ * The top cpuset may keep only boot time isolated CPUs. It then
+ * gets no sched domain.
*/
if ((cp->partition_root_state == PRS_ROOT) &&
- !cpumask_empty(cp->effective_cpus))
+ cpumask_intersects(cp->effective_cpus,
+ housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT)))
csa[ndoms++] = cp;
/*
This patch is correct AFAICS. I am not against it but it may not be necessary after we make sure that there is at least one HK online CPU in the top sched domain.
Cheers,
Longman