Re: sched/deadline, cpuset: total_bw corrupted by a sleeping DL task's cpuset move

[Juri Lelli]

Hello,

On 31/10/26 17:08, Weiheng Qiu wrote:
> Hi,
> 
> Moving a sleeping SCHED_DEADLINE task from the root cpuset into an
> exclusive partition and back, before it runs again, leaves the root
> domain short by the task's bandwidth. The root then over-admits, and
> when the task exits its total_bw underflows: small DL requests fail
> with -EBUSY while large ones wrap in __dl_overflow() and are admitted.
> The partition also keeps a stale reservation.
> 
> cpuset_can_attach() and set_cpus_allowed_dl() both decide through
> dl_task_needs_bw_move(), which tests task_rq(p)->rd. A sleeping task
> is not migrated when its affinity changes, so task_rq(p) still points
> at the CPU it last ran on, and on the move back the root domain is
> neither re-credited nor the partition debited.
> 
> Reproducer below; needs root, cgroup v2 and debugfs, and turns the
> last CPU into a partition, so use a test VM. On mainline c339f4010dea
> with 4 CPUs:
> 
>   cpu3 is a partition; sleeping DL task 76 (0.5 CPU) in the root cgroup
>   total_bw in CPUs (change since start):
>   start:                     root  0.650 (+0.000)   partition  0.050 (+0.000)
>   moved into the partition:  root  0.150 (-0.500)   partition  0.550 (+0.500)
>   moved back to the root:    root  0.150 (-0.500)   partition  0.550 (+0.500)
>   (expected: +0.000 and +0.000; the task is in state S and has not run)
>   root domain admits 28 more 0.1-CPU tasks: 0.650 + 28 x 0.1 = 3.450 CPU
>   reserved, limit 3.000 CPU (3 CPUs x 1000000/1000000)
>   those tasks exited:        root  0.150 (-0.500)   partition  0.550 (+0.500)
>   the DL task exited:        root -0.350 (-1.000)   partition  0.550 (+0.500)
>   DL tasks left: 0; new 0.010-CPU request: Device or resource busy
>   DL tasks left: 0; new 0.500-CPU request: accepted
>   after a root-domain rebuild (recomputed): root 0.150, partition 0.050
>   moved in while asleep, then SCHED_DEADLINE: Operation not permitted

Thanks for the report and reproducer.

Can you check if what below helps with it?

Best,
Juri
---
 kernel/sched/deadline.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 4d66e7298b686..2eebb688891a8 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3324,14 +3324,26 @@ static void task_woken_dl(struct rq *rq, struct task_struct *p)
 	}
 }
 
+/*
+ * Find the root domain where a DL task's bandwidth is currently
+ * accounted. For a sleeping task that was moved between cpusets
+ * without running, task_rq(p)->rd may be stale. Use the task's
+ * current cpus_ptr (not yet updated to the new mask) to find the
+ * actual root domain.
+ */
+static unsigned int dl_task_root_domain_cpu(struct task_struct *p)
+{
+	unsigned int cpu = cpumask_first_and(p->cpus_ptr, cpu_active_mask);
+
+	BUG_ON(cpu >= nr_cpu_ids);
+	return cpu;
+}
+
 static void set_cpus_allowed_dl(struct task_struct *p,
 				struct affinity_context *ctx)
 {
-	struct rq *rq;
-
 	WARN_ON_ONCE(!dl_task(p));
 
-	rq = task_rq(p);
 	/*
 	 * Migrating a SCHED_DEADLINE task between exclusive
 	 * cpusets (different root_domains) entails a bandwidth
@@ -3339,16 +3351,16 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 	 * domain (see cpuset_can_attach()).
 	 */
 	if (dl_task_needs_bw_move(p, ctx->new_mask)) {
-		struct dl_bw *src_dl_b;
+		unsigned int src_cpu = dl_task_root_domain_cpu(p);
+		struct dl_bw *src_dl_b = &cpu_rq(src_cpu)->rd->dl_bw;
 
-		src_dl_b = dl_bw_of(cpu_of(rq));
 		/*
 		 * We now free resources of the root_domain we are migrating
 		 * off. In the worst case, sched_setattr() may temporary fail
 		 * until we complete the update.
 		 */
 		raw_spin_lock(&src_dl_b->lock);
-		__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(task_cpu(p)));
+		__dl_sub(src_dl_b, p->dl.dl_bw, dl_bw_cpus(src_cpu));
 		raw_spin_unlock(&src_dl_b->lock);
 	}
 
@@ -3358,10 +3370,13 @@ static void set_cpus_allowed_dl(struct task_struct *p,
 bool dl_task_needs_bw_move(struct task_struct *p,
 			   const struct cpumask *new_mask)
 {
+	unsigned int cpu;
+
 	if (!dl_task(p))
 		return false;
 
-	return !cpumask_intersects(task_rq(p)->rd->span, new_mask);
+	cpu = dl_task_root_domain_cpu(p);
+	return !cpumask_intersects(cpu_rq(cpu)->rd->span, new_mask);
 }
 
 /* Assumes rq->lock is held */