[PATCH v3 18/24] sched/deadline: Credit a rebuild to the CPU recorded with the task's bandwidth

[Mathieu Desnoyers]

A rebuild of the root domains' accounting takes a CPU of the task's
cpuset for each deadline task it finds, dl_add_task_root_domain()
through dl_get_task_effective_cpus(), the first active one: a guess
that admission did not make, and three things go wrong with it.

 - The CPUs isolated by isolcpus="domain" belong to the default root
   domain, and the cpuset of a task that runs on a housekeeping CPU can
   hold them beside its housekeeping CPUs, as the root cpuset does;
   without cpusets the mask is every CPU the task may run on. The first
   active CPU of either is an isolated one where it comes first, and the
   task's bandwidth is counted in the default root domain instead of the
   one that admitted it, which underflows when the task leaves the
   class, or dies.
 - A task confined to isolated CPUs has them all taken offline, which
   nothing refuses: a deadline task of the default root domain does not
   keep the last of its CPUs online, dl_bw_deactivate() asking the root
   domain of the CPU going offline. The rebuild then finds no active CPU
   for it and BUG()s.
 - A throttled task whose CPUs went offline is moved by
   dl_task_offline_migration() to an active CPU, with its bandwidth,
   while its mask still names the offline ones: the rebuild counts it
   where the cpuset says, which is not where the move put it.

Take the CPU recorded with the task's bandwidth where it is active and
the task's cpuset still allows it: the rebuild then counts the bandwidth
where admission did, or where the last move put it, and the CPU that
dl_task_offline_migration() recorded answers for a task whose own CPUs
are offline. Where the record is of no use, its CPU offline or the
cpuset changed, take the first active CPU the cpuset allows, a
housekeeping one where it has any. The CPUs isolated at boot,
HK_TYPE_DOMAIN_BOOT, are the ones a cpuset can hold beside housekeeping
CPUs: those an isolated cpuset partition takes have a cpuset of their
own, and HK_TYPE_DOMAIN, which excludes them too, lags a change of
partitions by the rebuild that change makes,
cpuset_update_sd_hk_unlock() updating it after. Leave out a task with
no active CPU at all: the generation of "sched/deadline: Keep departing
bandwidth across a root domain rebuild" keeps the release of its
bandwidth out as well, so that nothing is taken from a root domain that
no longer counts it, and the next rebuild counts it again where it can.

Found by the Sashiko AI reviews of the first posting of this series,
the BUG() and the isolated CPU coming first, and of the second, the
housekeeping mask lagging a change of partitions and the task that
dl_task_offline_migration() moved.

Fixes: f9a25f776d78 ("cpusets: Rebuild root domain deadline accounting information")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
---
 kernel/sched/deadline.c | 45 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 40 insertions(+), 5 deletions(-)

diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index a058e0c65eca..bcf7f88b204a 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -3607,6 +3607,36 @@ static void dl_get_task_effective_cpus(struct task_struct *p, struct cpumask *cp
 	cpuset_cpus_allowed_locked(p, cpus);
 }
 
+/*
+ * A CPU of the root domain that a rebuild of the accounting counts @p's
+ * bandwidth in. The one recorded with it, where it is active and the
+ * cpuset of @p allows it: that of the root domain that admitted the
+ * bandwidth, or that the last move put it in, which the cpuset alone does
+ * not tell where it holds CPUs of two root domains, housekeeping ones
+ * beside the ones isolcpus="domain" leaves to the default root domain. The
+ * record answers as well for a task whose own CPUs went offline, moved to
+ * an active one by dl_task_offline_migration(). Else the first active CPU
+ * the cpuset allows, a housekeeping one where it has any; and nr_cpu_ids
+ * where it allows none, which the isolated CPUs allow, no deadline task
+ * of theirs keeping the last of them online.
+ */
+static unsigned int dl_task_rebuild_cpu(struct task_struct *p)
+{
+	struct cpumask *msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
+	int recorded = p->dl.dl_bw_cpu;
+	unsigned int cpu;
+
+	dl_get_task_effective_cpus(p, msk);
+	cpumask_and(msk, msk, cpu_active_mask);
+	if (recorded >= 0 && cpu_active(recorded) &&
+	    (cpumask_test_cpu(recorded, msk) || cpumask_empty(msk)))
+		return recorded;
+	cpu = cpumask_first_and(msk, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
+	if (cpu < nr_cpu_ids)
+		return cpu;
+	return cpumask_first(msk);
+}
+
 /* The caller should hold cpuset_mutex */
 void dl_add_task_root_domain(struct task_struct *p)
 {
@@ -3614,7 +3644,6 @@ void dl_add_task_root_domain(struct task_struct *p)
 	struct rq *rq;
 	struct dl_bw *dl_b;
 	unsigned int cpu;
-	struct cpumask *msk;
 
 	raw_spin_lock_irqsave(&p->pi_lock, rf.flags);
 	/*
@@ -3626,10 +3655,16 @@ void dl_add_task_root_domain(struct task_struct *p)
 		return;
 	}
 
-	msk = this_cpu_cpumask_var_ptr(local_cpu_mask_dl);
-	dl_get_task_effective_cpus(p, msk);
-	cpu = cpumask_first_and(cpu_active_mask, msk);
-	BUG_ON(cpu >= nr_cpu_ids);
+	cpu = dl_task_rebuild_cpu(p);
+	if (cpu >= nr_cpu_ids) {
+		/*
+		 * No active CPU to count this task's bandwidth: leave it out
+		 * of this rebuild, where dl_bw_gen keeps its release out as
+		 * well, until one has a CPU for it.
+		 */
+		raw_spin_unlock_irqrestore(&p->pi_lock, rf.flags);
+		return;
+	}
 
 	raw_spin_lock(&dl_departed_lock);
 	/*
-- 
2.47.3