[RFC PATCH v3 14/17] sched/cache: Consider LLC load when selecting a single-task source rq

[Jianyong Wu]

When a source sched group spans multiple LLCs and every eligible rq has
at most one runnable task, migrate_task selects the first eligible rq.
It cannot distinguish an LLC above the aggregation threshold from one
at the threshold. If the selected rq is in an LLC at the threshold, the
cache-aware migration checks can veto the pull, while the LLC above the
threshold is never selected. The excess then persists.

For example, consider a two-socket machine with this topology:

  Socket 0: NUMA nodes 0-3
  Socket 1: NUMA nodes 4-7
  Each node: four LLCs
  Each LLC: four cores with SMT2, exposing eight logical CPUs

With a 50% aggregation threshold, each LLC can hold four full-utilization
tasks at the threshold. Suppose a workload has its preferred LLC in
node 6, with the node affinity order starting with 6, 7, 4, 5, followed
by the nodes in socket 0. One LLC in node 7 holds five CPU-bound tasks,
while every other LLC in socket 1 holds four. Each task runs on its own
rq, and socket 0 has spare capacity.

When socket 0 balances against socket 1, rq-level task counts select
the first CPU in node 4. Moving its task would leave node 4 below the
threshold, so the migration checks forbid it and active balance is not
started. Every such pass fails the same way: the five-task LLC stays
above the threshold while socket 0 has room. Balancing within node 7
does not help either, since moving a task to a sibling LLC with four
tasks merely reverses their task counts.

With cache-aware scheduling enabled, at domains above the LLC, track a
candidate rq in the LLC with the highest utilization within the source
group. Use it when the maximum runnable task count is one. In the
example, socket 0 then pulls from the five-task LLC. Node 7 remains
filled, so the move is permitted and brings that LLC back to the
threshold.

Keep task-count priority when an eligible rq has more than one runnable
task. Use the per-CPU scratch mask added earlier to sample each LLC
once during the existing scan.

Signed-off-by: Jianyong Wu <wujianyong@xxxxxxxx>
---
 kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 1e286296cd65..20a6a7f5c2f7 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13357,8 +13357,18 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 #ifdef CONFIG_SCHED_CACHE
 	int last_node = -1, node_num = 0, llc_max;
 	int *cache_data, *affi;
+	unsigned long llc_util, llc_cap;
+	struct sched_domain *llc_sd, *sd_self;
+	struct rq *llc_busiest = NULL;
+	unsigned long llc_busiest_util = 0;
+	struct cpumask *llcs_cpumask = NULL;
 
 	lb_affi_scratch(&cache_data, &affi, &llc_max);
+	sd_self = rcu_dereference_all(this_rq()->sd);
+	if (sd_self)
+		llcs_cpumask = sd_self->llc_stats_mask;
+	if (llcs_cpumask)
+		cpumask_clear(llcs_cpumask);
 #endif
 
 	for_each_cpu_and(i, sched_group_span(group), env->cpus) {
@@ -13472,6 +13482,34 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 				busiest_nr = nr_running;
 				busiest = rq;
 			}
+#ifdef CONFIG_SCHED_CACHE
+			/*
+			 * When every eligible rq has at most one runnable task,
+			 * task-count comparisons select the first non-empty rq,
+			 * favoring lower-numbered CPUs regardless of LLC load.
+			 * This can leave LLC load uneven even when individual
+			 * runqueues have the same number of runnable tasks.
+			 *
+			 * Track an rq in the LLC with the highest utilization
+			 * during the same scan. Use it only if no eligible rq
+			 * has more than one runnable task, preserving task-count
+			 * priority while helping resolve LLC imbalance.
+			 */
+			if (busiest_nr < 2 &&
+				!(env->sd->flags & SD_SHARE_LLC) &&
+				nr_running == 1 && sched_cache_enabled() &&
+				llcs_cpumask && !cpumask_test_cpu(i, llcs_cpumask) &&
+				get_llc_stats(i, &llc_util, &llc_cap)) {
+				if (!llc_busiest || llc_util > llc_busiest_util) {
+					llc_busiest_util = llc_util;
+					llc_busiest = rq;
+				}
+				llc_sd = rcu_dereference_all(per_cpu(sd_llc, i));
+				if (llc_sd)
+					cpumask_or(llcs_cpumask, llcs_cpumask,
+						   sched_domain_span(llc_sd));
+			}
+#endif
 			break;
 
 		case migrate_misfit:
@@ -13514,6 +13552,11 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,
 		}
 	}
 
+#ifdef CONFIG_SCHED_CACHE
+	if (busiest_nr == 1 && llc_busiest)
+		return llc_busiest;
+#endif
+
 	return busiest;
 }
 
-- 
2.34.1