[RFC PATCH v3 13/17] sched/cache: Allow active balance to spread load from a full LLC

[Jianyong Wu]

Cache-aware aggregation can trigger active balance for migrate_llc_task
without waiting for nr_balance_failed. Spreading load from an LLC via
migrate_task normally requires nr_balance_failed to exceed
cache_nice_tries + 2. This delay can leave an LLC above the aggregation
threshold while another LLC has room.

Allow migrate_task to trigger active balance without that wait when the
source LLC exceeds the aggregation threshold and the destination can
accommodate one full CPU of additional load. Require the source to lead
the destination by at least two CPUs worth of load, so moving up to one
CPU of load does not reverse their utilization ordering and immediately
invite a migration back. The existing task migration checks still decide
whether the selected task may move.

Signed-off-by: Jianyong Wu <wujianyong@xxxxxxxx>
---
 kernel/sched/fair.c | 54 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c3f9ef8798d6..1e286296cd65 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13558,6 +13558,57 @@ imbalanced_active_balance(struct lb_env *env)
 	return 0;
 }
 
+#ifdef CONFIG_SCHED_CACHE
+/*
+ * Cache-aware aggregation can trigger active balance via migrate_llc_task
+ * without waiting for nr_balance_failed. Spreading load via migrate_task
+ * normally waits for nr_balance_failed to exceed cache_nice_tries + 2.
+ * This can leave one LLC above the aggregation threshold while another
+ * LLC has room.
+ *
+ * Allow that spread when the source is above the aggregation threshold,
+ * the destination has room for one CPU's load, and the source leads by
+ * at least two CPUs' load. The gap prevents one migration from reversing
+ * their utilization ordering and immediately inviting a migration back.
+ * The task migration checks still decide whether the task may move.
+ */
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	unsigned long src_util, src_cap, dst_util, dst_cap;
+
+	if (!sched_cache_enabled())
+		return false;
+
+	if (env->migration_type != migrate_task)
+		return false;
+
+	if (cpus_share_cache(env->src_cpu, env->dst_cpu))
+		return false;
+
+	if (!get_llc_stats(env->src_cpu, &src_util, &src_cap) ||
+	    !get_llc_stats(env->dst_cpu, &dst_util, &dst_cap))
+		return false;
+
+	if (fits_llc_capacity(src_util, src_cap, true))
+		return false;
+
+	/*
+	 * Reserve room for one CPU's load at the destination. A two-CPU gap
+	 * keeps one migration from reversing the utilization ordering and
+	 * causing the task to ping-pong between LLCs.
+	 */
+	if (!fits_llc_capacity(dst_util + SCHED_CAPACITY_SCALE, dst_cap, true))
+		return false;
+
+	return util_greater(src_util, dst_util);
+}
+#else
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	return false;
+}
+#endif
+
 static int need_active_balance(struct lb_env *env)
 {
 	struct sched_domain *sd = env->sd;
@@ -13571,6 +13622,9 @@ static int need_active_balance(struct lb_env *env)
 	if (imbalanced_active_balance(env))
 		return 1;
 
+	if (llc_spread_active_balance(env))
+		return 1;
+
 	/*
 	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
 	 * It's worth migrating the task if the src_cpu's capacity is reduced
-- 
2.34.1