[RFC PATCH v2 16/23] sched/cache: Allow un-throttled active balance to spread out of a full LLC

[Jianyong Wu]

Cache-aware aggregation reaches active balance through two asymmetric
paths. Pulling a task back to its preferred LLC uses migrate_llc_task,
which need_active_balance() lets through unconditionally. Pushing a task
the other way. Out of an LLC that is already over the aggregation cap
and into one that still has room. It only happens via migrate_task, which
first has to wait for nr_balance_failed to exceed cache_nice_tries + 2.

Let the spread direction reach active balance without the
nr_balance_failed wait as well. The source LLC must already be over the
aggregation cap and the destination must still fit under it with a full
CPU of load added, so this can only undo an overshoot and never fights
the aggregation itself.

Signed-off-by: Jianyong Wu <wujianyong@xxxxxxxx>
---
 kernel/sched/fair.c | 58 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 0defdd4316b8..42ab6347e667 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13506,6 +13506,61 @@ imbalanced_active_balance(struct lb_env *env)
 	return 0;
 }
 
+#ifdef CONFIG_SCHED_CACHE
+/*
+ * Cache-aware aggregation pulls tasks back to the preferred LLC via
+ * migrate_llc_task, which reaches active balance without any
+ * nr_balance_failed gating. Spreading tasks the other way. It only
+ * happens through migrate_task, which must first wait for
+ * nr_balance_failed to exceed cache_nice_tries + 2. That asymmetry
+ * lets a few LLCs stay oversubscribed while other LLCs of the same
+ * preferred region stay idle.
+ *
+ * Allow the spread direction to active balance un-throttled as well. The
+ * source LLC must be over the aggregation cap and the destination must
+ * still fit under it, so this can only undo an overshoot and never fights
+ * the aggregation itself. Region containment is still enforced by
+ * alb_break_llc()/can_migrate_node(), which run before this.
+ */
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	unsigned long src_util, src_cap, dst_util, dst_cap;
+
+	if (!sched_cache_enabled())
+		return false;
+
+	if (env->migration_type != migrate_task)
+		return false;
+
+	if (cpus_share_cache(env->src_cpu, env->dst_cpu))
+		return false;
+
+	if (!get_llc_stats(env->src_cpu, &src_util, &src_cap) ||
+	    !get_llc_stats(env->dst_cpu, &dst_util, &dst_cap))
+		return false;
+
+	if (fits_llc_capacity(src_util, src_cap))
+		return false;
+
+	/*
+	 * Only move when the destination still fits with a full CPU of the load
+	 * added, and when the source leads by at least two CPUs worth, so that
+	 * moving one task cannot invert the imbalance and bounce it straight
+	 * back.
+	 */
+	if (!fits_llc_capacity(dst_util + SCHED_CAPACITY_SCALE, dst_cap))
+		return false;
+
+	/* Avoid threads ping-pong migration */
+	return util_greater(src_util, dst_util);
+}
+#else
+static inline bool llc_spread_active_balance(struct lb_env *env)
+{
+	return false;
+}
+#endif
+
 static int need_active_balance(struct lb_env *env)
 {
 	struct sched_domain *sd = env->sd;
@@ -13519,6 +13574,9 @@ static int need_active_balance(struct lb_env *env)
 	if (imbalanced_active_balance(env))
 		return 1;
 
+	if (llc_spread_active_balance(env))
+		return 1;
+
 	/*
 	 * The dst_cpu is idle and the src_cpu CPU has only 1 CFS task.
 	 * It's worth migrating the task if the src_cpu's capacity is reduced
-- 
2.34.1