[PATCH] sched/cache: Fix group_llc_balance being unreachable above the first NUMA level

[Jianyong Wu]

sched_balance_find_src_group() only considers group_llc_balance when
sds.prefer_sibling is set:

	if (sds.prefer_sibling && local->group_type == group_has_spare &&
	    (busiest->group_type == group_llc_balance ||
	     sibling_imbalance(env, &sds, busiest, local) > 1))
		goto force_balance;

sds.prefer_sibling mirrors SD_PREFER_SIBLING of the busiest group's child
sched domain, and sd_init() clears that flag on every SD_NUMA domain.
So on any domain whose child is itself a NUMA domain, i.e. on systems
with more than one NUMA level such as multiple nodes per socket, groups
tagged with group_llc_balance by llc_balance() can never take the
force_balance path, and the group_llc_balance test there is dead code.

Such a group falls into the !group_overloaded branch below and in the
common case ends up in out_balanced, so a task that was placed beyond
the first NUMA level is never pulled into its preferred LLC.

On a 2-socket Hygon system (8 nodes, 32 LLCs, 4 cores per LLC), with
"spin 3"/"spin 4" (3/4 busy threads of one process), all threads were
in one LLC in only 41/140 and 0/142 samples; every failing case had a
straggler on the other socket. With this patch: 195/196 and 180/196.

Only require sds.prefer_sibling for the sibling_imbalance() case, which
is what it was originally introduced for.

Fixes: f38cc2f0d8a3 ("sched/cache: Prioritize tasks preferring destination LLC during balancing")
Reported-by: Yibin Liu <liuyibin@xxxxxxxx>
Signed-off-by: Jianyong Wu <wujianyong@xxxxxxxx>
---
 kernel/sched/fair.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 57360f5cdde4..205a0891b0a0 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -13294,9 +13294,10 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)
 	 * Try to move all excess tasks to a sibling domain of the busiest
 	 * group's child domain.
 	 */
-	if (sds.prefer_sibling && local->group_type == group_has_spare &&
+	if (local->group_type == group_has_spare &&
 	    (busiest->group_type == group_llc_balance ||
-	    sibling_imbalance(env, &sds, busiest, local) > 1))
+	     (sds.prefer_sibling &&
+	      sibling_imbalance(env, &sds, busiest, local) > 1)))
 		goto force_balance;
 
 	if (busiest->group_type != group_overloaded) {
-- 
2.43.0