[PATCH 1/3] sched/fair: Fix asym packing to select correct cpu

[Srikar Dronamraju]

If asymmetric packing is used when target cpu is busy,
update_sd_pick_busiest(), can select a lightly loaded cpu.
find_busiest_group() has checks to ensure asym packing is only used
when target cpu is not busy.  However it may not be able to avoid a
lightly loaded cpu selected by update_sd_pick_busiest from being
selected as source cpu for eventual load balancing.

Also when using asymmetric scheduling, always select higher cpu as
source cpu for load balancing.

While doing this change, move the check to see if target cpu is busy
into check_asym_packing().

1. Record per second ebizzy (32 threads) on a 64 cpu power 7 box. (5 iterations)
4.5.0-master/ebizzy_32.out
    N           Min           Max        Median           Avg        Stddev
x   5       5205896      17260530      12141204      10759008       4765419

4.5.0-master-asym-changes/ebizzy_32.out
    N           Min           Max        Median           Avg        Stddev
x   5       7044349      19112876      17440718      14947658       5263970

2. Record per second ebizzy (32 threads) on a 64 cpu power 7 box. (5 iterations)
4.5.0-master/ebizzy_64.out
    N           Min           Max        Median           Avg        Stddev
x   5       5400083      14091418       8673907     8872662.4     3389746.8

4.5.0-master-asym-changes/ebizzy_64.out
    N           Min           Max        Median           Avg        Stddev
x   5       7533907      17232623      15083583      13364894     3776877.9

3. Record per second ebizzy (32 threads) on a 64 cpu power 7 box. (5 iterations)
4.5.0-master/ebizzy_128.out
    N           Min           Max        Median           Avg        Stddev
x   5      35328039      41642699      37564951      38378409       2671280

4.5.0-master-asym-changes/ebizzy_128.out
    N           Min           Max        Median           Avg        Stddev
x   5      37102220      42736809      38442478      39529626     2298389.4

Signed-off-by: Srikar Dronamraju <srikar@xxxxxxxxxxxxxxxxxx>
---
 kernel/sched/fair.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 56b7d4b..9abfb16 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6517,6 +6517,8 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 	if (!(env->sd->flags & SD_ASYM_PACKING))
 		return true;
 
+	if (env->idle == CPU_NOT_IDLE)
+		return true;
 	/*
 	 * ASYM_PACKING needs to move all the work to the lowest
 	 * numbered CPUs in the group, therefore mark all groups
@@ -6526,7 +6528,7 @@ static bool update_sd_pick_busiest(struct lb_env *env,
 		if (!sds->busiest)
 			return true;
 
-		if (group_first_cpu(sds->busiest) > group_first_cpu(sg))
+		if (group_first_cpu(sds->busiest) < group_first_cpu(sg))
 			return true;
 	}
 
@@ -6672,6 +6674,9 @@ static int check_asym_packing(struct lb_env *env, struct sd_lb_stats *sds)
 	if (!(env->sd->flags & SD_ASYM_PACKING))
 		return 0;
 
+	if (env->idle == CPU_NOT_IDLE)
+		return 0;
+
 	if (!sds->busiest)
 		return 0;
 
@@ -6864,8 +6869,7 @@ static struct sched_group *find_busiest_group(struct lb_env *env)
 	busiest = &sds.busiest_stat;
 
 	/* ASYM feature bypasses nice load balance check */
-	if ((env->idle == CPU_IDLE || env->idle == CPU_NEWLY_IDLE) &&
-	    check_asym_packing(env, &sds))
+	if (check_asym_packing(env, &sds))
 		return sds.busiest;
 
 	/* There is no busy sibling group to pull tasks from */
-- 
1.8.3.1