Re: [PATCH][RFC] sched: Stop searching for idle cpu if the LLC domain is overloaded

[Peter Zijlstra]

On Mon, Feb 07, 2022 at 11:40:13AM +0800, Chen Yu wrote:
> It would be ideal to have a crystal ball to predict the success rate
> of finding an idle cpu/core in the LLC. If it is doomed to fail,
> there is no need to search in the LLC domain. There are many potential
> metrics which could be used to predict the success rate. And the
> metric should be carefully chosen that, it should help reduce the
> unnecessary cpu runqueue scan, but meanwhile not give up the opportunity
> to find an idle cpu.
> 
> Choose average cpu utilization as the candidate, since the util_avg is
> a metric of accumulated historic activity, which seems to be more accurate
> than instantaneous metrics(such as rq->nr_running) on calculating the probability
> of find an idle cpu. Only when the average cpu utilization has reaches
> 85% of the total cpu capacity, this domain is regarded as overloaded.
> The reason to choose 85% is that, this is the threshold of an overloaded
> LLC sched group(imbalance_pct = 117, threshold = 100 / 117 * 100%).

> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 5146163bfabb..1a58befe892d 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c

> @@ -6280,6 +6281,10 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>  	if (!this_sd)
>  		return -1;
>  
> +	sd_share = rcu_dereference(per_cpu(sd_llc_shared, target));
> +	if (sd_share && READ_ONCE(sd_share->overloaded))
> +		return -1;
> +
>  	cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);
>  
>  	if (sched_feat(SIS_PROP) && !has_idle_core) {

> @@ -9268,6 +9275,29 @@ static inline void update_sd_lb_stats(struct lb_env *env, struct sd_lb_stats *sd
>  		WRITE_ONCE(rd->overutilized, SG_OVERUTILIZED);
>  		trace_sched_overutilized_tp(rd, SG_OVERUTILIZED);
>  	}
> +
> +	/*
> +	 * Check if the LLC domain is overloaded. The overload hint
> +	 * could be used to skip the LLC domain idle cpu search in
> +	 * select_idle_cpu(). The update of this hint occurs during
> +	 * periodic load balancing, rather than frequent newidle balance.
> +	 */
> +	if (env->idle != CPU_NEWLY_IDLE &&
> +	    env->sd->span_weight == per_cpu(sd_llc_size, env->dst_cpu)) {
> +		struct sched_domain_shared *sd_share =
> +			rcu_dereference(per_cpu(sd_llc_shared, env->dst_cpu));
> +
> +		if (!sd_share)
> +			return;
> +
> +		/*
> +		 * Derived from group_is_overloaded(). The default imbalance_pct
> +		 * is 117 on LLC domain, which means the threshold of average
> +		 * utilization is 85%.
> +		 */
> +		WRITE_ONCE(sd_share->overloaded, (sds->total_capacity * 100) <
> +			   (sum_util * env->sd->imbalance_pct));
> +	}
>  }

So the only problem I have with this is that this is somewhat of a
binary toggle. The moment we hit that magical 85% we suddenly change
behaviour.

Would it not be possible to replace the SIS_PROP logic with something
based on this sum_util metric? Such that when sum_util is very low we
scan more, while when sum_util hits 85% we naturally stop scanning
entirely.

That way the behaviour is gradual.