Re: [PATCH v2] sched/fair: Prefer waker CPU for reciprocal sync wakeups

[Christian Loehle]

On 7/22/26 23:50, Shubhang Kaushik (Ampere) wrote:
> Pipe-style ping-pong workloads can be dominated by handoff cost. In
> such cases, placing the wakee on an idle CPU can be slower than keeping
> the pair on the same runqueue.
> 
> Use the existing last_wakee and wake_wide() state to identify narrow
> reciprocal WF_SYNC wakeups:
> 
> A wakes B
> B wakes A
> A wakes B
> ...
> 
> When the wake-affine domain allows SD_WAKE_AFFINE, prefer the waker CPU
> for these narrow reciprocal handoffs. Do so only when the waker CPU has no
> other runnable fair task, the wakee is allowed on that CPU, and the wakee
> fits there on asymmetric capacity systems.
> 
> Wakeups that do not match this pattern continue through the existing
> wake_affine() and select_idle_sibling() path.
> 
> Signed-off-by: Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>
> ---
> Tested on 80-core Ampere Altra: perf bench sched pipe -l 1000000 improved
> by about 30%, averaged over 20 runs. Hackbench, schbench and SPECjBB
> showed no material regression.
> 
> Baseline ~ v7.2-rc4 (mainline origin/master at 248951ddc14d)
> ---
> Changes in v2:
>   - Move the reciprocal handoff preference under the existing
>     SD_WAKE_AFFINE domain check.
>   - Drop futex from the changelog motivation.
>   - Refresh perf bench sched pipe results after rebasing.
> 
> Link to v1: https://lore.kernel.org/r/20260721-b4-sched-sync-wakeup-v1-1-dc94f184e27f@xxxxxxxxxx
> ---
>  kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
>  1 file changed, 28 insertions(+)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index d78467ec6ee1343050fcc2794dafb38ade3599e5..d188e91b85d74dd3ed3c8b99f171bd142e72da5e 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -8794,6 +8794,29 @@ static inline bool asym_fits_cpu(unsigned long util,
>  	return true;
>  }
>  
> +/*
> + * For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no
> + * other runnable fair task.
> + */
> +static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu)
> +{
> +	struct rq *rq = cpu_rq(cpu);
> +
> +	if ((rq->nr_running - cfs_h_nr_delayed(rq)) != 1)
> +		return false;
> +
> +	if (!cpumask_test_cpu(cpu, p->cpus_ptr))
> +		return false;
> +
> +	if (sched_asym_cpucap_active()) {
> +		sync_entity_load_avg(&p->se);
> +		if (!task_fits_cpu(p, cpu))
> +			return false;
> +	}
> +
> +	return true;
> +}

Maybe @Prateek:
What about SMT, would we want to require the sibling to be idle too?
I would appreciate a bench sched pipe on SMT result, if Shubhang doesn't
have one I can dust one off, too.

> +
>  /*
>   * Try and locate an idle core/thread in the LLC cache domain.
>   */
> @@ -9579,6 +9602,11 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
>  		 */
>  		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
>  		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
> +			if (sync &&
> +			    READ_ONCE(p->last_wakee) == current &&
> +			    prefer_sync_pair_cpu(p, cpu))
> +				return cpu;
> +
>  			if (cpu != prev_cpu)
>  				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
>  
> 
> ---
> base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
> change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da
> 
> Best regards,