[PATCH v2] sched/fair: Prefer waker CPU for reciprocal sync wakeups

[Shubhang Kaushik (Ampere)]

Pipe-style ping-pong workloads can be dominated by handoff cost. In
such cases, placing the wakee on an idle CPU can be slower than keeping
the pair on the same runqueue.

Use the existing last_wakee and wake_wide() state to identify narrow
reciprocal WF_SYNC wakeups:

A wakes B
B wakes A
A wakes B
...

When the wake-affine domain allows SD_WAKE_AFFINE, prefer the waker CPU
for these narrow reciprocal handoffs. Do so only when the waker CPU has no
other runnable fair task, the wakee is allowed on that CPU, and the wakee
fits there on asymmetric capacity systems.

Wakeups that do not match this pattern continue through the existing
wake_affine() and select_idle_sibling() path.

Signed-off-by: Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>
---
Tested on 80-core Ampere Altra: perf bench sched pipe -l 1000000 improved
by about 30%, averaged over 20 runs. Hackbench, schbench and SPECjBB
showed no material regression.

Baseline ~ v7.2-rc4 (mainline origin/master at 248951ddc14d)
---
Changes in v2:
  - Move the reciprocal handoff preference under the existing
    SD_WAKE_AFFINE domain check.
  - Drop futex from the changelog motivation.
  - Refresh perf bench sched pipe results after rebasing.

Link to v1: https://lore.kernel.org/r/20260721-b4-sched-sync-wakeup-v1-1-dc94f184e27f@xxxxxxxxxx
---
 kernel/sched/fair.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1343050fcc2794dafb38ade3599e5..d188e91b85d74dd3ed3c8b99f171bd142e72da5e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8794,6 +8794,29 @@ static inline bool asym_fits_cpu(unsigned long util,
 	return true;
 }
 
+/*
+ * For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no
+ * other runnable fair task.
+ */
+static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if ((rq->nr_running - cfs_h_nr_delayed(rq)) != 1)
+		return false;
+
+	if (!cpumask_test_cpu(cpu, p->cpus_ptr))
+		return false;
+
+	if (sched_asym_cpucap_active()) {
+		sync_entity_load_avg(&p->se);
+		if (!task_fits_cpu(p, cpu))
+			return false;
+	}
+
+	return true;
+}
+
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
@@ -9579,6 +9602,11 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		 */
 		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
 		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
+			if (sync &&
+			    READ_ONCE(p->last_wakee) == current &&
+			    prefer_sync_pair_cpu(p, cpu))
+				return cpu;
+
 			if (cpu != prev_cpu)
 				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
 

---
base-commit: 248951ddc14de84de3910f9b13f51491a8cd91df
change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da

Best regards,
-- 
Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>