[PATCH v3] sched/fair: Prefer waker CPU for non-SMT reciprocal sync wakeups

[Shubhang Kaushik (Ampere)]

Pipe-style ping-pong workloads can be dominated by handoff cost. In
such cases, placing the wakee on an idle CPU can be slower than keeping
the pair on the same runqueue.

Use the existing last_wakee and wake_wide() state to identify narrow
reciprocal WF_SYNC wakeups:

A wakes B
B wakes A
A wakes B
...

When the wake-affine domain allows SD_WAKE_AFFINE, prefer the waker CPU
for these narrow reciprocal handoffs on non-SMT systems. Do so only when
the waker CPU has no other runnable fair task and the wakee fits there on
asymmetric-capacity systems.

SMT systems, and wakeups that do not match this pattern, continue through
the existing wake_affine() and select_idle_sibling() path.

Signed-off-by: Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>
---
Tested on 80-core non-SMT Ampere Altra: perf bench sched pipe -l 1000000
improved by about 30%, averaged over 40 runs. Hackbench, schbench and
SPECjBB showed no material regression.

Baseline: v7.2-rc5
---
Changes in v3:
  - Limit the direct waker-CPU preference to !sched_smt_active(); SMT
    systems continue through the existing wake_affine() and
    select_idle_sibling() path.
  - Drop the redundant affinity check; want_affine already verifies the
    waker CPU is allowed.
  - Use a plain p->last_wakee read instead of READ_ONCE().
  - Rebase and refresh testing on v7.2-rc5.

Link to v2: https://lore.kernel.org/r/20260722-b4-sched-sync-wakeup-v2-1-f1164560b24b@xxxxxxxxxx

Changes in v2:
  - Move the reciprocal handoff preference under the existing
    SD_WAKE_AFFINE domain check.
  - Drop futex from the changelog motivation.
  - Refresh perf bench sched pipe results after rebasing.

Link to v1: https://lore.kernel.org/r/20260721-b4-sched-sync-wakeup-v1-1-dc94f184e27f@xxxxxxxxxx
---
 kernel/sched/fair.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1343050fcc2794dafb38ade3599e5..e61062d20da772d29da6f5f377a150b4b5128619 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8794,6 +8794,26 @@ static inline bool asym_fits_cpu(unsigned long util,
 	return true;
 }
 
+/*
+ * For reciprocal WF_SYNC handoffs, prefer the waker CPU when it has no
+ * other runnable fair task.
+ */
+static bool prefer_sync_pair_cpu(struct task_struct *p, int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if ((rq->nr_running - cfs_h_nr_delayed(rq)) != 1)
+		return false;
+
+	if (sched_asym_cpucap_active()) {
+		sync_entity_load_avg(&p->se);
+		if (!task_fits_cpu(p, cpu))
+			return false;
+	}
+
+	return true;
+}
+
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
@@ -9579,6 +9599,11 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 		 */
 		if (want_affine && (tmp->flags & SD_WAKE_AFFINE) &&
 		    cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))) {
+			if (sync && !sched_smt_active() &&
+			    p->last_wakee == current &&
+			    prefer_sync_pair_cpu(p, cpu))
+				return cpu;
+
 			if (cpu != prev_cpu)
 				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
 

---
base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff
change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da

Best regards,
-- 
Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>