[PATCH v7] sched/fair: Preserve wake-affine CPU for non-SMT reciprocal sync wakeups

[Shubhang Kaushik (Ampere)]

For WF_SYNC wakeups, wake_affine() may select or retain the waker CPU,
but select_idle_sibling() can subsequently move the wakee to another
idle CPU.

Pipe-style ping-pong workloads exercise the following handoff:

  A wakes B
  B wakes A
  A wakes B

When A wakes B for the second time, B last woke A. record_wakee(B)
records B as A's last wakee; together with B.last_wakee == A, this
identifies the reciprocal handoff.

Within the existing non-wide wake-affine path, on non-SMT systems, keep
this reciprocal WF_SYNC wakee on the waker CPU when wake-affine selected
or retained it and the waker has no other runnable task. This preserves
the direct handoff instead of searching for another idle CPU.

Record last_wakee history only in task context. In IRQ or softirq
context, current is the interrupted task rather than the logical waker.

This does not introduce a generic WF_SYNC placement rule. SMT systems
continue through select_idle_sibling(), which can choose an idle sibling
or idle core. On asymmetric-capacity systems, continue to require that
the wakee fits on the waker CPU.

Signed-off-by: Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>
---
Testing was performed on an 80-core non-SMT Ampere Altra system.

  perf bench sched pipe -l 1000000, 30 runs:

    baseline: mainline at a804c5742bff

    default:
      4.030 -> 2.366 usec/op mean,   about 41.3% improvement
      4.110 -> 2.334 usec/op median, about 43.2% improvement

    taskset -c 78,79:
      4.200 -> 2.348 usec/op mean,   about 44.1% improvement
      4.294 -> 2.316 usec/op median, about 46.1% improvement

Changes in v7:
  - Record the wakee before testing the reciprocal wakeup relationship.
  - Explicitly require both directions of the current <-> wakee history,
    making the reciprocal handoff visible at the placement decision.

Link to v6: https://lore.kernel.org/r/20261012-b4-sched-sync-wakeup-v6-1-aa70cde14463@xxxxxxxxxx
---
 kernel/sched/fair.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index b5546cd27dba08603fad190ba78f4bc1811c8452..0d7fb30170aa4850c2430c4747fe66499c636e3e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8470,6 +8470,10 @@ static unsigned long capacity_of(int cpu)
 
 static void record_wakee(struct task_struct *p)
 {
+	/* IRQ/softirq current is the interrupted task, not the logical waker. */
+	if (!in_task())
+		return;
+
 	/*
 	 * Only decay a single time; tasks that have less then 1 wakeup per
 	 * jiffy will not have built up many flips.
@@ -9145,6 +9149,22 @@ static inline bool asym_fits_cpu(unsigned long util,
 	return true;
 }
 
+static bool can_stack_on_waker(struct task_struct *p, int cpu)
+{
+	struct rq *rq = cpu_rq(cpu);
+
+	if ((rq->nr_running - cfs_h_nr_delayed(rq)) != 1)
+		return false;
+
+	if (sched_asym_cpucap_active()) {
+		sync_entity_load_avg(&p->se);
+		if (!task_fits_cpu(p, cpu))
+			return false;
+	}
+
+	return true;
+}
+
 /*
  * Try and locate an idle core/thread in the LLC cache domain.
  */
@@ -9953,6 +9973,19 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)
 			if (cpu != prev_cpu)
 				new_cpu = wake_affine(tmp, p, cpu, prev_cpu, sync);
 
+			/*
+			 * record_wakee() established current->last_wakee == p.
+			 * A reciprocal wakeup also requires that p was last
+			 * woken by current.
+			 */
+			if (sync && !sched_smt_active() &&
+			    new_cpu == cpu &&
+			    in_task() &&
+			    current->last_wakee == p &&
+			    p->last_wakee == current &&
+			    can_stack_on_waker(p, cpu))
+				return cpu;
+
 			sd = NULL; /* Prefer wake_affine over balance flags */
 			break;
 		}

---
base-commit: a804c5742bff224d4cb4d25dd383554d598be32b
change-id: 20260721-b4-sched-sync-wakeup-04d40cbeb1da

Best regards,
-- 
Shubhang Kaushik (Ampere) <sh@xxxxxxxxxx>