[PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share
From: Vinicius Costa Gomes
Date: Tue Aug 04 2026 - 19:14:33 EST
Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
strong" and could cause tasks to stack on a busy NUMA node while other
nodes are relatively idle.
As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
ep_poll_callback") improves real workloads a revert is not the answer.
The fix is to make the WF_SYNC "stack on waker" shortcut take into
account the load on this and prev's LLC, rejecting the shortcut only
when the waker (this) LLC is fully loaded and prev's LLC is less
loaded than the waker's.
Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@xxxxxxxxx>
---
We received a report of a regression on a openresty based
workload (the main metric being tail latencies) on a CWF SNC3 single
socket system, the main symptom that we could measure was one node
being overloaded while the other nodes were relatively idle.
Further investigation showed that spreading the NIC RX interrupts over
all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
synchronous wakeup support for ep_poll_callback") also helped.
Commit 900bbaae67e9 gave the hint, that perhaps for the WF_SYNC wakeup
path the 'sync' flag was too strong and was not taking the load into
account, which for workloads that are too quick for the NUMA load
balancer could perhaps cause this "over stacking".
The idea of this patch is to reject the stacking if it would overload
a LLC. I couldn't find a way to make it NUMA-only and only measured on
NUMA-like machines. One more reason for sending it as a RFC.
(I just became aware of [1], I haven't tested the series, but seems
that it's the same problem but approached from the "other side"; And
from [2], it's interesting, for this workload WF_SYNC is "too strong"
for others, it's too weak. Further documentation about the expected
behavior might be good indeed.)
To try and emulate the workload, which the wakeups happen mostly on
one node, I tried memcached + memtier_benchmark, with memcached
unpinned and memtier pinned to one of the NUMA nodes. As the customer
machine was single socket one, I offlined half the cores of the
machine I have access. (AI helped with this)
Kernels
-------
base master fc46aed51f62 (post-v7.2-rc5) vanilla
rfc base + sched/fair: decline WF_SYNC stacking when waker LLC is the
busier share
Results (memcached + memtier_benchmark, 10 repetitions, medians +/- stdev)
---------------------------------------------------------------
Results (memcached + memtier_benchmark, 10 repetitions, mean +/- stddev)
------------------------------------------------------------------------
Client (memtier, the wakers) pinned to one SNC node; memcached worker
threads span the whole socket. Latencies in milliseconds.
Load 64x32 (memtier --threads=64 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 3.179M +/- 0.027 3.072M +/- 0.039 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.488 +/- 0.006 0.493 +/- 0.013 +1.0%
p75 0.783 +/- 0.009 0.809 +/- 0.011 +3.3%
p90 1.265 +/- 0.010 1.312 +/- 0.017 +3.7%
p99 2.756 +/- 0.026 2.897 +/- 0.028 +5.1%
p99.9 3.753 +/- 0.064 4.005 +/- 0.117 +6.7%
p99.99 5.993 +/- 0.308 5.938 +/- 0.322 -0.9%
Load 96x32 (memtier --threads=96 --clients=32)
Throughput (higher is better)
metric base rfc delta
-------------------------------------------------------------
ops/sec 2.788M +/- 0.028 2.693M +/- 0.040 -3.4%
Latency (lower is better)
pctile base rfc delta
-------------------------------------------------------------
p50 (ms) 0.773 +/- 0.020 1.109 +/- 0.013 +43.5%
p75 1.321 +/- 0.013 1.135 +/- 0.016 -14.1%
p90 2.442 +/- 0.028 1.188 +/- 0.070 -51.4%
p99 4.857 +/- 0.020 1.974 +/- 0.141 -59.4%
p99.9 7.145 +/- 0.048 2.722 +/- 0.699 -61.9%
p99.99 9.637 +/- 0.230 4.324 +/- 0.888 -55.1%
[1] https://lore.kernel.org/all/20260714013940.4068189-4-srikar@xxxxxxxxxxxxx/
[2] https://lore.kernel.org/all/20260803-b4-sched-sync-wakeup-v4-1-52333b0cfb79@xxxxxxxxxx/
---
kernel/sched/fair.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 53 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index d78467ec6ee1..c0c6cdaefda4 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -8185,6 +8185,56 @@ static int wake_wide(struct task_struct *p)
return 1;
}
+/*
+ * Decline WF_SYNC "stack on waker" wakeup when it would overload this LLC.
+ *
+ * Only decline when prev and this do not share a cache, this LLC has
+ * run out of idle headroom and prev is actually less loaded.
+ *
+ * Returns true when WF_SYNC should be declined.
+ */
+static bool sched_llc_over_commited(int this_cpu, int prev_cpu)
+{
+ struct sched_domain_shared *sds;
+ int this_busy, this_size, prev_busy, prev_size;
+ int headroom;
+
+ /* Same LLC: stacking cannot spread anywhere. */
+ if (cpus_share_cache(this_cpu, prev_cpu))
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, this_cpu));
+ if (!sds)
+ return false;
+ /* FIXME: if NO_HZ nr_busy_cpus is not updated, always accept? */
+ this_busy = atomic_read(&sds->nr_busy_cpus);
+ this_size = per_cpu(sd_llc_size, this_cpu);
+
+ /*
+ * Decline only once the waker LLC has no spare CPU beyond the
+ * waker's own, which is going to be replaced by this sync
+ * wakeup (the caller already made sure that nr_running == 1).
+ * If at least another idle CPU remains, keep stacking.
+ */
+ headroom = 1;
+
+ if (this_size - this_busy > headroom)
+ return false;
+
+ sds = rcu_dereference_all(per_cpu(sd_llc_shared, prev_cpu));
+ if (!sds)
+ return false;
+ prev_busy = atomic_read(&sds->nr_busy_cpus);
+ prev_size = per_cpu(sd_llc_size, prev_cpu);
+
+ /*
+ * Waker LLC is near saturation: spread only if prev is the less
+ * busy fraction (cross multiplied to take into account different
+ * LLC sizes).
+ */
+ return (u64)this_busy * prev_size > (u64)prev_busy * this_size;
+}
+
/*
* The purpose of wake_affine() is to quickly determine on which CPU we can run
* soonest. For the purpose of speed we only consider the waking and previous
@@ -8218,7 +8268,8 @@ wake_affine_idle(int this_cpu, int prev_cpu, int sync)
if (sync) {
struct rq *rq = cpu_rq(this_cpu);
- if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1)
+ if ((rq->nr_running - cfs_h_nr_delayed(rq)) == 1 &&
+ !sched_llc_over_commited(this_cpu, prev_cpu))
return this_cpu;
}
@@ -8237,7 +8288,7 @@ wake_affine_weight(struct sched_domain *sd, struct task_struct *p,
this_eff_load = cpu_load(cpu_rq(this_cpu));
- if (sync) {
+ if (sync && !sched_llc_over_commited(this_cpu, prev_cpu)) {
unsigned long current_load = task_h_load(current);
if (current_load > this_eff_load)
---
base-commit: c21bb4193868a8de71fc4693fa741e195fdf5d86
change-id: 20260723-epoll-wasync-fix-v1-28d2531033c7
Best regards,
--
Vinicius