Re: [PATCH RFC] sched/fair: decline WF_SYNC stacking when waker LLC is the busier share

[Vinicius Costa Gomes]

Hillf Danton <hdanton@xxxxxxxx> writes:

> On Thu, 06 Aug 2026 10:44:18 -0700 Vinicius Costa Gomes wrote:
>>Hillf Danton <hdanton@xxxxxxxx> writes:
>>> On Tue, 04 Aug 2026 16:14:05 -0700 Vinicius Costa Gomes wrote:
>>>> Since commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>> ep_poll_callback"), epoll driven WF_SYNC wakeups have been "too
>>>> strong" and could cause tasks to stack on a busy NUMA node while other
>>>> nodes are relatively idle.
>>>> 
>>>> As commit 900bbaae67e9 ("epoll: Add synchronous wakeup support for
>>>> ep_poll_callback") improves real workloads a revert is not the answer.
>>>> The fix is to make the WF_SYNC "stack on waker" shortcut take into
>>>> account the load on this and prev's LLC, rejecting the shortcut only
>>>> when the waker (this) LLC is fully loaded and prev's LLC is less
>>>> loaded than the waker's.
>>>> 
>>>> Signed-off-by: Vinicius Costa Gomes <vinicius.gomes@xxxxxxxxx>
>>>> ---
>>>> We received a report of a regression on a openresty based
>>>> workload (the main metric being tail latencies) on a CWF SNC3 single
>>>> socket system, the main symptom that we could measure was one node
>>>> being overloaded while the other nodes were relatively idle.
>>>> 
>>>> Further investigation showed that spreading the NIC RX interrupts over
>>>> all NUMA nodes helped. Reverting commit 900bbaae67e9 ("epoll: Add
>>>> synchronous wakeup support for ep_poll_callback") also helped.
>>>> 
>>> The irq approach is prefered because anything that gets the eevdf offloaded
>>> is good, you see it is near to the knowall point, needless to say that they
>>> lie in different layers and from the scheduling-cpu pov irq is a gray rhino
>>> in the room while WF_SYNC is a tiger mosquito in the corner at best in your
>>> case where the mosquito failed to understand your workload.
>>
>> I don't think irq spreading across NUMA nodes is that good of an idea on
>> low loads/by default, as it loses the locality that the kernel (even
>> with irqbalance) try to maintain. I used that as a hackish way of
>> testing "if I spread tasks, does it improve the tail latencies?".
>> 
> Can you specify the root cause of the tail latencies, particularly after
> spending two minutes thinking if changing config in user space to solve the
> issue is a light year better than adding a couple lines of code in the
> wakeup path?
>

On a machine running the customer workload, I ran a bpftrace script that
tracks the time "openresty" tasks stay on the runqueue waiting to be run.

(this was a using an earlier version of the patch, on top of v7.1-rc7, I
don't have easy access to the machine anymore)

Before:

@runq_us:
[0]                 5376 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]                 5122 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
[2, 4)              5073 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   |
[4, 8)              3821 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)             1649 |@@@@@@@@@@@@@@@                                     |
[16, 32)             747 |@@@@@@@                                             |
[32, 64)             552 |@@@@@                                               |
[64, 128)            668 |@@@@@@                                              |
[128, 256)           907 |@@@@@@@@                                            |
[256, 512)          1235 |@@@@@@@@@@@                                         |
[512, 1K)           1946 |@@@@@@@@@@@@@@@@@@                                  |
[1K, 2K)            1832 |@@@@@@@@@@@@@@@@@                                   |
[2K, 4K)            1130 |@@@@@@@@@@                                          |
[4K, 8K)             561 |@@@@@                                               |
[8K, 16K)            340 |@@@                                                 |
[16K, 32K)           250 |@@                                                  |
[32K, 64K)           108 |@                                                   |
[64K, 128K)           33 |                                                    |
[128K, 256K)           0 |                                                    |
[256K, 512K)           0 |                                                    |
[512K, 1M)             1 |                                                    |

After:

@runq_us:
[0]              1781353 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1]               450223 |@@@@@@@@@@@@@                                       |
[2, 4)             71505 |@@                                                  |
[4, 8)             21345 |                                                    |
[8, 16)            22406 |                                                    |
[16, 32)           14792 |                                                    |
[32, 64)            7336 |                                                    |
[64, 128)           3867 |                                                    |
[128, 256)          4607 |                                                    |
[256, 512)          6886 |                                                    |
[512, 1K)           5862 |                                                    |
[1K, 2K)            2829 |                                                    |
[2K, 4K)             876 |                                                    |
[4K, 8K)              11 |                                                    |

This made me think that the almost unconditional stacking shortcut that
WF_SYNC promotes was causing tasks to wait on already busy CPUs, while
there were idle CPUs around.

That was as close to a root cause that I got.

>> Note that the "local-only reproducer" workload (memcached +
>> memtier_benchmark) runs over loopback (no NIC irqs here), I pin memtier
>> (the client) to one NUMA node, leave the server unpinned and I am able
>> to reproduce the issue: with the RFC patch the tail latencies reduce by
>> 2-3x. (on the customer workload the impact is even higher)
>> 
>> My expectation was that the scheduler would be able to say: "ugh, even
>> though respecting WF_SYNC is good most of the cases, this isn't one of
>
> No comment before root cause specified, even if I suspect it sounds like
> that two CPU cores could not provide line speed 24-port 1000MB ether switch
> before 2010 while 12 cores could.
>
>> them". And this is the spirit of the RFC, showing that those cases
>> exist and their impact.
>>
>>
>>Cheers,
>>-- 
>>Vinicius

-- 
Vinicius