Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Jens Axboe]

On 10/19/26 11:29 PM, Austin Schuh wrote:
> On Mon, Oct 19, 2026 at 5:54?PM Jens Axboe <axboe@xxxxxxxxx> wrote:
>> Patches like this have been posted before, and my question is
>> always "is this a common case?"? Reading this:
>>
>>> Notes:
>>>     Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
>>>     PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
>>>     change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
>>>     by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
>>>     with a value the futex word doesn't hold.  Happy to turn that into
>>>     a liburing test if it's useful.
>>
>> makes me very much think "nope, not at all", as it's a deliberately
>> failing test case. And yes if you keep issuing failing waits,
>> then it'll drain the cache. The cache is for the hot FAST path,
>> which is not an error path.
> 
> Fair on the test case, it only showed the mechanism.  But for an event
> loop, -EAGAIN from a futex wait isn't an error path.  Right before
> io_uring_enter() re-arms the wait that just fired, the event loop loads
> its futex word.  If the word has already changed, it handles the
> new message and re-arms with the new value instead.  A message published
> between that load and futex_wait_setup() checking the word makes the
> wait return -EAGAIN, and that is how the loop learns there is more to
> do.  The window can be shrunk, but not removed; nearly all of what is
> left is inside io_uring_enter().
> 
> Here's how often that happens, measured on a Jetson Orin Nano running
> for-7.4/io_uring with PREEMPT_RT.  The event loops are AOS (a robotics
> framework).  Each one keeps one IORING_OP_FUTEX_WAIT armed per channel,
> on the channel's queue index, and re-arms it after every wake.  One
> channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40
> subscriber, then low-priority processes publishing and subscribing on
> the same channel as well, then interrupt and system load on top.
> bpftrace on fexit:io_futex_wait counted, per process, the waits armed
> (IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN
> instead of sleeping.  Two minutes each:
> 
>                              real-time subscriber    low-priority procs
>                  messages/s  arms/s  -EAGAIN  rate   arms/s  -EAGAIN  rate
>   pair only           2,000   2,000        0    0%        -        -     -
>   + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
>   + load             12,200   9,830    2,797   28%    4,120      377    9%
> 
> (messages/s is everything published on the channel, arms/s and
> -EAGAIN are per second.  A subscriber arms fewer times than messages
> arrive because one wake can cover several messages.  Under load the
> low-priority processes arm less often because the load takes their CPUs.)
> 
> In all, 2,520 and 3,170 arms a second failed in the last two runs.
> bpftrace also read the futex word as each failed arm returned.  Of the
> 683,303 failures across both runs, 88% found the word exactly one
> message past the value the wait was armed with.  So the arm raced a
> single publish, not a backlog.  Load makes it more likely because the
> stretch between that load and the kernel's check gets longer.
> 
> Timed from syscall entry to io_futex_wait() returning, 74% of the
> real-time subscriber's failures came within 6 us.  For the low-priority
> processes, which can be preempted in there, a third took more than
> 40 us.
> 
> Without the patch, each of those failures is a kfree() and then a
> kmalloc() on the next arm: about 3,000 a second here, from a slab cache
> shared system-wide, and more the busier the system gets.  On PREEMPT_RT
> the slab slow path's list_lock is a sleeping lock, which is where this
> started for us.  The patch doesn't change the armed path at all; it
> only stops the cache from draining.
> 
> I can send a v2 with these numbers and this explanation in the commit
> message, in place of the synthetic test.
> 
> What I'm after is a futex wait an event loop can reliably re-arm without
> latency spikes.  Is there a better way to avoid the allocation here?  If
> you'd rather not have every failure refill the cache, I can limit it to
> -EAGAIN, the one failure an event loop hits in normal operation, and
> keep the kfree() for the rest.

That's great info, and I agree it differs from the usual forced
error path. Futexes are inherently racy. Thanks, I'll get this
queued up.

-- 
Jens Axboe