Re: [PATCH] io_uring/futex: return a wait's data to the cache when it fails to arm

[Austin Schuh]

On Mon, Oct 19, 2026 at 5:54 PM Jens Axboe <axboe@xxxxxxxxx> wrote:
> Patches like this have been posted before, and my question is
> always "is this a common case?"? Reading this:
>
> > Notes:
> >     Based on for-7.4/io_uring (5b01d5955ac2).  Built for arm64 with
> >     PREEMPT_RT, W=1, no new warnings.  The numbers are from an equivalent
> >     change on 6.18.39-rt on a Jetson Orin Nano, counting kmem:kmalloc/kfree
> >     by call site with bpftrace while a test loops on IORING_OP_FUTEX_WAIT
> >     with a value the futex word doesn't hold.  Happy to turn that into
> >     a liburing test if it's useful.
>
> makes me very much think "nope, not at all", as it's a deliberately
> failing test case. And yes if you keep issuing failing waits,
> then it'll drain the cache. The cache is for the hot FAST path,
> which is not an error path.

Fair on the test case, it only showed the mechanism.  But for an event
loop, -EAGAIN from a futex wait isn't an error path.  Right before
io_uring_enter() re-arms the wait that just fired, the event loop loads
its futex word.  If the word has already changed, it handles the
new message and re-arms with the new value instead.  A message published
between that load and futex_wait_setup() checking the word makes the
wait return -EAGAIN, and that is how the loop learns there is more to
do.  The window can be shrunk, but not removed; nearly all of what is
left is inside io_uring_enter().

Here's how often that happens, measured on a Jetson Orin Nano running
for-7.4/io_uring with PREEMPT_RT.  The event loops are AOS (a robotics
framework).  Each one keeps one IORING_OP_FUTEX_WAIT armed per channel,
on the channel's queue index, and re-arms it after every wake.  One
channel: a SCHED_FIFO 40 publisher at 2 kHz with a SCHED_FIFO 40
subscriber, then low-priority processes publishing and subscribing on
the same channel as well, then interrupt and system load on top.
bpftrace on fexit:io_futex_wait counted, per process, the waits armed
(IORING_OP_FUTEX_WAIT issued) and how many of them returned -EAGAIN
instead of sleeping.  Two minutes each:

                             real-time subscriber    low-priority procs
                 messages/s  arms/s  -EAGAIN  rate   arms/s  -EAGAIN  rate
  pair only           2,000   2,000        0    0%        -        -     -
  + low-priority     12,200  11,020    1,292   12%   14,760    1,229    8%
  + load             12,200   9,830    2,797   28%    4,120      377    9%

(messages/s is everything published on the channel, arms/s and
-EAGAIN are per second.  A subscriber arms fewer times than messages
arrive because one wake can cover several messages.  Under load the
low-priority processes arm less often because the load takes their CPUs.)

In all, 2,520 and 3,170 arms a second failed in the last two runs.
bpftrace also read the futex word as each failed arm returned.  Of the
683,303 failures across both runs, 88% found the word exactly one
message past the value the wait was armed with.  So the arm raced a
single publish, not a backlog.  Load makes it more likely because the
stretch between that load and the kernel's check gets longer.

Timed from syscall entry to io_futex_wait() returning, 74% of the
real-time subscriber's failures came within 6 us.  For the low-priority
processes, which can be preempted in there, a third took more than
40 us.

Without the patch, each of those failures is a kfree() and then a
kmalloc() on the next arm: about 3,000 a second here, from a slab cache
shared system-wide, and more the busier the system gets.  On PREEMPT_RT
the slab slow path's list_lock is a sleeping lock, which is where this
started for us.  The patch doesn't change the armed path at all; it
only stops the cache from draining.

I can send a v2 with these numbers and this explanation in the commit
message, in place of the synthetic test.

What I'm after is a futex wait an event loop can reliably re-arm without
latency spikes.  Is there a better way to avoid the allocation here?  If
you'd rather not have every failure refill the cache, I can limit it to
-EAGAIN, the one failure an event loop hits in normal operation, and
keep the kfree() for the rest.

Thanks for the prompt reply!
  Austin