[PATCH 0/4] io_uring: avoid req->flags store-to-load forwarding failures

From: Caleb Sander Mateos

Date: Fri Nov 13 2026 - 22:17:04 EST


clang likes to set/clear req->flags bits using the narrowest possible or/and
instructions (e.g. an 8-bit or if all flag bits are in the same byte). This
causes store-to-load forwarding failures on subsequent wider req->flags loads.

For example, clang compiles the req->flags |= REQ_F_ASYNC_DATA in
io_uring_cmd_prep() -> io_uring_alloc_cached_async_data() on x86_64 into an
orb $0x40, 0x4a(%rdi) instruction, since REQ_F_ASYNC_DATA = 0x40 << 16 and
offsetof(struct io_kiocb, flags) = 0x48. The subsequent 64-bit req->flags load
for unlikely(req->flags & (IO_REQ_LINK_FLAGS | REQ_F_FORCE_ASYNC | REQ_F_FAIL))
in io_submit_sqe() is extremely hot because it can't be served from the store
buffer and must wait for the REQ_F_ASYNC_DATA set to hit the L1 cache. On a
thread issuing ublk and NVMe passthrough commands, that one req->flags load
consumes around 1% of all CPU cycles.

Introduce io_req_set_flags() and io_req_clear_flags() helpers to set req->flags
using volatile writes so clang won't emit a store to only part of the field.
The downside of volatile writes is that consecutive req->flags updates must
store to req->flags multiple times. Where possible, convert these updates to
pass multiple flags to io_req_set_flags(), io_req_clear_flags(), or an extra
helper io_req_switch_flags() which can do both. An extra store would still be
much cheaper than a store-to-load forwarding stall.

Alternatively, -mllvm -combiner-reduce-load-op-store-width=false could be used
to disable the clang "optimization". It seemed ugly to refer to such an internal
optimization pass, and I wasn't sure what other impacts this flag might have
(even if restricted to io_uring). But I'm happy to implement that approach if
it's preferred.

Caleb Sander Mateos (4):
io_uring: return io_req_flags_t from io_slot_flags()
io_uring: add multishot_flags constant in io_wq_submit_work()
io_uring: add io_req_{set,clear,switch}_flags()
io_uring: use io_req_{set,clear,switch}_flags()

include/linux/io_uring.h | 35 +++++++++++++++++++++++++-----
io_uring/advise.c | 4 ++--
io_uring/cancel.c | 2 +-
io_uring/dma-buf.h | 4 ++--
io_uring/filetable.h | 2 +-
io_uring/fs.c | 25 +++++++++-------------
io_uring/futex.c | 4 ++--
io_uring/io_uring.c | 41 +++++++++++++++++------------------
io_uring/io_uring.h | 8 +++----
io_uring/kbuf.c | 31 +++++++++++++++------------
io_uring/kbuf.h | 4 ++--
io_uring/msg_ring.c | 4 ++--
io_uring/net.c | 46 ++++++++++++++++++++--------------------
io_uring/nop.c | 2 +-
io_uring/openclose.c | 6 +++---
io_uring/poll.c | 20 +++++++++--------
io_uring/refs.h | 4 ++--
io_uring/rsrc.c | 10 ++++-----
io_uring/rw.c | 28 ++++++++++++------------
io_uring/splice.c | 4 ++--
io_uring/statx.c | 3 +--
io_uring/sync.c | 6 +++---
io_uring/timeout.c | 8 +++----
io_uring/truncate.c | 2 +-
io_uring/uring_cmd.c | 8 +++----
io_uring/xattr.c | 8 +++----
26 files changed, 170 insertions(+), 149 deletions(-)


base-commit: ddcc50dd41d331dc66e0e5793492c3fa09941ff9
--
2.56.1