Re: [PATCH 0/4] io_uring: avoid req->flags store-to-load forwarding failures

[Jens Axboe]

On Fri, 13 Nov 2026 20:16:17 -0700, Caleb Sander Mateos wrote:
> clang likes to set/clear req->flags bits using the narrowest possible or/and
> instructions (e.g. an 8-bit or if all flag bits are in the same byte). This
> causes store-to-load forwarding failures on subsequent wider req->flags loads.
> 
> For example, clang compiles the req->flags |= REQ_F_ASYNC_DATA in
> io_uring_cmd_prep() -> io_uring_alloc_cached_async_data() on x86_64 into an
> orb $0x40, 0x4a(%rdi) instruction, since REQ_F_ASYNC_DATA = 0x40 << 16 and
> offsetof(struct io_kiocb, flags) = 0x48. The subsequent 64-bit req->flags load
> for unlikely(req->flags & (IO_REQ_LINK_FLAGS | REQ_F_FORCE_ASYNC | REQ_F_FAIL))
> in io_submit_sqe() is extremely hot because it can't be served from the store
> buffer and must wait for the REQ_F_ASYNC_DATA set to hit the L1 cache. On a
> thread issuing ublk and NVMe passthrough commands, that one req->flags load
> consumes around 1% of all CPU cycles.
> 
> [...]

Applied, thanks!

[1/4] io_uring: return io_req_flags_t from io_slot_flags()
      commit: 4d81c9390d0a326f2beadea2503d3e3929c739e1
[2/4] io_uring: add multishot_flags constant in io_wq_submit_work()
      commit: 91801f8eab5e5963befb1358efa6832e9aa78499
[3/4] io_uring: add io_req_{set,clear,switch}_flags()
      commit: 3e5dd5a145214bd8996744e8a828ceb249570301
[4/4] io_uring: use io_req_{set,clear,switch}_flags()
      commit: 33e97a6c608064795dc5e36643a9cc66d98bfa99

Best regards,
-- 
Jens Axboe