[PATCH] io_uring: fix CQ tail over-commit and CQE32 index corruption in io_cqe_cache_refill()

[Hui Peng]

io_cqe_cache_refill() has three issues when handling 32-byte CQEs on
IORING_SETUP_CQE32 and IORING_SETUP_CQE_MIXED rings:

1. When cqe32 is true and off + 1 == ctx->cq_entries,
   io_cqe_cache_refill() posts a 16-byte dummy IORING_CQE_F_SKIP CQE via
   io_fill_nop_cqe(ctx, off) even on pure IORING_SETUP_CQE32 rings
   (where IORING_SETUP_CQE_MIXED is not set). On a pure CQE32 ring,
   ctx->cq_entries is the number of 32-byte CQEs and rings->cqes is
   indexed by (off << 1), so writing a 16-byte skip CQE at
   &rings->cqes[off] corrupts the middle of the CQ ring and misaligns
   all subsequent CQEs. Restrict the wrap-around skip CQE to
   IORING_SETUP_CQE_MIXED rings.

2. On an IORING_SETUP_CQE_MIXED ring, when off + 1 == ctx->cq_entries
   and only 1 free CQ slot remains (ctx->cq_entries - io_cqring_queued()
   == 1), io_fill_nop_cqe(ctx, off) consumes that last slot and
   increments ctx->cached_cq_tail, after which free == 0 causes
   io_cqe_cache_refill() to return false without updating
   ctx->cqe_cached. Check that at least 2 free CQ slots exist before
   posting the dummy skip CQE.

3. On pure IORING_SETUP_CQE32 rings, len is in 32-byte CQE units prior
   to `len <<= 1`, so `len < (cqe32 + 1)` falsely requires 2 free
   32-byte CQEs instead of 1. Check `!len` before scaling `off` and
   `len` on IORING_SETUP_CQE32 rings, and in io_fill_cqe_aux() zero
   cqe->big_cqe[0..1] whenever IORING_SETUP_CQE32 is set on the ring.

Fixes: e26dca67fde1 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Assisted-by: LLM
Signed-off-by: Hui Peng <benquike@xxxxxxxxx>
---
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -733,7 +733,10 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow, bool cqe32)
 	 * Post dummy CQE if a 32b CQE is needed and there's only room for a
 	 * 16b CQE before the ring wraps.
 	 */
-	if (cqe32 && off + 1 == ctx->cq_entries) {
+	if (cqe32 && (ctx->flags & IORING_SETUP_CQE_MIXED) &&
+	    off + 1 == ctx->cq_entries) {
+		if (ctx->cq_entries - io_cqring_queued(ctx) < 2)
+			return false;
 		if (!io_fill_nop_cqe(ctx, off))
 			return false;
 		off = 0;
@@ -742,12 +745,13 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow, bool cqe32)
 	free = ctx->cq_entries - io_cqring_queued(ctx);
 	/* we need a contiguous range, limit based on the current array offset */
 	len = min(free, ctx->cq_entries - off);
-	if (len < (cqe32 + 1))
-		return false;
-
 	if (ctx->flags & IORING_SETUP_CQE32) {
+		if (!len)
+			return false;
 		off <<= 1;
 		len <<= 1;
+	} else if (len < (cqe32 + 1)) {
+		return false;
 	}
 
 	ctx->cqe_cached = &rings->cqes[off];
@@ -781,7 +785,7 @@ static bool io_fill_cqe_aux(struct io_ring_ctx *ctx, u64 user_data, s32 res,
 		WRITE_ONCE(cqe->res, res);
 		WRITE_ONCE(cqe->flags, cflags);
 
-		if (cqe32) {
+		if (cqe32 || (ctx->flags & IORING_SETUP_CQE32)) {
 			WRITE_ONCE(cqe->big_cqe[0], 0);
 			WRITE_ONCE(cqe->big_cqe[1], 0);
 		}
-- 
2.43.0