[RFC PATCH 2/3] io_uring/rsrc: bulk refill the node cache on allocation miss

[Uzair Beg]

A first fill of a sparse fixed file table takes one allocator round
trip per install, since the per-ring node cache starts empty and
nothing is freed back during the fill. At 4,096 slots that is 4,096
calls into the slab allocator.

When the cache has a dedicated kmem_cache, refill it in batches on a
miss: allocate up to IO_ALLOC_CACHE_REFILL objects with
kmem_cache_alloc_bulk(), return one and stash the remainder in the
cache. A 4,096-slot first fill then enters the allocator roughly once
per batch instead of once per object. This mirrors the existing bulk
allocation of requests from req_cachep.

kmem_cache_alloc_bulk() may return fewer objects than requested,
including zero; both cases are handled. Stashed objects have their
init_clear region zeroed and are poisoned like any other cached
entry, and the cache never grows past max_cached. Callers without a
dedicated slab are unchanged.

Bare-metal measurement shows this is neutral on the reported workload:
the per-object cost is in the SLUB allocation path itself, not in the
number of allocator entries. It is kept because the following patch
relies on the same bulk machinery.

Reported-by: Chengfeng Lin <lin2530632123@xxxxxxxxx>
Closes: https://lore.kernel.org/io-uring/CANGjgdmt0FQ=offsdfn+wEaDxbOFoAa6bi92X_vEo4S6aCZ56A@xxxxxxxxxxxxxx/
Tested-by: Chengfeng Lin <lin2530632123@xxxxxxxxx>
Signed-off-by: Uzair Beg <uzairbeg11@xxxxxxxxx>
---
 io_uring/alloc_cache.c | 29 ++++++++++++++++++++++++++---
 io_uring/alloc_cache.h |  1 +
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/io_uring/alloc_cache.c b/io_uring/alloc_cache.c
index a44b82a80f1..cba0e6c5d66 100644
--- a/io_uring/alloc_cache.c
+++ b/io_uring/alloc_cache.c
@@ -42,10 +42,33 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp)
 {
 	void *obj;
 
-	if (cache->slab)
-		obj = kmem_cache_alloc(cache->slab, gfp);
-	else
+	if (cache->slab) {
+		unsigned int room = cache->max_cached - cache->nr_cached;
+		void **slot = &cache->entries[cache->nr_cached];
+		unsigned int batch, got, i;
+
+		if (unlikely(!room))
+			return kmem_cache_alloc(cache->slab, gfp);
+
+		batch = min_t(unsigned int, IO_ALLOC_CACHE_REFILL, room);
+		got = kmem_cache_alloc_bulk(cache->slab, gfp, batch, slot);
+		if (unlikely(!got))
+			return NULL;
+
+		/* return one object, stash the rest in the cache */
+		obj = slot[got - 1];
+		for (i = 0; i < got - 1; i++) {
+			if (cache->init_clear)
+				memset(slot[i], 0, cache->init_clear);
+			if (unlikely(!kasan_mempool_poison_object(slot[i])))
+				break;
+			cache->nr_cached++;
+		}
+		for (; i < got - 1; i++)
+			kmem_cache_free(cache->slab, slot[i]);
+	} else {
 		obj = kmalloc(cache->elem_size, gfp);
+	}
 	if (obj && cache->init_clear)
 		memset(obj, 0, cache->init_clear);
 	return obj;
diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h
index b288bfccc91..82d552c7517 100644
--- a/io_uring/alloc_cache.h
+++ b/io_uring/alloc_cache.h
@@ -7,6 +7,7 @@
  * Don't allow the cache to grow beyond this size.
  */
 #define IO_ALLOC_CACHE_MAX	128
+#define IO_ALLOC_CACHE_REFILL	32
 
 void io_alloc_cache_free(struct io_alloc_cache *cache,
 			 void (*free)(const void *));
-- 
2.43.0