Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Tue, 8 Sep 2026, Tim Menninger wrote:
> Strict makes it worse. Across five runs, the three all-local CQ
> placements ran at 38, 36, and 39 GB/s, while the two split placements
> both ran at 46 GB/s.
>
>     cpu:         37 GBps
>     smt:         43 GBps
>     cache:       47 GBps
>     cache_shard: 47 GBps
>     numa:        47 GBps

Thanks, that settles the scope question. Anything at LLC size or
larger is OK on your box, finer is worse, and the pool-lock cost
the series was after does not show up there at all. I'll drop the
patches that change the WQ affinity, for now.


> The lock profile is dominated by __slab_free in both cases.
> ...
>     __slab_free:
>         2.07M contentions
>         3.83 minutes aggregate wait
>         111 us average wait

That is on the order of twenty CPUs on node 0 doing nothing but
spinning on one lock, in every configuration you measured. It is a
much bigger target than the workqueue scope.

The lock is the SLUB per-node list_lock for the nfs_page cache.
Since 7.3-rc1, every ordinary kmem_cache gets per-CPU sheaves by
default (for nfs_page, 60 objects per sheaf), and a plain free
lands in the local sheaf without touching list_lock. It reaches
__slab_free only when a CPU's sheaves are full and the per-node
barn, which holds at most ten full sheaves, is full too. Then the
whole sheaf gets flushed to the slabs under list_lock.

Your workload appears to defeat that optimization: nfs_page
objects are allocated on the application CPUs across both nodes and
freed in bursts by rpciod on the CQ CPUs. The freeing CPUs never
allocate, so their sheaves stay full, and a 600-object barn is
microseconds of buffering at your free rate. If that is what is
going on, it is an allocator scaling issue rather than an NFS one,
and the right place to take it is linux-mm and Vlastimil.

Before that, I'd like to confirm this analysis. Could you:

1. Check that sheaves are actually on for that cache:

       cat /sys/kernel/slab/nfs_page/sheaf_capacity

   A zero means your kernel predates sheaves or the cache has debug
   flags, and the contention is the older remote-free slab
   transition path. That is a different report.

2. Build with CONFIG_SLUB_STATS=y and sample these before and after
   a 10-second all-local window:

       cd /sys/kernel/slab/nfs_page
       grep . free_fastpath free_slowpath sheaf_flush barn_put \
           barn_put_fail barn_get barn_get_fail alloc_slowpath

   If barn_put_fail and sheaf_flush track the contention count, the
   barn limit is the bottleneck. If free_slowpath dominates and the
   sheaf counters are quiet, the frees are bypassing sheaves
   entirely and my analysis is wrong.

3. Run perf lock contention with -l on the same window so the lock
   shows up by address, and note whether the callers under
   list_lock are sheaf_flush_unused / __kmem_cache_free_bulk or
   direct __slab_free from kmem_cache_free.

4. Tell me the I/O pattern: direct reads, I assume, but the I/O
   size, number of threads, and how they are spread across the two
   nodes. I would like to attempt to reproduce the
   alloc-here-free-there imbalance rather than speculate.

I don't have a NUMA system in my lab, so I cannot recreate the
per-node barn directly. What I do have is a 24-core/48-thread Ryzen,
which is a single node but has several CCDs. The per-CPU part of the
imbalance exists without NUMA. If barn_put_fail climbs there too, I
would have a reproducer that does not need specialist hardware.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)