Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Sat, 12 Sep 2026, Tim Menninger wrote:
> 10-second deltas:
>
>     free_fastpath:    +49,161,373
>     free_slowpath:    +70,391,872
>     sheaf_flush:       +1,582,440
>     barn_put:            +673,877
>     barn_put_fail:        +26,374
>     barn_get:            +673,860
>     barn_get_fail:     +1,199,576
>     alloc_slowpath:             0

Thanks. No worries on the delayed response, IMO there is something
here worth digging out.

Your results confirm the barn-limit theory was wrong. The frees are
bypassing the sheaves.

can_free_to_pcs() refuses any object whose slab is on a different
NUMA node than the freeing CPU. A remote free goes directly to
__slab_free and never refills a barn. With elbencho unpinned and
every CQ on node 0, rpciod on node 0 frees every nfs_page object
that a node 1 thread allocated. That is the 59% slowpath share.
Node 1's barn stays empty, so its allocators refill from partial
slabs via __refill_objects_node and get_partial_node_bulk, the
other two callers under the contended lock. The contention count
fits as well: __slab_free takes list_lock only on a full-to-partial
or partial-to-empty transition, and the bulk refills keep slabs
full.

Still unknown is whether this lock bounds throughput or only
absorbs idle CPU time. __slab_free led the lock profile in both
placements, and the split placement still ran at 46 GB/s.

To separate the two, keep the all-local CQ placement and pin the
workload to node 0:

    numactl --cpunodebind=0 --membind=0 elbencho <same args>

Sample the same nfs_page counters and perf lock -l over the same
10-second window. free_slowpath should drop toward zero.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)