Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Tue, 8 Sep 2026, Tim Menninger wrote:

> Patches 1-2 alone do not reproduce it across 5 trials ...
>
> Patches 3-8 do reproduce it (as well as the full 1-8) ...

Thanks, that confirms which part of the series is responsible. The
scope change on rpciod is the cause, and the XID and recv_lock
changes are not involved.


>                         smt         cache_shard     smt
>     aggregate           15%         32%             19%
>     CPU 0-23,48-71      0-1%        6-9%            0-2%
>     CPU 24-47,72-95     28-32%      55-61%          35-40%

Hrm. Under smt, node 0 is saturated and node 1 is busier too, yet
throughput is lower than under cache_shard, where both nodes have
more idle time. The smt scope is burning CPU on something that is
not moving data. That is a different failure than the one I
expected (node 0 starved while node 1 sleeps), so the next step
is to find out what those cycles are doing.


> Would there be a downside to using WQ_AFFN_CACHE_SHARD here instead of
> WQ_AFFN_SMT?

cache_shard is the system default, and that is what the workqueue
used before this series. On the 12-core system where I developed
the series, cache_shard produces only two pools of six cores each,
and 30-40% of cycles went to the pool lock's spinlock slowpath,
accompanied with a measurable loss in throughput.

Your machine gets six pods of sixteen threads, which is a different
regime, and it is possible that cache_shard is simply the right
answer there. What I'm ultimately shooting for is something that
automatically configures the correct behavior. I thought smt would
be that configuration.

A few experiments, all with the full series applied and all CQs on
node 0. Each is a single sysfs write on rpciod except the last.

1. Toggle affinity_strict under smt:

       echo 1 > /sys/bus/workqueue/devices/rpciod/affinity_strict

   Strict pins each pool's kworkers to its SMT pair. If strict
   recovers throughput, the loss comes from non-strict workers
   being wake-affined or migrated onto the saturated node. If
   strict makes it worse, node 0's pools are starved and the fix
   is to let work spill to node 1. Either result cuts the
   hypothesis space in half, so if you have time for only one of
   these, this is the one.

2. Walk the scope ladder: cpu, smt, cache, cache_shard, numa, and
   report throughput for each. If cpu is as bad as smt, pool
   granularity itself is the problem. If cache already recovers,
   the threshold sits between 2-thread and 16-thread pods.

3. Profile the smt and cache_shard windows of one run, node 0 CPUs
   only, so the two captures differ in nothing but the scope:

       perf record -a -g -C 0-23,48-71 -- sleep 10
       perf lock contention -a -C 0-23,48-71 -- sleep 10
       perf stat -a -C 0-23,48-71 \
           -e context-switches,cpu-migrations,sched:sched_wakeup \
           -- sleep 10

   The two candidates I have in mind are a downstream lock, such as
   the transport's queue_lock or recv_lock, contended by 24 small
   pools running completions in parallel; or scheduler overhead
   from each pool waking its own kworkers. The %sys, %irq, and
   %soft columns from the same mpstat runs would help too, since
   idle alone does not say what the busy CPUs are doing.

4. With the even 17/15 CQ split and rpciod at cache_shard, a perf
   profile shows whether the pool-lock slowpath the series targets
   appears on your box at all. If it does not, cache_shard is
   correct for your system, and the series needs a way to express
   that rather than one hardcoded scope.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)