Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Wed, Sep 2, 2026, at 4:40 PM, Tim Menninger wrote:
> I am seeing a significant throughput regression from the rpciod SMT
> affinity change on a high-throughput NFS/RDMA workload.

Thanks for the report!

I'd like to learn a bit more about the setup before drawing
any firm conclusions.

1. Does v2 of the series behave the same way? v2 dropped the
   patch that ran the first RPC states in the submitter's
   context, so the rpciod traffic pattern differs from v1.
   The same smt -> cache_shard -> smt toggle on v2 would tell
   us whether that matters.
  
2. How are the RDMA device's completion interrupts placed?
   Please share, on a bad run:
   - /proc/interrupts lines for the device's completion vectors
   - the smp_affinity_list for each of those IRQs
   - whether irqbalance is running, and whether you've pinned
     the IRQs by hand

3. Where is CPU time going on a bad run versus a good one?
   A short capture of each would help:
     perf record -a -g -- sleep 10
     perf report --sort comm,cpu --stdio | head -80
   In particular I'm interested in which CPUs the kworker
   threads for rpciod and the ib-comp-wq threads run on under
   each scope.

4. tools/workqueue/wq_monitor.py rpciod, sampled for a few
   seconds under each scope, would show whether the pools are
   evenly loaded.

5. The workload itself: thread count, I/O size and direction,
   number of mounts and RDMA connections, and the mount
   options (nconnect in particular).

6. The base kernel the series was applied to, and the RDMA
   device and driver.
  
If you have a way to capture a bad run reliably shortly after
boot, a "before and after" of items 2 through 4 across a single
scope toggle would be the most direct evidence.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)