Re: [PATCH RFC 6/8] SUNRPC: Reduce rpciod workqueue contention

[Chuck Lever]

On Thu, 3 Sep 2026, Tim Menninger wrote:
> The v2 series behaves similarly insofar as some runs see ~15 GB/s and
> other runs see ~45 GB/s, but the low-throughput state no longer appears
> to be directly controlled by the rpciod affinity scope.

Based on your follow-up, I have a theory about the bimodal performance
behavior:

> The substantial completion traffic in these captures is on
> 0000:2a:00.0. Its mlx5 completion IRQs are individually affinitized to
> CPUs. For example:
>
>     mlx5_comp0   -> CPU 0
>     mlx5_comp1   -> CPU 1
>     ...
>     mlx5_comp23  -> CPU 23
>     mlx5_comp24  -> CPU 48
>     ...
>     mlx5_comp47  -> CPU 71
>     mlx5_comp48  -> CPU 24
>     ...
>     mlx5_comp62  -> CPU 38

That is the mlx5 default spread. It fills the NIC's local NUMA node
first, both SMT threads, and overflows the remaining vectors onto the
other socket. If I read the numbering right, CPUs 0-23 and 48-71 are
the two threads of the node that 0000:2a:00.0 hangs off, and comp48
through comp62 land on the remote socket. Please confirm with:

    cat /sys/bus/pci/devices/0000:2a:00.0/numa_node
    lscpu | grep 'NUMA node'

rpcrdma.ko allocates a send and a receive completion queue for each
transport, and the RDMA core assigns the completion vector from a
host-global round-robin counter. The completion vector is selected at
transport connect time and is fixed for the life of the connection.

Completions on a remote vector run the CQ handler, the RPC reply
processing, and the rpciod wakeup all on the wrong socket. My
suggestion is that you should pin the NIC's interrupt vectors to
CPUs on the NIC's local node so that the round-robin cannot select
a remote one. Something like the following, adjusted once you have
confirmed the local CPU list:

    for irq in $(awk '/mlx5_comp(4[89]|5[0-9]|6[0-2])@pci:0000:2a:00.0/ \
                      { sub(":", "", $1); print $1 }' /proc/interrupts); do
        echo 0-23,48-71 > /proc/irq/$irq/smp_affinity_list
    done

Keep irqbalance disabled, since it will undo the interrupt steering.


-- 
Chuck Lever