Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Sagi Grimberg]

Hey Saravanan,

On 20/08/2026 11:36, Saravanan D wrote:
> nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time,
> before any I/O exists, as the least loaded CPU in the queue's blk-mq
> map group, and all socket work then runs there for the connection's
> lifetime.

That is true. It effectively takes a sane heuristics given that there are
multiple factors it is not aware of.

>   On hosts that partition CPUs between pinned workloads a map
> group can straddle a partition boundary, so the pick can land one
> workload's socket processing on CPUs owned by another. On a 384 cpu
> multi tenant host with one VM driving ~1.1 GB/s of writes, 9% of
> nvme_tcp_io_work executions ran outside the submitting VM's cpuset,
> all on io_cpus of boundary straddling map groups, observed by the
> neighbor as steal time it did not cause.

Can you share a concrete example?
What is the output of:
1. lscpu
2. grep -r ''  /sys/block/nvmeXnY/mq/*/cpu_list

Also, what would be the VMs cpusets on such a server?

In an ideal setting, the nvme controller would support $NR_CPUS worth of
io queues, and both nvme_tcp_set_queue_io_cpu() and blk_mq_map_queues() 
would
map cpu-queue with perfect alignment.

FWIW, I don't think this is an nvme-tcp specific problem, other than the 
fact that TCP
is more CPU consuming than other transports (doing stuff that is not 
offloaded by
the HW...)

> Adopt the submitting CPU as io_cpu for every command except the
> fabrics Connect. The submitter is a member of the map group by
> construction, and the nvme_tcp_cpu_queues accounting moves with each
> adoption. Connect is the only command on an I/O queue that does not
> represent the data path, since it is injected on an arbitrary CPU by
> blk_mq_alloc_request_hctx(), so it is skipped and the first real read
> or write decides. User passthrough is submitted from a real task on
> the submitting CPU and adopts like any other command.
>
> Queues outlive the workloads that submit through them, so adoption
> re-arms after 30 seconds of queue quiet. An idle queue is reclaimed by
> its next submitter, while a busy queue keeps a stable io_cpu and
> cannot ping pong between two live submitters. Concurrent writers on
> different CPUs serialize on a cmpxchg on io_cpu.
>
> The behavior is opt in per controller via the io_cpu_adopt fabrics
> option at connect time. wq_unbound takes precedence when set.

I think that for controllers with low queue count this approach would not
work very well. Plus the cpu<->queue mapping by the block layer is not
guaranteed to align with how workloads/VMs are scheduled on the host...

To me the problem statement sounds to me like you are looking to get a 
"private"
controller (set of IO queues) for each VM. Wouldn't a better approach 
would be to pass
a "private controller" connect-param which would for TCP mean that the 
driver would
create a dedicated UNBOUND workqueue which allows setting its cpumask?

Something like: nvme connect -t tcp -a <traddr> --queues_scope=private ?
Then you'd set its IO queues to something like:
echo [cpumask] > /sys/devices/virtual/workqueue/nvmeX-unb-wq/cpumask

Even explicit cpu-groups would be a better approach IMO. This would make
nvme-tcp to override blk_mq_map_queues() and adhere to the passed groups and
try to divide queues between them.

Something like the below example:
nvme connect -t tcp -a <traddr> --cpu-groups=0-63,64-127,128-191,192-255

In this case, nvme-tcp would try to both assign queue->io_cpu within these
ranges, and also assign the cpu<->mapping according to these groups.