Re: [PATCH v2] nvme-tcp: pin io_cpu to submitter cpu

[Saravanan D]

On Mon, 31 Aug 2026 01:30:08 +0300 Sagi Grimberg <sagi@xxxxxxxxxxx> wrote:
> I just think that the adoption approach has downsides in many other
> workloads. I wouldn't categorize your use-case as the common use-case.
> Not to say that it shouldn't be solved.

Adoption in v2 is opt in per controller through the io_cpu_adopt
connect option, so every workload that does not enable it
keeps today's behavior unchanged. The downsides would only be borne by
deployments that choose it. I concur with the broader point. Adoption
makes the driver itself decide where socket work belongs, through a
fixed rule that cannot fit every deployment, while a writable io_cpu
is very flexible and leaves the decision to the control plane.

> I agree with you that queue io_cpu selection at connection time is too
> limiting. Perhaps we can have each nvme-tcp queue expose a sysfs entry
> to set the io_cpu?
>
> Something like:
> echo $CPU > /sys/class/nvme/nvme0/tcp_queues/$QUEUE/io_cpu
>
> I am just trying to think of a way that will allow some control plane
> to set it exactly as it wants, without the guessing part...

The per queue sysfs io_cpu works for us, and in one respect it works
better than a connect option because a sysfs attribute can be set after
the fact on any existing connection. The control plane already knows each
VM's cpuset and can read the queue to cpu grouping from mq/*/cpu_list,
so it can nail the socket work placement.

Two questions before I spin the next version around this.

1. On reconnect the queues come back with the connect time io_cpu, so
the control plane has to notice and reapply its placement. Is relying
on the existing nvme udev events for controller state the intended
signal, or would you want a dedicated notification ?

2. What should the write path enforce ? I would validate the cpu is
online, rebalance the nvme_tcp_cpu_queues accounting, and let the new
value take effect from the next queued io_work. I intend to restrict the
value to the queue's map group.

If this direction sounds right I will drop the adoption heuristic and
send the next version v3 exposing each queue's io_cpu as a writeable
sysfs attribute.

Thanks,
Saravanan D.