Re: [PATCH v3] nvme-tcp: allow setting per-queue io_cpu through sysfs

[Nilay Shroff]

On 9/10/26 2:44 AM, Saravanan D wrote:
> nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time
> as the least loaded CPU in the queue's blk-mq map group, and all socket
> work runs there for the connection's lifetime. This decision falls
> short when the host partitions its CPUs after connect time. On a 384
> cpu multi tenant host with 128 queue controllers, blk-mq folds three
> CPUs into every map group, some groups straddle two tenants' cpusets,
> and 9% of nvme_tcp_io_work executions ran outside the submitting VM's
> cpuset, seen by the neighbor as steal time.
>
> Expose each I/O queue's io_cpu as a writable sysfs attribute
>
>    /sys/class/nvme/nvmeX/tcp_queues/<qid>/io_cpu

I wonder if we should avoid introducing a transport-specific tcp_queues hierarchy
here. io_cpu describes CPU placement of an NVMe I/O queue rather than something
inherently specific to TCP, and other transports may have similar queue-level
attributes in the future. Could we instead introduce a generic "queues/<qid>/"
hierarchy under each NVMe controller, e.g.

     /sys/class/nvme/nvmeX/queues/<qid>/

and have transports expose queue-specific attributes there? The "queues/<qid>"
object could be created for every controller, while transports that need queue-
specific sysfs attributes can populate the corresponding <qid> objects. This
would give us a transport-independent ABI and avoid introducing separate
tcp_queues, rdma_queues, etc. hierarchies.

> so a control plane that owns CPU placement can set it directly instead
> of relying on the driver's heuristic. A written value persists across
> reconnects, marked by NVME_TCP_Q_IO_CPU_USER. Writing -1 clears the
> mark and re-runs the connect time selection. Reading returns the CPU,
> or -1 when the queue is unbound.
>
> The store, the connect time selection and queue stop serialize their
> accounting of nvme_tcp_cpu_queues under the queue lock.
Also, based on the current design, it seems possible to have some queues whose
io_cpu is driver-managed while other queues are explicitly managed by the
control plane. This gives useful flexibility for the tenant use case. However,
looking at the io_cpu sysfs attribute alone, it is not possible to determine
whether the current CPU was selected by the driver or explicitly assigned by
the control plane. So could we expose this state through an additional read-only
attribute alongside io_cpu, for example:

    /sys/class/nvme/nvmeX/queues/<qid>/io_cpu
    /sys/class/nvme/nvmeX/queues/<qid>/managed

where "managed" indicates who currently owns the CPU placement. For example, "1"
could indicate that io_cpu is driver-managed and "0" indicates it is managed by
the control plane or user. This would make the current ownership state observable
without making any wild guess.

Thanks,
--Nilay