Re: [PATCH v5 2/2] nvme-tcp: allow setting per-queue io_cpu through sysfs

[Saravanan D]

On 10/21/26 12:28 AM, Hannes Reinecke wrote:
[...]
> I think the 'managed' sysfs attribute needs to be per controller, as it
> would be extremely awkward to allow some queues to be managed from the
> admin and some queues automatically placed.

The mix is intentional, and it is the case this series is for. On a
host that partitions its CPUs between tenants, a queue only needs a
user assigned io_cpu when its blk-mq map group straddles two tenants'
cpusets. On our 384 cpu hosts with 128 queue controllers that is one
or two queues per controller per VM, and the driver's least loaded
selection already places the others well. With ownership per
controller, the control plane would have to take over every queue of
the controller to fix those one or two, and reimplement the least
loaded selection for all the others.

Per queue granularity is also where Sagi steered the series after the
earlier per controller connect option, since the CPU partitioning
changes after connect time and only some queues are affected [1].
Sagi, does the per queue model still match what you had in mind?

> And I think you need to check for interactions with the existing
> 'wq_unbound' module option; both claim to do similar things so we
> would need to clarify the details here.

They are independent. wq_unbound picks the workqueue type for all of
nvme-tcp at module load, and when it is set the driver never selects an
io_cpu, so every driver managed queue stays unbound. io_cpu places the
work of a single queue. When both are used, a written io_cpu selects the
worker pool for that cpu, and how closely the work follows it depends on
the affinity_scope of nvme_tcp_wq. To keep the work on exactly that cpu,
nvme_tcp_wq needs affinity_scope=cpu and affinity_strict=1, and the cpu
has to be in its cpumask. The ABI entry in this patch documents that,
and I can expand the text in v6 if it reads ambiguous.

> And should be we able to specify a cpu _set_ here?

For this series I would keep a single cpu. io_work is one work item
queued with queue_work_on(), which takes a single cpu, so honoring a
set would need a separate unbound workqueue per queue with its own
cpumask. A single cpu inside the owning tenant's cpuset covers the
case above, and a cpumask could be added later as a separate attribute
without changing io_cpu.

[1] https://lore.kernel.org/linux-nvme/13c160d0-0657-47ce-9a16-58d348267ab0@xxxxxxxxxxx/

Thanks,
Saravanan D.