Re: [PATCH v5 2/2] nvme-tcp: allow setting per-queue io_cpu through sysfs

[Hannes Reinecke]

On 10/20/26 5:28 PM, Daniel Wagner wrote:
> On Sat, Oct 17, 2026 at 06:00:59PM -0700, Saravanan D wrote:
> > nvme_tcp_set_queue_io_cpu() picks each queue's io_cpu at connect time
> > as the least loaded CPU in the queue's blk-mq map group, and all socket
> > work runs there for the connection's lifetime. This decision falls
> > short when the host partitions its CPUs after connect time. On a 384
> > cpu multi tenant host with 128 queue controllers, blk-mq folds three
> > CPUs into every map group, some groups straddle two tenants' cpusets,
> > and 9% of nvme_tcp_io_work executions ran outside the submitting VM's
> > cpuset, seen by the neighbor as steal time.
> >
> > Expose the io_cpu in each I/O queue's directory
> >
> >    /sys/class/nvme/nvmeX/queues/<qid>/io_cpu
> >
> > so a control plane that owns CPU placement can set it directly instead
> > of relying on the driver's heuristic. The attribute accepts a decimal
> > cpu number, -1 or "unbound". A user assignment, unbound included, is
> > marked in the queue flags, persists across reconnects and is never
> > reselected by the driver. The read-only managed attribute reports 1
> > while io_cpu is driver managed and 0 once the user assigned it. The
> > accounting of nvme_tcp_cpu_queues applies regardless of who set the
> > io_cpu.
> >
> > A queue's directory is added when the queue is allocated and removed
> > when it is freed, before the queue is torn down, so a sysfs store
> > never runs on a freed queue. The queues directory is created before
> > the I/O queues are set up in parallel. All io_cpu writes, from sysfs
> > and from the driver, serialize with the accounting under
> > nvme_tcp_cpu_queues_lock, while the I/O paths read io_cpu locklessly.
> > The queue stop now clears NVME_TCP_Q_IO_CPU_SET under that lock, since
> > a sysfs store may move io_cpu in between.
> >
> > nvme_tcp_free_queue() now clears NVME_TCP_Q_IO_CPU_SET and drops its
> > count, because a queue can still be counted when it is freed. A sysfs
> > store may have run after the queue stop, or the connect may have
> > failed, and the setup error path frees such a queue without calling
> > nvme_tcp_stop_queue(), where the count is normally dropped.
>   
> This looks good, just trying to wrap my head around with isolcpu (yeah,
> doing a bit of work in that area makes me a paranoid).
>
> I think it make sense to allow that the user can change queue CPU
> pinning during runtime even with isolcpu enabled. And then the question
> is should this be connected to cgroups?

Not sure if cgroups is the right concept here; cgroups are user/
application centric whereas nvme-tcp is pretty much queue/cpu centric.
IE after all the cgroup calculations are done.
Don't think cgroups are a good fit here.

Cheers,

Hannes
--
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@xxxxxxx                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich