Re: [PATCH v15 8/8] docs: add io_queue flag to isolcpus

[Sebastian Andrzej Siewior]

On 2026-08-05 12:22:59 [-0400], Aaron Tomlin wrote:
> Hi Sebastian,
Hi Aaron,

> Yes. Though the underlying mechanisms differ in a fundamental way between
> the two flags:
> 
> isolcpus=managed_irq (post-allocation, best-effort):
> 
>     managed_irq operates downstream at the IRQ affinity level (i.e.,
>     kernel/irq/manage.c and kernel/irq/cpuhotplug.c). It attempts to steer
>     the effective affinity of already-allocated IRQ vectors away from
>     isolated CPUs. However, when a multiqueue storage or networking driver
>     allocates one hardware queue per CPU, the affinity mask for an isolated
>     CPU's queue contains only that specific CPU. Because the intersection
>     between the queue's mask and the housekeeping CPUs is empty,
>     managed_irq falls back to assigning the interrupt to the isolated CPU.
>     It is therefore inherently "best effort" and cannot guarantee isolation
>     for 1-queue-per-CPU devices.
>
>
> isolcpus=io_queue (pre-allocation, strict constraint):
> 
>     io_queue operates upstream during device probe, MSI-X vector
>     calculation, and block-multiqueue topology setup
>     (irq_calc_affinity_vectors(), irq_create_affinity_masks(), and blk-mq).
>     Rather than attempting to steer vectors after allocation, it caps the
>     total number of hardware queues and MSI-X vectors to match the weight
>     of the online housekeeping CPUs. Isolated CPUs are strictly excluded
>     during initial vector mask generation (group_mask_cpus_evenly()). I/O
>     requests submitted from isolated CPUs are routed through shared
>     submission queues mapped onto housekeeping cores, ensuring completion
>     interrupts land exclusively on housekeeping CPUs.
> 
> In summary, while managed_irq influences where already-created interrupts
> land, io_queue restricts how many queues are created and how the hardware
> topology is constructed.
> 
> I hope this helps.

Is there a difference if this happens before or after allocation? I
*think* if it is made post-allocation then the driver expects a certain
amount of interrupts which need to be there. But otherwise…

There is no way to limit the amount of queues as far as I am aware which
makes the isolcpus=managed_irq useless as long as there are more queues
than CPUs in the system.

Is there a benefit of having 4 queues which are handled by 2 CPUs or
would it make more sense to use only 2 queues and not "overcommit"?

The problem I am having is to figure out where the managed_irq makes
sense since you need less queues than CPUs and this is hardware
dependent. Say you replace your NVME after the old one broke and boom,
now are all CPUs utilised.

So if managed_irqs would also act pre-allocation then the only
difference would be "best effort" vs "mandatory".
There might be the details around what happens if the CPU goes offline
and can the CPU go offline. I think the block layers drains the I/O
queues and the actual driver does nothing. In the CPUs > queues case the
queues (or a few) are mapped to more than one CPU so if one of the CPUs
goes offline, that IRQ is simply moved to another CPU within its mask
(which could go to the initially isolated CPU if my memory serves me
well).

I am not aware that networking is using this. Judging by
pci_alloc_irq_vectors_affinity() this is mostly scsi and there is one
networking driver (which makes a bit curious how CPU hotplug is
handled).  Anyway, there is some effort on the networking side to
consolidate this:
	https://lore.kernel.org/20260819-flo-net-7-2-make-stmmac-default-affinity-aware-v1-0-3f79a99cadaf@xxxxxxxxxxx
 
> > If so, can this be integrated?
> 
> I do not want managed_irq unconditionally enforced vector capping. Instead
> of creating a standalone HK_TYPE_IO_QUEUE housekeeping flag, the strict
> vector capping and mask restriction could be integrated under an opt-in
> flag?

I'm asking for breaking the current option because it makes no sense and
so far nobody responded by saying yes, or no that is stupid.

> How about an opt-in flag:
> 
>   - isolcpus=managed_irq
>         Keeps best-effort post-allocation IRQ steering without capping
>         hardware queues
> 
>   - isolcpus=managed_irq_strict (or managed_irq:strict)
>         Enables strict pre-allocation queue capping (i.e.,
>         irq_calc_affinity_vectors() and blk_mq queue limits)
> 
> If the preference is to avoid adding a new top-level isolcpus= flag, I am
> more than happy to refactor io_queue into a strict sub-parameter under
> managed_irq (e.g., isolcpus=managed_irq:strict). This probably makes more
> sense.

Not sure what Frederick's thinks here but I would aim for
managed_irq_strict if we can't change the current behaviour.
It should be also documented if both can be used (or are invalid) and
what to expect if they are mixed.

…
> > This is named io_queue but it does affect I/O in general unless the
> > driver uses managed_irqs. This may or may not be clear to the user.
> > So the io_queue might not be the perfect name given that affects alles
> > devices in the system. We do have just I/O devices using it right now.
> > 
> > I think this still deserves an explanation in
> > Documentation/core-api/irq/managed_irq.rst like how to use it, what is
> > the expected outcome and how does it interact with the managed_irq
> > argument. Does it make sense to use both, is it exclusive or, what would
> > be the use case for both arguments.
> 
> Regarding the name io_queue, you are correct that it specifically affects
> devices employing managed IRQs (i.e., NVMe, SCSI, and blk-mq devices).
> The name was chosen because its primary operational objective is to shield
> real-time and latency-sensitive isolated cores from multiqueue storage and
> block I/O completion interrupts. Non-managed IRQ devices continue to be
> governed by standard irqaffinity settings and irqbalance.
> 
> I would be happy to provide clear usage guidelines based on the above, if
> we can agree on the new proposal, irq_managed:strict?

I *think* ':' will be an invalid character and ignored so maybe the '_'
instead.

> Kind regards,

Sebastian