Re: [PATCH v15 8/8] docs: add io_queue flag to isolcpus

[Aaron Tomlin]

On Fri, Aug 21, 2026 at 05:16:43PM +0200, Sebastian Andrzej Siewior wrote:
> Hi Aaron,

Hi Sebastian,

As always, thank you for the detailed feedback and insights.

> Is there a difference if this happens before or after allocation? I
> *think* if it is made post-allocation then the driver expects a certain
> amount of interrupts which need to be there. But otherwise…
> 
> There is no way to limit the amount of queues as far as I am aware which
> makes the isolcpus=managed_irq useless as long as there are more queues
> than CPUs in the system.
> 
> Is there a benefit of having 4 queues which are handled by 2 CPUs or
> would it make more sense to use only 2 queues and not "overcommit"?

Yes, this is the exact issue. In multiqueue drivers using PCI_IRQ_AFFINITY
(e.g., see function nvme_setup_irqs() in drivers/nvme/host/pci.c or
vp_request_msix_vectors() in drivers/virtio/virtio_pci_common.c), the
driver requests a vector range [minvec, maxvec] based on available CPUs,
and function irq_calc_affinity_vectors() in kernel/irq/affinity.c
determines the allocated vector count returned to the driver.

There is no benefit to allocating 4 hardware queues across 2 housekeeping
CPUs:

    1.  Allocating 4 queues on 2 housekeeping cores wastes MSI-X vector
        slots on architectures where per-CPU vector space is constrained
        (e.g., x86), whilst consuming extra memory for unneeded ring
        buffers and queue contexts.

    2.  When 4 queues are allocated for 4 CPUs, group_cpus_evenly() assigns
        a single CPU to each queue (Q0 -> {0}, Q1 -> {1}, Q2 -> {2}, Q3 ->
        {3}). For isolated CPUs 2 and 3, their masks share no intersection
        with housekeeping CPUs {0, 1}. Post-allocation managed_irq in
        irq_do_set_affinity() evaluates:

            cpumask_and(tmp_mask, mask, hk_mask);
            if (!cpumask_intersects(tmp_mask, cpu_online_mask))
                prog_mask = mask;

        Because tmp_mask is empty, it falls back to the isolated CPU,
        rendering isolation completely ineffective whenever queues >= CPUs.

Restricting vector allocation pre-allocation to the housekeeping count
(e.g., queues = 2) cleanly resolves this. It prevents MSI-X vector
exhaustion, avoids overcommit, and guarantees that no hardware interrupt
vector is ever created for an isolated CPU.


> The problem I am having is to figure out where the managed_irq makes
> sense since you need less queues than CPUs and this is hardware
> dependent. Say you replace your NVME after the old one broke and boom,
> now are all CPUs utilised.

Precisely. Modern storage drivers calculate queues based on possible CPUs
by default:

 - File: drivers/nvme/host/pci.c:

    static void nvme_calc_irq_sets(struct irq_affinity *affd, unsigned int nrirqs)
    {
        ...

        dev->io_queues[HCTX_TYPE_DEFAULT] = nrirqs - nr_read_queues;
        affd->set_size[HCTX_TYPE_DEFAULT] = nrirqs - nr_read_queues;

        ...
    }

Typically, modern devices allocate 1 queue per CPU (queues == CPUs),
single-CPU affinity masks are generated, making post-allocation managed_irq
completely ineffective. Relying on hardware queue scarcity for CPU
isolation is fragile and unpredictable.

> 
> So if managed_irqs would also act pre-allocation then the only
> difference would be "best effort" vs "mandatory".
> There might be the details around what happens if the CPU goes offline
> and can the CPU go offline. I think the block layers drains the I/O
> queues and the actual driver does nothing. In the CPUs > queues case the
> queues (or a few) are mapped to more than one CPU so if one of the CPUs
> goes offline, that IRQ is simply moved to another CPU within its mask
> (which could go to the initially isolated CPU if my memory serves me
> well).

Yes, and this highlights a key behavioural difference:

    1.  Isolated CPUs remain part of the descriptor mask. When a
        housekeeping CPU in that mask goes offline, kernel/irq/cpuhotplug.c
        migrates the managed interrupt onto whatever CPUs remain in the
        mask, including the isolated CPU.

    2.  Under strict pre-allocation (i.e., group_mask_cpus_evenly()),
        isolated cores are excluded when creating the affinity masks.
        See irq_create_affinity_masks():

            if (hk_enabled)
                mask = hk_mask;
            else
                mask = cpu_possible_mask;

            result = group_mask_cpus_evenly(this_vecs, mask, &nr_masks);

Because isolated CPUs are never part of the mask, hotplug migration onto an
isolated CPU is structurally impossible.

Regarding queue draining, blk-mq does not drain hardware queues when a CPU
goes offline; it simply unmaps the offline CPU's software context (ctx). If
all housekeeping cores serving a shared hctx were offlined, an active
isolated core submitting I/O to that hctx would suffer an indefinite stall.

This is why blk_mq_hctx_can_offline_hk_cpu() was added in patch 6/8.
See blk_mq_hctx_notify_offline():

        if (housekeeping_enabled(HK_TYPE_IO_QUEUE)) {
            if (!blk_mq_hctx_can_offline_hk_cpu(hctx, cpu))
                return -EINVAL;
        }

It explicitly prevents an administrator from offlining the final
housekeeping core serving an online isolated core during runtime
(while permitting suspend via cpuhp_tasks_frozen).

> I am not aware that networking is using this. Judging by
> pci_alloc_irq_vectors_affinity() this is mostly scsi and there is one
> networking driver (which makes a bit curious how CPU hotplug is
> handled).  Anyway, there is some effort on the networking side to
> consolidate this:
> https://lore.kernel.org/20260819-flo-net-7-2-make-stmmac-default-affinity-aware-v1-0-3f79a99cadaf@xxxxxxxxxxx

Indeed. Searching for PCI_IRQ_AFFINITY shows it is used almost exclusively
by storage multiqueue drivers (i.e., NVMe, SCSI, virtio-pci) with
drivers/net/ethernet/wangxun/libwx/wx_lib.c being the solitary networking
user.

Network multiqueue operates on unmanaged IRQs, using XPS, RPS/RFS, RSS
indirection tables, and ethtool channel steering, where user space can
adjust affinity via irqbalance or /proc/irq/. Storage multiqueue, by
contrast, relies on immutable kernel-managed IRQs tied to
submission/completion ring pairs, which is why pre-allocation queue capping
is specifically required for blk-mq.

> I'm asking for breaking the current option because it makes no sense and
> so far nobody responded by saying yes, or no that is stupid.
> 
> > How about an opt-in flag:
> > 
> >   - isolcpus=managed_irq
> >         Keeps best-effort post-allocation IRQ steering without capping
> >         hardware queues
> > 
> >   - isolcpus=managed_irq_strict (or managed_irq:strict)
> >         Enables strict pre-allocation queue capping (i.e.,
> >         irq_calc_affinity_vectors() and blk_mq queue limits)
> > 
> > If the preference is to avoid adding a new top-level isolcpus= flag, I am
> > more than happy to refactor io_queue into a strict sub-parameter under
> > managed_irq (e.g., isolcpus=managed_irq:strict). This probably makes more
> > sense.
> 
> Not sure what Frederick's thinks here but I would aim for
> managed_irq_strict if we can't change the current behaviour.
> It should be also documented if both can be used (or are invalid) and
> what to expect if they are mixed.

I completely agree that fixing/or breaking the legacy behaviour makes the most
technical sense given that post-allocation steering is broken on modern
hardware. However, to eliminate regression risk for existing users,
introducing isolcpus=managed_irq_strict (and dropping io_queue) provides an
explicit opt-in.

I will Cc Frederic Weisbecker on the next iteration to establish if
upgrading managed_irq directly or adopting managed_irq_strict is preferred.

Regarding precedence, if both managed_irq and managed_irq_strict are
supplied, managed_irq_strict will take precedence since it is a strict
superset (enforcing pre-allocation queue capping and mask exclusion). This
will be documented in Documentation/admin-guide/kernel-parameters.txt and
Documentation/core-api/irq/managed_irq.rst

> I *think* ':' will be an invalid character and ignored so maybe the '_'
> instead.

Yes; according to housekeeping_isolcpus_setup() in
kernel/sched/isolation.c:

        for (par = str, len = 0; *str && *str != ','; str++, len++) {
            if (!isalpha(*str) && *str != '_')
                illegal = true;
        }

        if (illegal) {
            pr_warn("isolcpus: Invalid flag %.*s\n", len, par);
            return 0;
        }

Any character other than [a-zA-Z_] triggers illegal = true, and aborts
isolcpus parsing. Therefore, managed_irq_strict will be used in the next
iteration.


Kind regards,
-- 
Aaron Tomlin