Re: [PATCH v16 0/9] blk: honor isolcpus configuration

[Aaron Tomlin]

On Mon, Sep 21, 2026 at 04:53:38PM +0300, Ionut Nechita wrote:
> 
> Hi Aaron, Daniel,
> 
> As promised, here are the end-to-end results from exercising v16 on our
> 6.18 LTS PREEMPT_RT tree. Short version: managed_irq_strict behaves
> exactly as documented on multiqueue NVMe with a deliberately narrow
> housekeeping pool, including the nr_vectors >= nr_cpus case.
> 
> Series under test (as backported to 6.18 LTS)
> ---------------------------------------------
>   0001 scsi-aacraid: use block-layer helpers for queue count
>   0002 lib/group_cpus: remove dead SMP code
>   0003 lib/group_cpus: add group_mask_cpus_evenly()
>   0004 sched/isolation: prevent out-of-bounds read in isolcpus parsing
>   0005 isolation: introduce the managed_irq_strict isolcpus type
>   0006 blk-mq: use housekeeping CPUs only with isolcpus=managed_irq_strict
>   0007 blk-mq: prevent offlining housekeeping CPUs with online-associated
>        queues
>   0008 genirq/affinity: restrict managed IRQ affinity to housekeeping CPUs
>   0009 docs: document the managed_irq_strict isolcpus flag
>   0010 genirq/affinity: confine reserved pre/post vectors to housekeeping
>        CPUs

Hi Ionut,

First of all, please accept my apologies for the delay in getting back to you on this.

Thank you very much for this comprehensive and thorough test report.
A few comments inline below.


> Note: 0010 is a local follow-up (see section 3), not part of the
> upstream v16 series.
> 
> Test environment
> ----------------
>   Kernel:   6.18.15-rt and 6.18.52-rt (PREEMPT_RT), v16 backported
>             (6.18.52 is the current 6.18 LTS; behaviour is identical on
>             both, the data below is from 6.18.15)
>   CPU:      single-socket Intel Xeon (Granite Rapids), 40C/80T,
>             1 NUMA node (all 80 CPUs on node0)
>   Storage:  2x NVMe (SK hynix PE9010), b6:00.0 and b7:00.0
>   NICs:     8-port E830-CC + 4-port E825-C (managed IRQs)
>   MSI-X:    2235 vectors allocated system-wide
> 
>   cmdline (relevant bits):
>     nohz_full=2-38,42-78
>     isolcpus=nohz,domain,managed_irq_strict,2-38,42-78
>     rcu_nocbs=2-39,42-79 kthread_cpus=0-1,40-41 irqaffinity=39,79
> 
>   => isolated (isolcpus) = 2-38,42-78   (74 CPUs)
>      housekeeping (HK)   = 0,1,39,40,41,79   (6 CPUs)
> 
> Because HK is only 6 CPUs while the box carries 2235 vectors, this is
> squarely the "narrow housekeeping pool, nr_vectors >= nr_cpus" scenario.
> 
> 1) blk-mq queue allocation and mapping (patches 6, 7)
> -----------------------------------------------------
> Both NVMe devices allocate exactly 6 hardware queues, matching the HK
> weight. Each hctx is owned by exactly one HK CPU:
> 
>     nvme0n1 / nvme1n1:
>       hctx0: 0   hctx1: 1   hctx2: 39
>       hctx3: 40  hctx4: 41  hctx5: 79
> 
> No isolated CPU owns a queue. The sysfs cpu_list only shows the HK CPUs
> because blk_mq_map_swqueue() strips the isolated CPUs from hctx->cpumask
> (as intended). The full software-context routing from debugfs confirms
> the 74 isolated CPUs are still mapped onto these 6 HK queues rather than
> left unserved, evenly (~12-13 isolated CPUs per queue), so I/O submitted
> from an isolated core completes on a housekeeping CPU:
> 
>     hctx0 (owner 0):  0,2,8,14,20,26,32,38,47,53,59,65,71,77
>     hctx1 (owner 1):  1,3,9,15,21,27,33,42,48,54,60,66,72,78
>     hctx2 (owner 39): 4,10,16,22,28,34,39,43,49,55,61,67,73
>     hctx3 (owner 40): 5,11,17,23,29,35,40,44,50,56,62,68,74
>     hctx4 (owner 41): 6,12,18,24,30,36,41,45,51,57,63,69,75
>     hctx5 (owner 79): 7,13,19,25,31,37,46,52,58,64,70,76,79
> 
> All 80 CPUs are covered exactly once. This also lines up with the
> hotplug guard below: isolated CPU 3 is routed to hctx1 (owned by HK CPU
> 1), which is precisely why offlining CPU 1 is refused.

This is brilliant to see. It verifies that group_mask_cpus_evenly() and the
updated blk_mq_map_swqueue() logic distribute the isolated submission
contexts evenly across the housekeeping hardware contexts without leaving
any core unserved or over-subscribing a single queue.

> 
> 2) Managed IRQ affinity (patch 8)
> ---------------------------------
> Every NVMe completion-queue vector has its effective_affinity confined
> to the HK set:
> 
>     nvme0q1..q6 eff = 0, 1, 39, 40, 41, 79
>     nvme1q1..q6 eff = 0, 1, 39, 40, 41, 79
> 
> No managed interrupt targets an isolated CPU.

Excellent.

> 
> 3) Reserved pre/post vectors
> ----------------------------
> The NVMe admin (pre_vector) queues also stay off the isolated set:
> 
>     nvme0q0 eff = 79
>     nvme1q0 eff = 39
> 
> Note: in the stock series the reserved pre/post vectors still default to
> irq_default_affinity, which on this config spans the isolated CPUs, so a
> freshly probed admin queue can briefly land on an isolated core before
> any userspace re-steer. On PREEMPT_RT that stray wakeup is precisely
> what we are trying to avoid, so we carry a small follow-up that confines
> the affd->pre_vectors / post_vectors to the housekeeping mask when
> HK_TYPE_MANAGED_IRQ_STRICT is set. The affinities above are with that
> follow-up applied. Happy to post it separately if there is interest in
> closing that gap upstream.

Okay. In kernel/irq/affinity.c, irq_create_affinity_masks() copies
irq_default_affinity for the pre_vectors and post_vectors loops. When
isolcpus=managed_irq_strict is in effect, have you written something
similiar to the following:

--- a/kernel/irq/affinity.c
+++ b/kernel/irq/affinity.c
@@ -30,6 +30,7 @@ irq_create_affinity_masks(unsigned int nvecs, struct irq_affinity *affd)
     struct irq_affinity_desc *masks = NULL;
     const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ_STRICT);
     bool hk_enabled = housekeeping_enabled(HK_TYPE_MANAGED_IRQ_STRICT);
+    const struct cpumask *def_mask;

     /*
      * Determine the number of vectors which need interrupt affinities
@@ -63,9 +64,14 @@ irq_create_affinity_masks(unsigned int nvecs, struct irq_affinity *affd)
     if (!masks)
         return NULL;

+    if (hk_enabled)
+        def_mask = hk_mask;
+    else
+        def_mask = irq_default_affinity;
+
     /* Fill out vectors at the beginning that don't need affinity */
     for (curvec = 0; curvec < affd->pre_vectors; curvec++)
-        cpumask_copy(&masks[curvec].mask, irq_default_affinity);
+        cpumask_copy(&masks[curvec].mask, def_mask);

     /*
      * Spread on present CPUs starting from affd->pre_vectors. If we
@@ -104,7 +110,7 @@ irq_create_affinity_masks(unsigned int nvecs, struct irq_affinity *affd)
     else
         curvec = affd->pre_vectors + usedvecs;
     for (; curvec < nvecs; curvec++)
-        cpumask_copy(&masks[curvec].mask, irq_default_affinity);
+        cpumask_copy(&masks[curvec].mask, def_mask);

     /* Mark the managed interrupts */
     for (i = affd->pre_vectors; i < nvecs - affd->post_vectors; i++)


The intention would be to fold the above into Patch 8 ("genirq/affinity:
Restrict managed IRQ affinity to housekeeping CPUs"). However, we would
welcome your patch. Please feel free to post your follow-up patch to the
list so we can review it, or alternatively, we will use the above (or
similiar), and fold it directly into the upcoming v17 revision.

> 
> 4) blk-mq hotplug guard (patch 7)
> ---------------------------------
> Offlining a housekeeping CPU that still serves an online isolated CPU is
> refused:
> 
>     # echo 0 > /sys/devices/system/cpu/cpu1/online
>     -> fails (-EINVAL)
>     blk-mq: cannot offline CPU 1: online isolated CPU 3 is still
>             mapped to hctx1
> 
> After offlining the dependent isolated CPUs first, offlining the HK CPU
> succeeds. suspend/hibernate (cpuhp_tasks_frozen) is correctly exempted.

Very glad to see the hotplug notifier path operating cleanly and rejecting
the teardown with informative diagnostic output.

> 
> 5) Vector count / probe robustness (patch 8)
> --------------------------------------------
> With 2235 vectors against 6 HK CPUs, no device aborted probe: no -ENOSPC
> and no nr_io_queues=0. The minimum-vector safety net in
> irq_calc_affinity_vectors() holds on this narrow-HK box.

This confirms that irq_calc_affinity_vectors() gracefully handles vector
exhaustion risks while maintaining minimum required driver vector bounds.

> 6) Runtime completion locality (PREEMPT_RT threaded IRQs)
> ---------------------------------------------------------
> fio randread 64k, 4 jobs, iodepth=64, 120s, pinned to a single isolated
> CPU via a device-plugin extended resource on Kubernetes.
> 
>   Results:
>     IOPS       : 56.9k
>     Bandwidth  : 3559 MiB/s (3732 MB/s), 417 GiB in 120s
>     clat       : avg 4489us, stdev 30.99us, p99 4555us
>     total IOs  : 6,833,178
> 
> While fio was submitting from that isolated CPU, every threaded NVMe
> IRQ handler on the host was scheduled on a housekeeping CPU
> (SCHED_FIFO, as expected on PREEMPT_RT). `ps -eLo psr,cls,comm |
> grep irq.*nvme` during the run:
> 
>    0 FF irq/195-nvme0q3   0 FF irq/196-nvme0q4   0 FF irq/198-nvme0q6
>    1 FF irq/187-nvme1q2   1 FF irq/188-nvme1q3   1 FF irq/194-nvme0q2
>   39 FF irq/185-nvme1q0
>   40 FF irq/186-nvme1q1  40 FF irq/189-nvme1q4  40 FF irq/193-nvme1q6
>   41 FF irq/191-nvme1q5  41 FF irq/192-nvme0q1  41 FF irq/197-nvme0q5
>   79 FF irq/190-nvme0q0
> 
> All 14 threaded handlers land on {0,1,39,40,41,79} = HK. Zero
> irq/*-nvme* threads scheduled on any of the 74 isolated CPUs.
> 
> The /proc/interrupts delta over the 120s run confirms it at the counter
> level. Every nvme* IRQ that fired incremented only on an HK CPU:
> 
>   irq 192 nvme0q1 CPU0 : +6,809,271   (fio's IOs, one HK owner)
>   irq 187 nvme1q2 CPU1 : +684
>   irq 186 nvme1q1 CPU0 : +598
>   irq 189 nvme1q4 CPU40: +568
>   irq 191 nvme1q5 CPU41: +512
>   irq 188 nvme1q3 CPU39: +304
>   irq 193 nvme1q6 CPU79: +271
>   irq 196 nvme0q4 CPU40: +4
>   irq 194 nvme0q2 CPU1 : +3
> 
> Every single delta is on {0,1,39,40,41,79}. Zero completions on any of
> the 74 isolated CPUs. I/O submitted by a task pinned to an isolated
> core completes entirely on housekeeping cores.

These figures provide definitive proof of completion locality. Over 6.8
million I/O completions and threaded interrupt wakeups without a single
stray interrupt or context switch landing on an isolated core is exactly
the guarantee managed_irq_strict aims to provide.

> 
> Still on my list: a negative-control boot with plain managed_irq to
> show the confinement disappears. I will follow up with that.
> 
> On option 1 vs option 2, my earlier preference stands: keep
> managed_irq_strict as an explicit opt-in and leave managed_irq
> semantics unchanged, to avoid a silent behavioural change for existing
> deployments.
> 
> Tested-by: Ionut Nechita <ionut.nechita@xxxxxxxxxxxxx>
> 
> Thanks,
> Ionut

Thank you again for your time and testing efforts. We will add your
Tested-by tag to the respective patches in v17.

Kind regards,
-- 
Aaron Tomlin