Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[santhosh kumar]

On Wed, Jul 29, 2026 at 4:22 AM Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
>
> On Tue, Jul 28 2026 at 12:12, santhosh kumar wrote:
> > Observed:
> > - Linux 6.13
> > - 384 CPUs
> > - 1000+ NVMe devices
> >
> >
> > With PCI_IRQ_AFFINITY:
> > - some NVMe devices fail to obtain dedicated MSI-X vectors
> >
> >
> > Without PCI_IRQ_AFFINITY:
> > - all NVMe devices obtain 2 MSI-X vectors
> >
> > Investigation suggests an interaction between:
> > - group_cpus_evenly()
> > - irq_create_affinity_masks()
> > - x86 vector allocation
> >
> > Looking for feedback on whether affinity-constrained vector allocation
> > could explain this behavior.
>
> Math perhaps?
>
> Number of CPUs:  384
>
> Total number of available device vectors: ~ (200 * 384) = 76800
>
> NVMe devices try to allocate min(nr_queues, NR_CPUS) queues where each
> queue requires a dedicated interrupt. Add the managament queue to it and
> then it's obvious that the total number of required vectors is larger
> than the number of available vectors in the system when the number of
> devices gets large enough.
>
> Nothing to see here. It's simply resource exhaustion.
>
> If you want that odd setup to be supported you have to talk to the NVME
> people and not to a random list of folks which have absolutely nothing
> to do with NVME.
>
I am trying to create 1024 nvme devices , each device requesting 2 msix vectors.
Total we need 1024*2=2048 msix vectors
Total number of available device vectors: ~ (200 * 384) = 76800
But still seeing following messages:
  [ 9166.370948] Interrupt reservation exceeds available resources
  [ 9167.455029] Interrupt reservation exceeds available resources
  [ 9167.455572] Interrupt reservation exceeds available resources
  [ 9167.455888] Interrupt reservation exceeds available resources

● Summary: Affinity Mask Generation in Linux 6.13
  The affinity mask is generated in multiple layers, here's the exact flow:
  Key Files:
  1. lib/group_cpus.c:347 - group_cpus_evenly()
    - This function creates per-interrupt CPU masks
    - Spreads interrupts across CPUs in a NUMA-aware manner
    - Each interrupt gets assigned to 1 or a small set of specific CPUs
  2. lib/group_cpus.c:14 - grp_spread_init_one()
    - Actually picks which specific CPUs go into each mask
    - Tries to use sibling CPUs (same core) for cache locality
  3. kernel/irq/affinity.c:26 - irq_create_affinity_masks()
    - Calls group_cpus_evenly() to get CPU assignments
    - Creates affinity descriptor array with masks
    - Marks interrupts as "managed" (cannot be changed from userspace)
  4. drivers/pci/msi/msi.c:669 - msix_setup_interrupts()
    - Calls irq_create_affinity_masks() when PCI_IRQ_AFFINITY is set
    - Attaches the masks to MSI-X descriptors

  5. arch/x86/kernel/apic/vector.c:295 - assign_irq_vector_any_locked()
    - Reads the affinity mask created above
    - ONLY allocates from CPUs in that specific mask
    - No fallback to other CPUs!


  The Problem Code:
  // lib/group_cpus.c - Creates restricted CPU masks
  grp_spread_init_one()
  {
      cpu = cpumask_first(nmsk);  // Pick CPU 68 for nvme0q0
      cpumask_set_cpu(cpu, irqmsk);  // Mask = {68} only!
  }

  // arch/x86/kernel/apic/vector.c - Can only use that mask
  assign_irq_vector_any_locked()
  {
      affmsk = irq_data_get_affinity_mask(irqd);  // Gets {68}
      assign_vector_locked(irqd, affmsk);  // ONLY tries CPU 68!
      // ❌ No fallback to CPU 200-383 even if they have free vectors
  }
  This is why disabling PCI_IRQ_AFFINITY fixes the issue - it prevents
the restrictive masks from being created in the first place!
   In Linux 6.13, PCI_IRQ_AFFINITY causes MSI-X vectors to receive
affinity masks generated through irq_create_affinity_masks() and
group_cpus_evenly(). On very large systems with hundreds of CPUs and
thousands of devices, these masks can       become sufficiently
restrictive that vector allocation is limited to a relatively small
subset of CPUs. When those CPUs become vector-constrained, some NVMe
devices receive fewer MSI-X vectors or fall back to shared interrupt
usage. Disabling  PCI_IRQ_AFFINITY removes these affinity constraints
and allows the allocator to place vectors more flexibly, which
explains why all NVMe devices successfully obtain two MSI-X vectors in
the same configuration

> Thanks,
>
>         tglx
>
>
>
>