Re: [RFC] PCI_IRQ_AFFINITY limits MSI-X allocation on 384 CPU / 1000+ NVMe system

[Keith Busch]

On Tue, Aug 04, 2026 at 04:10:23PM +0200, Christoph Hellwig wrote:
> On Mon, Aug 03, 2026 at 05:43:59AM -1000, Keith Busch wrote:
> > On Thu, Jul 30, 2026 at 09:44:21PM +0200, Thomas Gleixner wrote:
> > > On Thu, Jul 30 2026 at 07:28, Keith Busch wrote:
> > > > We can introduce a module parameter to throttle down the maximum number
> > > > of IO queues to allocate per controller. I don't think the driver can
> > > > automatically reason out what the correct number should be because it
> > > > doesn't know how many devices it's going to see.
> > > 
> > > I might be missing something here as usual, but the number of io queues
> > > per controller does not affect that at all as I explained before.
> > 
> > Oh, you did explain that. I skipped to the end as I had too many
> > distractions last week. Sorry about that, and thanks for the
> > re-explanation.
> > 
> > I mistakenly was thinking the effective_affinity was the only hardware
> > resource used. Back to the drawing board.
> 
> But given that we don't use multiple tag_set by default this can't
> be the issue of the reported unless they didnt report enabling
> multiple tag sets.

I believe the use of additional sets makes the problem worse, but I
think we still have a problem even if we have only the one default set.
This is the quote from Thomas that I'm reading into:

  "the per CPU vector consumption is always _one_ per set."

So just the default set with only 1 NVMe IO queue managed IRQ will still
consume 384 vectors for the reported system even though the effective
affinity is pinned to 1 CPU.