Re: [BUG 4.15-rc7] IRQ matrix management errors

[Thomas Gleixner]

On Wed, 17 Jan 2018, Thomas Gleixner wrote:

> On Wed, 17 Jan 2018, Keith Busch wrote:
> > On Wed, Jan 17, 2018 at 08:34:22AM +0100, Thomas Gleixner wrote:
> > > Can you trace the matrix allocations from the very beginning or tell me how
> > > to reproduce. I'd like to figure out why this is happening.
> > 
> > Sure, I'll get the irq_matrix events.
> > 
> > I reproduce this on a machine with 112 CPUs and 3 NVMe controllers. The
> > first two NVMe want 112 MSI-x vectors, and the last only 31 vectors. The
> > test runs 'modprobe nvme' and 'modprobe -r nvme' in a loop with 10
> > second delay between each step. Repro occurs within a few iterations,
> > sometimes already broken after the initial boot.
> 
> That doesn't sound right. The vectors should be spread evenly accross the
> CPUs. So ENOSPC should never happen.
> 
> Can you please take snapshots of /sys/kernel/debug/irq/ between the
> modprobe and modprobe -r steps?

The allocation fails because CPU1 has exhausted it's vector space here:

[002] d...   333.028216: irq_matrix_alloc_managed: bit=34 cpu=1 online=1 avl=0 alloc=202 managed=2 online_maps=112 global_avl=22085, global_rsvd=158, total_alloc=460

Now the interesting question is how that happens.

Thanks,

	tglx