Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt

From: John Garry
Date: Tue Dec 10 2019 - 07:05:26 EST

On 10/12/2019 11:36, Marc Zyngier wrote:
On 2019-12-10 10:59, John Garry wrote:

There is no lockup, just a potential performance boost in this change.

My colleague Xiang Chen can provide specifics of the test, as he is
the one running it.

But one key bit of info - which I did not think most relevant before
- that is we have 2x SAS controllers running the throughput test on
the same host.

As such, the completion queue interrupts would be spread identically
over the CPUs for each controller. I notice that ARM GICv3 ITS
interrupt controller (which we use) does not use the generic irq
matrix allocator, which I think would really help with this.

Hi Marc,

Is there any reason for which we couldn't utilise of the generic irq
matrix allocator for GICv3?

Hi Marc,

For a start, the ITS code predates the matrix allocator by about three
years. Also, my understanding of this allocator is that it allows
x86 to cope with a very small number of possible interrupt vectors
per CPU. The ITS doesn't have such issue, as:
1) the namespace is global, and not per CPU
2) the namespace is *huge*
Now, what property of the matrix allocator is the ITS code missing?
I'd be more than happy to improve it.

I think specifically the property that the matrix allocator will try
to find a CPU for irq affinity which "has the lowest number of managed
IRQs allocated" - I'm quoting the comment on matrix_find_best_cpu_managed().

But that decision is due to allocation constraints. You can have at most
256 interrupts per CPU, so the allocator tries to balance it.

On the contrary, the ITS does care about how many interrupt target any
given CPU. The whole 2^24 interrupt namespace can be thrown at a single

The ITS code will make the lowest online CPU in the affinity mask the
target CPU for the interrupt, which may result in some CPUs handling
so many interrupts.

If what you want is for the *default* affinity to be spread around,
that should be achieved pretty easily. Let me have a think about how
to do that.

Cool, I anticipate that it should help my case.

I can also seek out some NVMe cards to see how it would help a more "generic" scenario.