Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt

From: John Garry
Date: Fri Dec 13 2019 - 07:08:58 EST

Hi Marc,

JFYI, we're still testing this and the patch itself seems to work as
intended.

Here's the kernel log if you just want to see how the interrupts are
getting assigned:
https://pastebin.com/hh3r810g

It is a bit hard to make sense of this dump, specially on such a wide
machine (I want one!)

So do I :) That's the newer "D06CS" board.

without really knowing the topology of the system.

So it's 2x socket, each socket has 2x CPU dies, and each die has 6 clusters of 4 CPUs, which gives 96 in total.


For me, I did get a performance boost for NVMe testing, but my
colleague Xiang Chen saw a drop for our storage test of interest -
that's the HiSi SAS controller. We're trying to make sense of it now.

One of the difference is that with this patch, the initial affinity
is picked inside the NUMA node that matches the ITS.

Is that even for managed interrupts? We're testing the storage controller which uses managed interrupts. I should have made that clearer.

In your case,
that's either node 0 or 2. But it is unclear whether which CPUs these
map to.

Given that I see interrupts mapped to CPUs 0-23 on one side, and 48-71
on the other, it looks like half of your machine gets starved,

Seems that way.

So this is a mystery to me:

[ 23.584192] picked CPU62 IRQ147

147: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ITS-MSI 94404626 Edge hisi_sas_v3_hw cq


and

[ 25.896728] picked CPU62 IRQ183

183: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ITS-MSI 94437398 Edge hisi_sas_v3_hw cq


But mpstat reports for CPU62:

12:44:58 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
12:45:00 AM 62 6.54 0.00 42.99 0.00 6.54 12.15 0.00 0.00 6.54 25.23

I don't know what interrupts they are...

It's the "hisi_sas_v3_hw cq" interrupts which we're interested in.

and that
may be because no ITS targets the NUMA nodes they are part of.

So both storage controllers (which we're interested in for this test) are on socket #0, node #0.

It would
be interesting to see what happens if you manually set the affinity
of the interrupts outside of the NUMA node.


Again, managed, so I don't think it's possible.

Thanks,
John