Re: [PATCH RFC 1/1] genirq: Make threaded handler use irq affinity for managed interrupt

[John Garry]

On 06/12/2019 15:22, Marc Zyngier wrote:

Hi Marc,

> On 2019-12-06 14:35, John Garry wrote:
> > Currently the cpu allowed mask for the threaded part of a threaded irq
> > handler will be set to the effective affinity of the hard irq.
> >
> > Typically the effective affinity of the hard irq will be for a single
> > cpu. As such,
> > the threaded handler would always run on the same cpu as the hard irq.
> >
> > We have seen scenarios in high data-rate throughput testing that the cpu
> > handling the interrupt can be totally saturated handling both the hard
> > interrupt and threaded handler parts, limiting throughput.
> >
> > For when the interrupt is managed, allow the threaded part to run on all
> > cpus in the irq affinity mask.
> >
> > Signed-off-by: John Garry <john.garry@xxxxxxxxxx>
> > ---
> > Âkernel/irq/manage.c | 6 +++++-
> > Â1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
> > index 1753486b440c..8e7f8e758a88 100644
> > --- a/kernel/irq/manage.c
> > +++ b/kernel/irq/manage.c
> > @@ -968,7 +968,11 @@ irq_thread_check_affinity(struct irq_desc *desc,
> > struct irqaction *action)
> > ÂÂÂÂ if (cpumask_available(desc->irq_common_data.affinity)) {
> > ÂÂÂÂÂÂÂÂ const struct cpumask *m;
> >
> > -ÂÂÂÂÂÂÂ m = irq_data_get_effective_affinity_mask(&desc->irq_data);
> > +ÂÂÂÂÂÂÂ if (irqd_affinity_is_managed(&desc->irq_data))
> > +ÂÂÂÂÂÂÂÂÂÂÂ m = desc->irq_common_data.affinity;
> > +ÂÂÂÂÂÂÂ else
> > +ÂÂÂÂÂÂÂÂÂÂÂ m = irq_data_get_effective_affinity_mask(
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ &desc->irq_data);
> > ÂÂÂÂÂÂÂÂ cpumask_copy(mask, m);
> > ÂÂÂÂ } else {
> > ÂÂÂÂÂÂÂÂ valid = false;
>
> Although I completely understand that there are cases where you
> really want to let your thread roam all CPUs, I feel like changing
> this based on a seemingly unrelated property is likely to trigger
> yet another whack-a-mole episode. I'd feel much more comfortable
> if there was a way to let the IRQ subsystem know about what is best.
>
> Shouldn't the endpoint driver know better about it? 

I did propose that same idea here:
https://lore.kernel.org/lkml/fd7d6101-37f4-2d34-f2f7-cfeade610278@xxxxxxxxxx/

And that fits my agenda to get best throughput figures, while not 
possibly affecting others.

But it seems that we could do better to make this a common policy: allow 
the threaded part to roam when that CPU is overloaded, but how...?

Note that
> I have no data supporting an approach or the other, hence playing
> the role of the village idiot here.

Understood. My data is that we get an ~11% throughput boost for our 
storage test with this change.

> Thanks,
>
>  ÂÂÂÂÂÂÂ M.

Thanks,
John