Re: [PATCH 2/2] iommu/vt-d: Share DMAR fault IRQ to prevent vector exhaustion

From: Jacob Pan
Date: Mon Apr 08 2024 - 13:34:23 EST


Hi Jacob,

On Mon, 8 Apr 2024 09:05:56 -0700, Jacob Pan
<jacob.jun.pan@xxxxxxxxxxxxxxx> wrote:

> Hi Kevin,
>
> On Mon, 8 Apr 2024 08:48:54 +0000, "Tian, Kevin" <kevin.tian@xxxxxxxxx>
> wrote:
>
> > > From: Jacob Pan <jacob.jun.pan@xxxxxxxxxxxxxxx>
> > > Sent: Thursday, April 4, 2024 7:46 AM
> > >
> > > DMAR fault interrupt is used for per-IOMMU unrecoverable fault
> > > reporting, it occurs only if there is a kernel programming error or
> > > serious hardware failure. In other words, they should never occur
> > > under normal circumstances.
> >
> > this is not accurate. When a device is assigned to a malicious guest
> > then it's not unusual to observe faults.
> >
> Right, a malicious guest kernel could cause unrecoverable faults, e.g.
> wrong privilege.
>
> > in this context you probably meant that it's not a performance path
> > hence sharing the vector is acceptable.
> >
> Yes.
> > >
> > > @@ -1182,7 +1182,6 @@ static void free_iommu(struct intel_iommu
> > > *iommu)
> > > iommu->pr_irq = 0;
> > > }
> > > free_irq(iommu->fault_irq, iommu);
> > > - dmar_free_hwirq(iommu->fault_irq);
> >
> > You still want to free the vector for the iommu which first gets the
> > vector allocated.
> >
> I think we always want to keep this vector since the system always needs
> one vector to share. We will never offline all the IOMMUs, right?
>
> > > @@ -1956,9 +1955,8 @@ void dmar_msi_mask(struct irq_data *data)
> > > raw_spin_unlock_irqrestore(&iommu->register_lock, flag);
> > > }
> > >
> > > -void dmar_msi_write(int irq, struct msi_msg *msg)
> > > +static void dmar_msi_write_msg(struct intel_iommu *iommu, int irq,
> > > struct msi_msg *msg)
> > > {
> >
> > what about iommu_msi_write_msg() to match the first parameter?
> >
> > otherwise it leads to a slightly circled calltrace:
> > dmar_msi_write_msg()
> > dmar_msi_write()
> > dmar_msi_write_msg()
> >
> Good point, will do.
>
> > > +
> > > + /*
> > > + * Only the owner IOMMU of the shared IRQ has its fault event
> > > + * interrupt unmasked after request_irq(), the rest are
> > > explicitly
> > > + * unmasked.
> > > + */
> > > + if (!(iommu->flags & VTD_FLAG_FAULT_IRQ_OWNER))
> > > + dmar_fault_irq_unmask(iommu);
> > > +
> >
> > em there is a problem in dmar_msi_mask() and dmar_msi_mask()
> > which only touches the owner IOMMU. With this shared vector
> > approach we should mask/unmask all IOMMU's together.
> I thought about this as well, in addition to fault_irq,
> dmar_msi_mask/unmask() are used for other DMAR irqs, page request and
> perfmon. So we need a special case for fault_irq there, it is not pretty.
>
> I added a special case here in this patch, thinking we never mask the
> fault_irq since we need to cover the lifetime of the system. I have looked
> at:
> 1.IOMMU suspend/resume, no mask/unmask
Actually, we do call mask/unmask in suspend/unmask noirq phase.
And DMAR-MSI chip has IRQCHIP_SKIP_SET_WAKE flag.

So you are right, I am missing this case where non-owner IOMMU's fault_irqs
are not masked/unmasked.

> 2.IRQ migration, added IRQF_NOBALANCING
>
> maybe I missed some cases?
>
>
> Thanks,
>
> Jacob


Thanks,

Jacob