Re: [RFC PATCH v1 2/8] iommu/arm-smmu-v3: Perform invalidations over installed_smmus

[Michael Shavit]

On Fri, Aug 18, 2023 at 9:51 PM Jason Gunthorpe <jgg@xxxxxxxxxx> wrote:
>
> On Fri, Aug 18, 2023 at 11:44:55AM +0800, Michael Shavit wrote:
> > On Fri, Aug 18, 2023 at 3:41 AM Robin Murphy <robin.murphy@xxxxxxx> wrote:
> > >
> > > On 2023-08-17 20:20, Jason Gunthorpe wrote:
> > > > It is certainly wrong to skip invalidations generated for any other
> > > > reason.
> > > >
> > > >  From what I can tell SVA domains should have their CD table entry
> > > > programmed with "ASET=0" and normal paging domains should be
> > > > programmed with "ASET=1". This causes only the SVA domains to listen
> > > > to the BTM invalidations.
> > >
> > > Correct.
> > >
> > > Thanks,
> > > Robin.
> >
> > Would it be fair to rename arm_smmu_tlb_inv_asid (or move into
> > arm-smmu-v3-sva) to make it explicit that it shouldn't be used outside
> > of SVA then? Or add a parameter such as skip_btm_capable_devices.
>
> ???
>
> arm_smmu_tlb_inv_asid() is generally used in many places and has
> nothing to do with BTM..
>
> Did you mean arm_smmu_tlb_inv_range_asid ?

Whoops yes that's what I meant.


>
> Broadly, invalidation is not SVA specific..
>
> Notice that arm_smmu_tlb_inv_range_asid() already duplicates
> arm_smmu_tlb_inv_range_domain().
>
> IMHO I would split the ATC step out of arm_smmu_mm_invalidate_range(),
> get rid of arm_smmu_tlb_inv_range_domain(), and have the mmu notifier
> just do as it already does:
>
>         if (!(smmu_domain->smmu->features & ARM_SMMU_FEAT_BTM))
>                 arm_smmu_tlb_inv_range_domain_no_atc(start, size, smmu_mn->cd->asid,
>                                             PAGE_SIZE, false, smmu_domain);
>         arm_smmu_atc_inv_domain(smmu_domain, start, size);
>
> And make arm_smmu_tlb_inv_range_domain() just call
>    arm_smmu_tlb_inv_range_domain_no_atc();
>    arm_smmu_atc_inv_domain();

That's a nice clean-up but doesn't really solve the problem faced by this patch.

This patch series eliminates the smmu_domain->smmu handle, replacing
it for a list of SMMUs. So SVA can no longer optimize the
arm_smmu_tlb_inv_range_asid call away by checking whether the SMMU BTM
feature is enabled since there's now a list of SMMUs with possibly
heterogeneous support for the feature. Since there's now a loop over a
series of SMMUs inside arm_smmu_tlb_inv_range_asid, it makes sense to
move the check into that loop. This technically works because only SVA
is calling arm_smmu_tlb_inv_range_asid but can (IMO) risk introducing
bugs in the future since it's not obvious from the function name.

The suggestion was then to introduce a parameter to
arm_smmu_tlb_inv_range_asid (or arm_smmu_tlb_inv_range_domain_no_atc)
to make this behavior explicit in the API.