Re: [PATCH v2 06/11] iommu/arm-smmu-v3: Introduce arm_smmu_s2_parent_tlb_ invalidation helpers

[Jason Gunthorpe]

On Mon, Apr 14, 2025 at 09:57:41PM -0700, Nicolin Chen wrote:
> An S2 nest_parent domain can be shared across vSMMUs in the same VM, since
> the S2 domain is basically the IPA mappings for the entire RAM of the VM.
> 
> Meanwhile, each vSMMU can have its own VMID, so the VMID allocation should
> be done per vSMMU instance v.s. per S2 nest_parent domain.
> 
> However, an S2 domain can be also allocated when a physical SMMU instance
> doesn't support S1. So, the structure has to retain the s2_cfg and vmid.
> 
> Add a per-domain "vsmmus" list pairing with a spinlock, maintaining a list
> of vSMMUs in the S2 parent domain.
> 
> Provide two arm_smmu_s2_parent_tlb_ helpers that will be used for nesting
> cases to invalidate S2 cache using vsmmu->vmid by iterating this "vsmmus"
> list.

I was rather hoping to fix the normal S2 case as well, the nested case
is really not so different.

The challenge with that is to rework the list of invalidation
instructions stored in the smmu_domain to be more general and have
more information, how to invalidate for vsmmu is just another special
case.

> @@ -859,6 +859,10 @@ struct arm_smmu_domain {
>  		struct arm_smmu_ctx_desc	cd;
>  		struct arm_smmu_s2_cfg		s2_cfg;
>  	};
> +	struct {
> +		struct list_head list;
> +		spinlock_t lock;
> +	} vsmmus;

So this approach of just adding more lists is functional, but it isn't
very general :\

This is why it is a tough project, because carefully generalizing the
invalidation data without degrading the performance is certainly
somewhat tricky.

But what I was broadly thinking is to have an allocated array attached
to each domain with something like:

struct invalidation_op {
    struct arm_smmu_device *smmu;
    enum {ATS,S2_VMDIA_IPA,S2_VMID,S1_ASID} invalidation_op;
    union {
        u16 vmid;
        u32 asid;
    	u32 ats_id;
    };
    refcount_t users;
};

Then invalidation would just iterate over this list following each
instruction. 

When things are attached the list is mutated:
 - Normal S1/S2 attach would reuse an ASID for the same instance or
   allocate a new list entry, users keeps track of ID sharing
 - VMID attach would use the VMID of the vSMMU
 - ATS enabled would add entries for each PCI device instead of the
   seperate ATS list

To do this without locking on the invalidation side would require
using RCU to manage the list, which suggests it is probably an array
that is re-allocated each time it is changed.

That means some fancy algorithms to copy and mutate the array, deal
with error cases and sort it (ATS must follow ID, want things grouped
by instance).

There is some tricky memory barriers needed and RCU would require that
SMMU unplug do a synchronize_rcu(). IIRC riscv did this in their
driver.

But the end result is we fully disconnect the domain from the smmu
instance and all domain types can be shared across all instances if
they support the pagetable layout. The invalidation also becomes
somewhat simpler as it just sweeps the list and does what it is
told. The special ATS list, counter and locking is removed too.

Jason