RE: [RFC][PATCH v2 00/13] iommu/arm-smmu-v3: Add NVIDIA implementation

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Thursday, September 2, 2021 10:45 PM
> 
> On Wed, Sep 01, 2021 at 06:55:55AM +0000, Tian, Kevin wrote:
> > > From: Alex Williamson
> > > Sent: Wednesday, September 1, 2021 12:16 AM
> > >
> > > On Mon, 30 Aug 2021 19:59:10 -0700
> > > Nicolin Chen <nicolinc@xxxxxxxxxx> wrote:
> > >
> > > > The SMMUv3 devices implemented in the Grace SoC support NVIDIA's
> > > custom
> > > > CMDQ-Virtualization (CMDQV) hardware. Like the new ECMDQ feature
> first
> > > > introduced in the ARM SMMUv3.3 specification, CMDQV adds multiple
> > > VCMDQ
> > > > interfaces to supplement the single architected SMMU_CMDQ in an
> effort
> > > > to reduce contention.
> > > >
> > > > This series of patches add CMDQV support with its preparational
> changes:
> > > >
> > > > * PATCH-1 to PATCH-8 are related to shared VMID feature: they are
> used
> > > >   first to improve TLB utilization, second to bind a shared VMID with a
> > > >   VCMDQ interface for hardware configuring requirement.
> > >
> > > The vfio changes would need to be implemented in alignment with the
> > > /dev/iommu proposals[1].  AIUI, the VMID is essentially binding
> > > multiple containers together for TLB invalidation, which I expect in
> > > the proposal below is largely already taken care of in that a single
> > > iommu-fd can support multiple I/O address spaces and it's largely
> > > expected that a hypervisor would use a single iommu-fd so this explicit
> > > connection by userspace across containers wouldn't be necessary.
> >
> > Agree. VMID is equivalent to DID (domain id) in other vendor iommus.
> > with /dev/iommu multiple I/O address spaces can share the same VMID
> > via nesting. No need of exposing VMID to userspace to build the
> > connection.
> 
> Indeed, this looks like a flavour of the accelerated invalidation
> stuff we've talked about already.
> 
> I would see it probably exposed as some HW specific IOCTL on the iommu
> fd to get access to the accelerated invalidation for IOASID's in the
> FD.
> 
> Indeed, this seems like a further example of why /dev/iommu is looking
> like a good idea as this RFC is very complicated to do something
> fairly simple.
> 
> Where are thing on the /dev/iommu work these days?
> 

We are actively working on the basic skeleton. Our original plan is to send
out the 1st draft before LPC, with support of vfio type1 semantics and
and pci dev only (single-device group). But later we realized that adding
multi-devices group support is also necessary even in the 1st draft to avoid
some dirty hacks and build the complete picture. This will add some time
though. If things go well, we'll still try hit the original plan. If not, it will be
soon after LPC.

Thanks
Kevin