RE: [PATCH v2 5/8] iommu: Add PASID support for DMA mapping API users

[Tian, Kevin]

> From: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Sent: Tuesday, March 15, 2022 10:22 PM
> 
> On Tue, Mar 15, 2022 at 11:16:41AM +0000, Robin Murphy wrote:
> > On 2022-03-15 05:07, Jacob Pan wrote:
> > > DMA mapping API is the de facto standard for in-kernel DMA. It operates
> > > on a per device/RID basis which is not PASID-aware.
> > >
> > > Some modern devices such as Intel Data Streaming Accelerator, PASID is
> > > required for certain work submissions. To allow such devices use DMA
> > > mapping API, we need the following functionalities:
> > > 1. Provide device a way to retrieve a PASID for work submission within
> > > the kernel
> > > 2. Enable the kernel PASID on the IOMMU for the device
> > > 3. Attach the kernel PASID to the device's default DMA domain, let it
> > > be IOVA or physical address in case of pass-through.
> > >
> > > This patch introduces a driver facing API that enables DMA API
> > > PASID usage. Once enabled, device drivers can continue to use DMA APIs
> as
> > > is. There is no difference in dma_handle between without PASID and with
> > > PASID.
> >
> > Surely the main point of PASIDs is to be able to use more than one
> > of them?
> 
> IMHO, not for the DMA API.
> 
> I can't think of good reasons why a single in-kernel device should
> require more than one iommu_domain for use by the DMA API. Even with
> the SIOV cases we have been looking at we don't really see a use case
> for more than one DMA API iommu_domain on a single physical device.

This is correct.

> Do you know of something on the horizon?
> 
> From my view the main point of PASIDs is to assign iommu_domains that
> are not used by the DMA API.
> 
> IMHO it is a device mis-design of IDXD to require all DMA be PASID
> tagged. Devices should be able to do DMA on their RID when the PCI
> function is controlled by a kernel driver. I see this driver facing
> API as addressing a device quirk by aliasing the DMA API of the RID
> into a PASID and that is really all it is good for.

One clarification as I don't agree it is a mis-design.

The IDXD device does support RID-based DMA as the base. PASID-based
DMA API comes to play in two scenarios:

1) When the kernel wants a more scalable way of using IDXD e.g. having
multiple CPUs simultaneously submitting works in a lockless way to a 
shared work queue via a new instruction (ENQCMD) which carries PASID.
Having PASID in the instruction is to identify multiple CPU address spaces 
attached to the shared queue (as required by user SVA). For kernel use 
of IDXD (e.g. page zeroing) in this series all CPUs are accessing the same
IOVA space associated with DMA API thus just requires one PASID attached
to DMA API to use ENQCMD in this optimized mode.

2) When the host wants to share a workqueue between multiple VMs.
In that case the virtual IDXD device exposed to each VM will only support
the shared workqueue mode. Only in this case the DMA API in the
guest must be attached by a PASID as ENQCMD is the only way to submit
works.

So it's just a trick to enable a more scalable use of the IDXD device. Other
than that IDXD does support normal RID-based DMA API when its work
queues are configured in dedicated mode. 😊

> 
> In any case I think we are better to wait for an actual user for multi
> DMA API iommu_domains to come forward before we try to build an API
> for it.
> 
> Jason