RE: [PATCH v2] PCI: Add ATS-disable quirk for AMD Stoney GPUs

[Deucher, Alexander]

> -----Original Message-----
> From: David Woodhouse [mailto:dwmw2@xxxxxxxxxxxxx]
> Sent: Friday, May 26, 2017 8:55 AM
> To: Deucher, Alexander; 'Joerg Roedel'
> Cc: 'Joerg Roedel'; Bjorn Helgaas; linux-pci@xxxxxxxxxxxxxxx; linux-
> kernel@xxxxxxxxxxxxxxx; Daniel Drake; Samuel Sieb
> Subject: Re: [PATCH v2] PCI: Add ATS-disable quirk for AMD Stoney GPUs
> 
> On Fri, 2017-05-26 at 11:57 +0000, Deucher, Alexander wrote:
> >
> > FWIW, the GPU driver does not actually use ATS at the moment so I
> > don't think we should see any ATS transactions.
> 
> That's a confusing sentence. The "GPU driver", if you mean software
> running in the OS, wouldn't be expected to have anything to do with
> ATS.
> 
> ATS is something that the CPU itself (or its DMA engine) would do.
> Instead of just performing a DMA transaction to a given bus address,
> and letting the IOMMU do the translation, the hardware might choose to
> first perform an IOTLB lookup, and then later do the actual DMA
> transaction to the pre-translated, raw physical address. Which kind of
> makes a mockery of any kind of protection the IOMMU is supposed to give
> you, but does shave a cycle or two of latency off the DMA when it
> finally happens, since the translation can be done in advance.

+ John, Suravee

Full disclosure, I'm not by any means an expert with ATS.  I guess I'm thinking of PRI support rather than ATS per se.  On the GPU side the GPU's memory controller has multiple paths to system memory, the non-ATS/PRI path and the ATS/PRI path.  The GPU has its own integrated MMU to virtualize the GPU's internal address space per GPU client.  The non-ATS/PRI path uses the GPU's MMU and is just "regular" dma to addresses potentially translated by the IOMMU just like any other device that may not have ATS support.  The system memory has to be resident because if the GPU faults, it can't retry the transaction.  For the ATS/PRI path, the GPU's MMU is bypassed and PASIDs need to be setup on the IOMMU for each client, but once done, transactions that use that interface support retries on GPU page faults (after the OS had paged the memory in and the IOMMU tables been updated) and other features.  I think only the ATS/PRI case uses the ATC on the end point.  John, Suravee, correct me if I'm wrong.

Alex