Re: DMA-buf and uncached system memory

[Christian König]

Am 23.06.22 um 10:04 schrieb Lucas Stach:
> Am Donnerstag, dem 23.06.2022 um 09:26 +0200 schrieb Christian König:
> > Am 23.06.22 um 09:13 schrieb Pekka Paalanen:
> > > On Thu, 23 Jun 2022 08:59:41 +0200
> > > Christian König <christian.koenig@xxxxxxx> wrote:
> > >
> > > > The exporter isn't doing anything wrong here. DMA-buf are supposed to be
> > > > CPU cached and can also be cache hot.
> > > Hi,
> > >
> > > what is that statement based on?
> > On the design documentation of DMA-buf and the actual driver
> > implementations.
> >
> > Coherency and snooping of the CPU cache is mandatory for devices and
> > root complexes in the PCI specification. Incoherent access is just an
> > extension.
> >
> > We inherited that by basing DMA-buf on the Linux kernel DMA-API which in
> > turn is largely based on the PCI specification.
> >
> > > Were the (mandatory for CPU access) cpu_access_begin/end functions &
> > > ioctls not supposed to ensure that CPU cache is up-to-date / CPU cache
> > > is fully flushed out?
> > No, those functions are to inform the exporter that the importer has
> > started and finished accessing the buffer using the CPU.
> >
> > There is no signaling in the other direction. In other words the
> > exporter doesn't inform the importer about CPU accesses because it is
> > the owner of the buffer.
> >
> > It's the responsibility of the importer to make sure that it can
> > actually access the data in the buffer. If it can't guarantee that the
> > importer shouldn't import the buffer in the first place.
> This is not really correct. DMA-buf inherited the the map/unmap part
> from the DMA API, which on cache coherent architecture is mostly a no-
> op or ties into the IOMMU implementation to set up the pagetables for
> the translation. On non cache coherent architectures this is the point
> where any any necessary cache maintenance happens. DRM breaks this
> model by caching the DMA-buf mapping for performance reasons.

That's not only because of performance reasons, but also because of 
correctness.

At least the Vulkan API and a bunch of OpenGL extensions make it 
mandatory for the buffer to be cache coherent. The kernel is simply not 
informed about domain transfers.

For example you can just do a CPU copy to a ring buffer and the 
expectation is that an already running shader sees that.

> In the DMA API keeping things mapped is also a valid use-case, but then
> you need to do explicit domain transfers via the dma_sync_* family,
> which DMA-buf has not inherited. Again those sync are no-ops on cache
> coherent architectures, but do any necessary cache maintenance on non
> coherent arches.

Correct, yes. Coherency is mandatory for DMA-buf, you can't use 
dma_sync_* on it when you are the importer.

The exporter could of course make use of that because he is the owner of 
the buffer.

Regards,
Christian.

> Regards,
> Lucas