Re: [ANNOUNCE] PUCK Notes - 2024.01.24 - Memtypes for non-coherent DMA

[Jason Gunthorpe]

On Wed, Jan 24, 2024 at 10:24:44AM -0800, Sean Christopherson wrote:

>  - ARM's architecture doesn't guarantee coherency for mismatched memtypes, so
>    KVM still needs to figure out a solution for ARM, and possibly RISC-V as
>    well.  But for CPU<->CPU access, KVM guarantees host safety, just not
>    functional correctness for the guest, i.e. finding a solution can likely be
>    deferred until a use case comes along.

Regarding the side discussion on ARM DMA coherency enforcement..

Reading the docs more fully, SMMU has no analog to the Intel/AMD
per-page "ignore no snoop" functionality. The driver does the correct
things at the IOMMU API level to indicate this.

Various things say SMMU should map PCIe No Snoop to Normal-iNC-oNC-OSH
on the output transaction.

ARM docs recommend that the VMM clear the "No Snoop Enable" in the PCI
endpoint config space if they want to block No Snoop. I guess this
works for everything and is something we should think about
generically in VFIO to better support iommu drivers that lack
IOMMU_CAP_ENFORCE_CACHE_COHERENCY.

ARM KVM probably needs to do something with
kvm_arch_register_noncoherent_dma() to understand that the VM can
still make the cache incoherent even if FWB is set.

Relatedly the SMMU nested translation design is similar to KVM where
the S1 can contribute memory attributes. The STE.S2FWB behaves
similarly to the KVM where it prevents the S1 from overriding
cachable in the S2.

The nested patches are still to be posted but the current draft does
not set S2FWB, I will get that fixed.

We may have another vfio/iommufd/smmu issue where non-RAM pages are
mapped into the SMMU with IOMMU_CACHABLE, unclear when this would be
practically important but it seems wrong.

Jason