Re: [PATCH v3 1/1] KVM: arm64: Allow cacheable stage 2 mapping using VMA flags

[Jason Gunthorpe]

On Wed, Mar 26, 2025 at 11:51:57AM -0700, Oliver Upton wrote:
> 
>  1) If struct page memory, use a cacheable mapping. CMO for non-FWB.
> 
>  2) If cacheable PFNMAP:
>    a) With FWB, use a cacheable mapping
>    b) Without FWB, fail.
> 
>  3) If VM_ALLOW_ANY_UNCACHED, use Normal Non-Cacheable mapping
> 
>  4) Otherwise, Device-nGnRE
> 
> I understand 2b breaks ABI, but the 'typical' VFIO usages fall into (3)
> and (4).

+1 (and +1 to Sean's remark about strictly tracking the VMA as well)

IMHO not doing 2b is a "security" bug today. Catalin suggested we fix
it as a first step to get agreement on this assement and fix. Once
fixed there is no way for KVM to create a S2 with cachable semantics
different from the VMA, or to have missing CMOs.

That simplifies the discussion of adding 2a to strictly track
cachable.

I also don't see a need for a flag here.

> A pedantic but correct live migration / snapshotting implementation
> on non-FWB would need to do CMOs in case the VM used a non-WB
> mapping for memory.

>From a live migration perspecive, we have already built in a lot of
things on the VFIO side where a live migration can be attempted and
then fail because of late-detected HW incompatibilities. We've sort of
punted this for now to the orchestrator and operator. There is an
expectation the environment will somehow ensure that live migration
machine pools are sufficiently uniform when using VFIO. It is much
more restricted than normal no-VFIO VM live migration.

Given you can't have VM live migration of MMIO backed Cachable memory
without a VFIO live migration driver, I wouldn't worry too much about
these fine details from the KVM side. If /proc/cpuinfo shows the FWB
that would be approximately similar discoverability to all the other
limitations.

Jason