Re: [PATCH v13 00/26] Add AMD Secure Nested Paging (SEV-SNP) Hypervisor Support

[Paolo Bonzini]

On Thu, Apr 18, 2024 at 9:42 PM Michael Roth <michael.roth@xxxxxxx> wrote:
>
> This patchset is also available at:
>
>   https://github.com/amdese/linux/commits/snp-host-v13
>
> and is based on commit 4d2deb62185f (as suggested by Paolo) from:
>
>   https://git.kernel.org/pub/scm/virt/kvm/kvm.git/log/?h=kvm-coco-queue

This is pretty much ready to go into kvm-coco-queue. Let me know if
you want to do a quick v14 with the few changes I suggested, or I can
do them too.

Then the next steps are:

1) get the mm acks

2) figure out the state of patches 1-3

3) wait for more reviews of course

4) merge everything into kvm/next.

Seems in good shape for a 6.10 target.

Paolo

>
> Patch Layout
> ------------
>
> 01-03: These patches are minor dependencies for this series and are already
>        included in both tip/master and mainline, so are only included here
>        as a stop-gap until merged from one of those trees. These are needed
>        by patch #8 in this series which makes use of CC_ATTR_HOST_SEV_SNP
>
> 04:    This is a small general fix-up for guest_memfd that can be applied
>        independently of this series.
>
> 05-08: These patches add some basic infrastructure and introduces a new
>        KVM_X86_SNP_VM vm_type to handle differences verses the existing
>        KVM_X86_SEV_VM and KVM_X86_SEV_ES_VM types.
>
> 09-11: These implement the KVM API to handle the creation of a
>        cryptographic launch context, encrypt/measure the initial image
>        into guest memory, and finalize it before launching it.
>
> 12-17: These implement handling for various guest-generated events such
>        as page state changes, onlining of additional vCPUs, etc.
>
> 18-21: These implement the gmem hooks needed to prepare gmem-allocated
>        pages before mapping them into guest private memory ranges as
>        well as cleaning them up prior to returning them to the host for
>        use as normal memory. Because this supplants certain activities
>        like issued WBINVDs during KVM MMU invalidations, there's also
>        a patch to avoid duplicating that work to avoid unecessary
>        overhead.
>
> 22:    With all the core support in place, the patch adds a kvm_amd module
>        parameter to enable SNP support.
>
> 23-26: These patches all deal with the servicing of guest requests to handle
>        things like attestation, as well as some related host-management
>        interfaces.
>
>
> Testing
> -------
>
> For testing this via QEMU, use the following tree:
>
>   https://github.com/amdese/qemu/commits/snp-v4-wip3
>
> A patched OVMF is also needed due to upstream KVM no longer supporting MMIO
> ranges that are mapped as private. It is recommended you build the AmdSevX64
> variant as it provides the kernel-hashing support present in this series:
>
>   https://github.com/amdese/ovmf/commits/apic-mmio-fix1d
>
> A basic command-line invocation for SNP would be:
>
>  qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2
>   -machine q35,confidential-guest-support=sev0,memory-backend=ram1
>   -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false
>   -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=
>   -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d-AmdSevX64.fd
>
> With kernel-hashing and certificate data supplied:
>
>  qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2
>   -machine q35,confidential-guest-support=sev0,memory-backend=ram1
>   -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false
>   -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=,certs-path=/home/mroth/cert.blob,kernel-hashes=on
>   -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d-AmdSevX64.fd
>   -kernel /boot/vmlinuz-$ver
>   -initrd /boot/initrd.img-$ver
>   -append "root=UUID=d72a6d1c-06cf-4b79-af43-f1bac4f620f9 ro console=ttyS0,115200n8"
>
> With standard X64 OVMF package with separate image for persistent NVRAM:
>
>  qemu-system-x86_64 -smp 32,maxcpus=255 -cpu EPYC-Milan-v2
>   -machine q35,confidential-guest-support=sev0,memory-backend=ram1
>   -object memory-backend-memfd,id=ram1,size=4G,share=true,reserve=false
>   -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=
>   -bios OVMF_CODE-upstream-20240410-apic-mmio-fix1d.fd
>   -drive if=pflash,format=raw,unit=0,file=OVMF_VARS-upstream-20240410-apic-mmio-fix1d.fd,readonly=off
>
>
> Known issues / TODOs
> --------------------
>
>  * SEV-ES guests may trigger the following warning:
>
>      WARNING: CPU: 151 PID: 4003 at arch/x86/kvm/mmu/mmu.c:5855 kvm_mmu_page_fault+0x33b/0x860 [kvm]
>
>    It is assumed here that these will be resolved once the transition to
>    PFERR_PRIVATE_ACCESS is fully completed, but if that's not the case let me
>    know and will investigate further.
>
>  * Base tree in some cases reports "Unpatched return thunk in use. This should
>    not happen!" the first time it runs an SVM/SEV/SNP guests. This a recent
>    regression upstream and unrelated to this series:
>
>      https://lore.kernel.org/linux-kernel/CANpmjNOcKzEvLHoGGeL-boWDHJobwfwyVxUqMq2kWeka3N4tXA@xxxxxxxxxxxxxx/T/
>
>  * 2MB hugepage support has been dropped pending discussion on how we plan to
>    re-enable it in gmem.
>
>  * Host kexec should work, but there is a known issue with host kdump support
>    while SNP guests are running that will be addressed as a follow-up.
>
>  * SNP kselftests are currently a WIP and will be included as part of SNP
>    upstreaming efforts in the near-term.
>
>
> SEV-SNP Overview
> ----------------
>
> This part of the Secure Encrypted Paging (SEV-SNP) series focuses on the
> changes required to add KVM support for SEV-SNP. This series builds upon
> SEV-SNP guest support, which is now in mainline, and and SEV-SNP host
> initialization support, which is now in linux-next.
>
> While series provides the basic building blocks to support booting the
> SEV-SNP VMs, it does not cover all the security enhancement introduced by
> the SEV-SNP such as interrupt protection, which will added in the future.
>
> With SNP, when pages are marked as guest-owned in the RMP table, they are
> assigned to a specific guest/ASID, as well as a specific GFN with in the
> guest. Any attempts to map it in the RMP table to a different guest/ASID,
> or a different GFN within a guest/ASID, will result in an RMP nested page
> fault.
>
> Prior to accessing a guest-owned page, the guest must validate it with a
> special PVALIDATE instruction which will set a special bit in the RMP table
> for the guest. This is the only way to set the validated bit outside of the
> initial pre-encrypted guest payload/image; any attempts outside the guest to
> modify the RMP entry from that point forward will result in the validated
> bit being cleared, at which point the guest will trigger an exception if it
> attempts to access that page so it can be made aware of possible tampering.
>
> One exception to this is the initial guest payload, which is pre-validated
> by the firmware prior to launching. The guest can use Guest Message requests
> to fetch an attestation report which will include the measurement of the
> initial image so that the guest can verify it was booted with the expected
> image/environment.
>
> After boot, guests can use Page State Change requests to switch pages
> between shared/hypervisor-owned and private/guest-owned to share data for
> things like DMA, virtio buffers, and other GHCB requests.
>
> In this implementation of SEV-SNP, private guest memory is managed by a new
> kernel framework called guest_memfd (gmem). With gmem, a new
> KVM_SET_MEMORY_ATTRIBUTES KVM ioctl has been added to tell the KVM
> MMU whether a particular GFN should be backed by shared (normal) memory or
> private (gmem-allocated) memory. To tie into this, Page State Change
> requests are forward to userspace via KVM_EXIT_VMGEXIT exits, which will
> then issue the corresponding KVM_SET_MEMORY_ATTRIBUTES call to set the
> private/shared state in the KVM MMU.
>
> The gmem / KVM MMU hooks implemented in this series will then update the RMP
> table entries for the backing PFNs to set them to guest-owned/private when
> mapping private pages into the guest via KVM MMU, or use the normal KVM MMU
> handling in the case of shared pages where the corresponding RMP table
> entries are left in the default shared/hypervisor-owned state.
>
> Feedback/review is very much appreciated!
>
> -Mike
>
>
> Changes since v12:
>
>  * rebased to latest kvm-coco-queue branch (commit 4d2deb62185f)
>  * add more input validation for SNP_LAUNCH_START, especially for handling
>    things like MBO/MBZ policy bits, and API major/minor minimums. (Paolo)
>  * block SNP KVM instances from being able to run legacy SEV commands (Paolo)
>  * don't attempt to measure VMSA for vcpu 0/BSP before the others, let
>    userspace deal with the ordering just like with SEV-ES (Paolo)
>  * fix up docs for SNP_LAUNCH_FINISH (Paolo)
>  * introduce svm->sev_es.snp_has_guest_vmsa flag to better distinguish
>    handling for guest-mapped vs non-guest-mapped VMSAs, rename
>    'snp_ap_create' flag to 'snp_ap_waiting_for_reset' (Paolo)
>  * drop "KVM: SEV: Use a VMSA physical address variable for populating VMCB"
>    as it is no longer needed due to above VMSA rework
>  * replace pr_debug_ratelimited() messages for RMP #NPFs with a single trace
>    event
>  * handle transient PSMASH_FAIL_INUSE return codes in kvm_gmem_invalidate(),
>    switch to WARN_ON*()'s to indicate remaining error cases are not expected
>    and should not be seen in practice. (Paolo)
>  * add a cond_resched() in kvm_gmem_invalidate() to avoid soft lock-ups when
>    cleaning up large guest memory ranges.
>  * rename VLEK_REQUIRED to VCEK_DISABLE. it's be more applicable if another
>    key type ever gets added.
>  * don't allow attestation to be paused while an attestation request is
>    being processed by firmware (Tom)
>  * add missing Documentation entry for SNP_VLEK_LOAD
>  * collect Reviewed-by's from Paolo and Tom
>
> Changes since v11:
>
>  * Rebase series on kvm-coco-queue and re-work to leverage more
>    infrastructure between SNP/TDX series.
>  * Drop KVM_SNP_INIT in favor of the new KVM_SEV_INIT2 interface introduced
>    here (Paolo):
>      https://lore.kernel.org/lkml/20240318233352.2728327-1-pbonzini@xxxxxxxxxx/
>  * Drop exposure API fields related to things like VMPL levels, migration
>    agents, etc., until they are actually supported/used (Sean)
>  * Rework KVM_SEV_SNP_LAUNCH_UPDATE handling to use a new
>    kvm_gmem_populate() interface instead of copying data directly into
>    gmem-allocated pages (Sean)
>  * Add support for SNP_LOAD_VLEK, rework the SNP_SET_CONFIG_{START,END} to
>    have simpler semantics that are applicable to management of SNP_LOAD_VLEK
>    updates as well, rename interfaces to the now more appropriate
>    SNP_{PAUSE,RESUME}_ATTESTATION
>  * Fix up documentation wording and do print warnings for
>    userspace-triggerable failures (Peter, Sean)
>  * Fix a race with AP_CREATION wake-up events (Jacob, Sean)
>  * Fix a memory leak with VMSA pages (Sean)
>  * Tighten up handling of RMP page faults to better distinguish between real
>    and spurious cases (Tom)
>  * Various patch/documentation rewording, cleanups, etc.
>
>
> ----------------------------------------------------------------
> Ashish Kalra (1):
>       KVM: SEV: Avoid WBINVD for HVA-based MMU notifications for SNP
>
> Borislav Petkov (AMD) (3):
>       [TEMP] x86/kvm/Kconfig: Have KVM_AMD_SEV select ARCH_HAS_CC_PLATFORM
>       [TEMP] x86/cc: Add cc_platform_set/_clear() helpers
>       [TEMP] x86/CPU/AMD: Track SNP host status with cc_platform_*()
>
> Brijesh Singh (10):
>       KVM: SEV: Add GHCB handling for Hypervisor Feature Support requests
>       KVM: SEV: Add KVM_SEV_SNP_LAUNCH_START command
>       KVM: SEV: Add KVM_SEV_SNP_LAUNCH_UPDATE command
>       KVM: SEV: Add KVM_SEV_SNP_LAUNCH_FINISH command
>       KVM: SEV: Add support to handle GHCB GPA register VMGEXIT
>       KVM: SEV: Add support to handle MSR based Page State Change VMGEXIT
>       KVM: SEV: Add support to handle Page State Change VMGEXIT
>       KVM: SEV: Add support to handle RMP nested page faults
>       KVM: SVM: Add module parameter to enable SEV-SNP
>       KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event
>
> Michael Roth (10):
>       KVM: guest_memfd: Fix PTR_ERR() handling in __kvm_gmem_get_pfn()
>       KVM: SEV: Select KVM_GENERIC_PRIVATE_MEM when CONFIG_KVM_AMD_SEV=y
>       KVM: SEV: Add initial SEV-SNP support
>       KVM: SEV: Add support for GHCB-based termination requests
>       KVM: SEV: Implement gmem hook for initializing private pages
>       KVM: SEV: Implement gmem hook for invalidating private pages
>       KVM: x86: Implement gmem hook for determining max NPT mapping level
>       crypto: ccp: Add the SNP_VLEK_LOAD command
>       crypto: ccp: Add the SNP_{PAUSE,RESUME}_ATTESTATION commands
>       KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event
>
> Tom Lendacky (2):
>       KVM: SEV: Add support to handle AP reset MSR protocol
>       KVM: SEV: Support SEV-SNP AP Creation NAE event
>
>  Documentation/virt/coco/sev-guest.rst              |   69 +-
>  Documentation/virt/kvm/api.rst                     |   73 +
>  .../virt/kvm/x86/amd-memory-encryption.rst         |   88 +-
>  arch/x86/coco/core.c                               |   52 +
>  arch/x86/include/asm/kvm_host.h                    |    2 +
>  arch/x86/include/asm/sev-common.h                  |   22 +-
>  arch/x86/include/asm/sev.h                         |   19 +-
>  arch/x86/include/asm/svm.h                         |    9 +-
>  arch/x86/include/uapi/asm/kvm.h                    |   39 +
>  arch/x86/kernel/cpu/amd.c                          |   38 +-
>  arch/x86/kernel/cpu/mtrr/generic.c                 |    2 +-
>  arch/x86/kernel/sev.c                              |   10 -
>  arch/x86/kvm/Kconfig                               |    4 +
>  arch/x86/kvm/mmu.h                                 |    2 -
>  arch/x86/kvm/mmu/mmu.c                             |    1 +
>  arch/x86/kvm/svm/sev.c                             | 1444 +++++++++++++++++++-
>  arch/x86/kvm/svm/svm.c                             |   39 +-
>  arch/x86/kvm/svm/svm.h                             |   50 +
>  arch/x86/kvm/trace.h                               |   31 +
>  arch/x86/kvm/x86.c                                 |   19 +-
>  arch/x86/virt/svm/sev.c                            |  106 +-
>  drivers/crypto/ccp/sev-dev.c                       |   85 +-
>  drivers/iommu/amd/init.c                           |    4 +-
>  include/linux/cc_platform.h                        |   12 +
>  include/linux/psp-sev.h                            |    4 +-
>  include/uapi/linux/kvm.h                           |   28 +
>  include/uapi/linux/psp-sev.h                       |   39 +
>  include/uapi/linux/sev-guest.h                     |    9 +
>  virt/kvm/guest_memfd.c                             |    8 +-
>  29 files changed, 2229 insertions(+), 79 deletions(-)
>
>