Re: [PATCH v7 00/14] KVM: mm: fd-based approach for supporting KVM guest private memory

[Gupta, Pankaj]

> This is the v7 of this series which tries to implement the fd-based KVM
> guest private memory. The patches are based on latest kvm/queue branch
> commit:
>
>    b9b71f43683a (kvm/queue) KVM: x86/mmu: Buffer nested MMU
> split_desc_cache only by default capacity
>
> Introduction
> ------------
> In general this patch series introduce fd-based memslot which provides
> guest memory through memory file descriptor fd[offset,size] instead of
> hva/size. The fd can be created from a supported memory filesystem
> like tmpfs/hugetlbfs etc. which we refer as memory backing store. KVM

Thinking a bit, As host side fd on tmpfs or shmem will store memory on 
host page cache instead of mapping pages into userspace address space. 
Can we hit double (un-coordinated) page cache problem with this when 
guest page cache is also used?

Thanks,
Pankaj

> and the the memory backing store exchange callbacks when such memslot
> gets created. At runtime KVM will call into callbacks provided by the
> backing store to get the pfn with the fd+offset. Memory backing store
> will also call into KVM callbacks when userspace punch hole on the fd
> to notify KVM to unmap secondary MMU page table entries.
>
> Comparing to existing hva-based memslot, this new type of memslot allows
> guest memory unmapped from host userspace like QEMU and even the kernel
> itself, therefore reduce attack surface and prevent bugs.
>
> Based on this fd-based memslot, we can build guest private memory that
> is going to be used in confidential computing environments such as Intel
> TDX and AMD SEV. When supported, the memory backing store can provide
> more enforcement on the fd and KVM can use a single memslot to hold both
> the private and shared part of the guest memory.
>
> mm extension
> ---------------------
> Introduces new MFD_INACCESSIBLE flag for memfd_create(), the file
> created with these flags cannot read(), write() or mmap() etc via normal
> MMU operations. The file content can only be used with the newly
> introduced memfile_notifier extension.
>
> The memfile_notifier extension provides two sets of callbacks for KVM to
> interact with the memory backing store:
>    - memfile_notifier_ops: callbacks for memory backing store to notify
>      KVM when memory gets invalidated.
>    - backing store callbacks: callbacks for KVM to call into memory
>      backing store to request memory pages for guest private memory.
>
> The memfile_notifier extension also provides APIs for memory backing
> store to register/unregister itself and to trigger the notifier when the
> bookmarked memory gets invalidated.
>
> The patchset also introduces a new memfd seal F_SEAL_AUTO_ALLOCATE to
> prevent double allocation caused by unintentional guest when we only
> have a single side of the shared/private memfds effective.
>
> memslot extension
> -----------------
> Add the private fd and the fd offset to existing 'shared' memslot so
> that both private/shared guest memory can live in one single memslot.
> A page in the memslot is either private or shared. Whether a guest page
> is private or shared is maintained through reusing existing SEV ioctls
> KVM_MEMORY_ENCRYPT_{UN,}REG_REGION.
>
> Test
> ----
> To test the new functionalities of this patch TDX patchset is needed.
> Since TDX patchset has not been merged so I did two kinds of test:
>
> -  Regresion test on kvm/queue (this patchset)
>     Most new code are not covered. Code also in below repo:
>     https://github.com/chao-p/linux/tree/privmem-v7
>
> -  New Funational test on latest TDX code
>     The patch is rebased to latest TDX code and tested the new
>     funcationalities. See below repos:
>     Linux: https://github.com/chao-p/linux/tree/privmem-v7-tdx
>     QEMU: https://github.com/chao-p/qemu/tree/privmem-v7
>
> An example QEMU command line for TDX test:
> -object tdx-guest,id=tdx,debug=off,sept-ve-disable=off \
> -machine confidential-guest-support=tdx \
> -object memory-backend-memfd-private,id=ram1,size=${mem} \
> -machine memory-backend=ram1
>
> Changelog
> ----------
> v7:
>    - Move the private/shared info from backing store to KVM.
>    - Introduce F_SEAL_AUTO_ALLOCATE to avoid double allocation.
>    - Rework on the sync mechanism between zap/page fault paths.
>    - Addressed other comments in v6.
> v6:
>    - Re-organzied patch for both mm/KVM parts.
>    - Added flags for memfile_notifier so its consumers can state their
>      features and memory backing store can check against these flags.
>    - Put a backing store reference in the memfile_notifier and move pfn_ops
>      into backing store.
>    - Only support boot time backing store register.
>    - Overall KVM part improvement suggested by Sean and some others.
> v5:
>    - Removed userspace visible F_SEAL_INACCESSIBLE, instead using an
>      in-kernel flag (SHM_F_INACCESSIBLE for shmem). Private fd can only
>      be created by MFD_INACCESSIBLE.
>    - Introduced new APIs for backing store to register itself to
>      memfile_notifier instead of direct function call.
>    - Added the accounting and restriction for MFD_INACCESSIBLE memory.
>    - Added KVM API doc for new memslot extensions and man page for the new
>      MFD_INACCESSIBLE flag.
>    - Removed the overlap check for mapping the same file+offset into
>      multiple gfns due to perf consideration, warned in document.
>    - Addressed other comments in v4.
> v4:
>    - Decoupled the callbacks between KVM/mm from memfd and use new
>      name 'memfile_notifier'.
>    - Supported register multiple memslots to the same backing store.
>    - Added per-memslot pfn_ops instead of per-system.
>    - Reworked the invalidation part.
>    - Improved new KVM uAPIs (private memslot extension and memory
>      error) per Sean's suggestions.
>    - Addressed many other minor fixes for comments from v3.
> v3:
>    - Added locking protection when calling
>      invalidate_page_range/fallocate callbacks.
>    - Changed memslot structure to keep use useraddr for shared memory.
>    - Re-organized F_SEAL_INACCESSIBLE and MEMFD_OPS.
>    - Added MFD_INACCESSIBLE flag to force F_SEAL_INACCESSIBLE.
>    - Commit message improvement.
>    - Many small fixes for comments from the last version.
>
> Links to previous discussions
> -----------------------------
> [1] Original design proposal:
> https://lkml.kernel.org/kvm/20210824005248.200037-1-seanjc@xxxxxxxxxx/
> [2] Updated proposal and RFC patch v1:
> https://lkml.kernel.org/linux-fsdevel/20211111141352.26311-1-chao.p.peng@xxxxxxxxxxxxxxx/
> [3] Patch v5: https://lkml.org/lkml/2022/5/19/861
>
> Chao Peng (12):
>    mm: Add F_SEAL_AUTO_ALLOCATE seal to memfd
>    selftests/memfd: Add tests for F_SEAL_AUTO_ALLOCATE
>    mm: Introduce memfile_notifier
>    mm/memfd: Introduce MFD_INACCESSIBLE flag
>    KVM: Rename KVM_PRIVATE_MEM_SLOTS to KVM_INTERNAL_MEM_SLOTS
>    KVM: Use gfn instead of hva for mmu_notifier_retry
>    KVM: Rename mmu_notifier_*
>    KVM: Extend the memslot to support fd-based private memory
>    KVM: Add KVM_EXIT_MEMORY_FAULT exit
>    KVM: Register/unregister the guest private memory regions
>    KVM: Handle page fault for private memory
>    KVM: Enable and expose KVM_MEM_PRIVATE
>
> Kirill A. Shutemov (1):
>    mm/shmem: Support memfile_notifier
>
>   Documentation/virt/kvm/api.rst             |  77 +++++-
>   arch/arm64/kvm/mmu.c                       |   8 +-
>   arch/mips/include/asm/kvm_host.h           |   2 +-
>   arch/mips/kvm/mmu.c                        |  10 +-
>   arch/powerpc/include/asm/kvm_book3s_64.h   |   2 +-
>   arch/powerpc/kvm/book3s_64_mmu_host.c      |   4 +-
>   arch/powerpc/kvm/book3s_64_mmu_hv.c        |   4 +-
>   arch/powerpc/kvm/book3s_64_mmu_radix.c     |   6 +-
>   arch/powerpc/kvm/book3s_hv_nested.c        |   2 +-
>   arch/powerpc/kvm/book3s_hv_rm_mmu.c        |   8 +-
>   arch/powerpc/kvm/e500_mmu_host.c           |   4 +-
>   arch/riscv/kvm/mmu.c                       |   4 +-
>   arch/x86/include/asm/kvm_host.h            |   3 +-
>   arch/x86/kvm/Kconfig                       |   3 +
>   arch/x86/kvm/mmu.h                         |   2 -
>   arch/x86/kvm/mmu/mmu.c                     |  74 +++++-
>   arch/x86/kvm/mmu/mmu_internal.h            |  18 ++
>   arch/x86/kvm/mmu/mmutrace.h                |   1 +
>   arch/x86/kvm/mmu/paging_tmpl.h             |   4 +-
>   arch/x86/kvm/x86.c                         |   2 +-
>   include/linux/kvm_host.h                   | 105 +++++---
>   include/linux/memfile_notifier.h           |  91 +++++++
>   include/linux/shmem_fs.h                   |   2 +
>   include/uapi/linux/fcntl.h                 |   1 +
>   include/uapi/linux/kvm.h                   |  37 +++
>   include/uapi/linux/memfd.h                 |   1 +
>   mm/Kconfig                                 |   4 +
>   mm/Makefile                                |   1 +
>   mm/memfd.c                                 |  18 +-
>   mm/memfile_notifier.c                      | 123 ++++++++++
>   mm/shmem.c                                 | 125 +++++++++-
>   tools/testing/selftests/memfd/memfd_test.c | 166 +++++++++++++
>   virt/kvm/Kconfig                           |   3 +
>   virt/kvm/kvm_main.c                        | 272 ++++++++++++++++++---
>   virt/kvm/pfncache.c                        |  14 +-
>   35 files changed, 1074 insertions(+), 127 deletions(-)
>   create mode 100644 include/linux/memfile_notifier.h
>   create mode 100644 mm/memfile_notifier.c