[PATCH v2 00/20] KVM: Use atomic SRCU for gfn-to-pfn cache, reinstate guest mode for x86 nesting

[David Woodhouse]

From: David Woodhouse <dwmw@xxxxxxxxxxxx>

Based on v7.4-rc1 (kvm/next doesn't have the atomic SRCU flavour yet),
this series:

 • Converts the GPC to use atomic SRCU, ditching the rwlock which was a
   scaling and latency problem, and which PREEMPT_RT hated because it
   turned the rwlock into a sleeping lock¹.

 • Fixes the pre-existing bugs which torture-testing that conversion
   shook out: a guest-reachable dirty ring overflow from the hypercall
   page write (128 pushes of the same gfn against 64 reserved entries,
   without PML), and a lost TLB flush request in record_steal_time().

 • Adds KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE, for VMMs which
   intercept the guest's MSR write and need to replay it from host
   context².

 • Cleans up the intentional lack of dirty-tracking of Xen shinfo and
   vcpu_info pages (Sean).

 • Converts steal-time / preempted status to use gfn_to_pfn_cache.
   (Carsten)

 • Reinstates guest-mode pinning: replacing the old GUEST_USES_PFN
   mode, letting nested state pages be cached and pinned while in
   guest context.

 • nVMX: Uses GPC for the L1 MSR bitmap, and guest mode for APIC
   access and vAPIC pages. (Fred)

 • nSVM: Uses GPC for vmcb12 / MSR-permissions / IO-permissions pages.
   This cuts the vmcb12 transition cost measurably (selftest included).

 • Re-posts KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates so the
   pinned caches are revalidated lazily, riding out the transient
   KVM_MEMSLOT_INVALID window of an in-flight update via the pending
   request rather than ever entering L2 with a stale latched address.

I had briefly split out some of the preliminary fixes and cleanups
into a separate series which didn't depend on atomic SRCU. But I've
joined it all back together now that atomic SRCU is merged. And I've
ditched half of those 'fixes' as a bad idea anyway³.


Tested: per-commit builds across the series; KVM selftests including
the new invalidation, transition and race tests on 192-way AMD, and
the applicable subset on Intel under PROVE_LOCKING + PROVE_RCU +
DEBUG_ATOMIC_SLEEP (which catches the attribute-ioctl srcu fix on the
very first xen_shinfo_test run when it's reverted; that config is now
a standing part of the testing here). The new mmu_notifier race test
asserts that every one of 5000 event channel deliveries into HVA-based
caches survives continuous MADV_DONTNEED invalidation of the backing
page, and soaks at 10/10.

¹ https://syzkaller.appspot.com/bug?extid=208f7f3e5f59c11aeb90
² https://lore.kernel.org/kvm/c855535b4262ecd41f67734d19e8f48a7f014c2a.camel@xxxxxxxxxxxxx/
³ https://lore.kernel.org/kvm/ac6b1891d6bdf3a855a5b840c462a6e7d464a146.camel@xxxxxxxxxxxxx/

v2: https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/gpc-srcu-v2

 • Absorb the fixes/cleanups which had been split out (and drop the
   delivery-retry half of them³, plus the SCHEDOP_poll fix which Sean
   is picking up separately)
 • Fix the mmu_notifier walk to wait out a grace period still owed by
   a concurrent mutator
 • Keep kvm->srcu in the Xen attribute ioctls for the memslot lookups
 • Pair smp_mb__after_atomic()/__after_spinlock() for the publish/veto
   store->load ordering
 • Nested: treat a refresh failure against a KVM_MEMSLOT_INVALID
   placeholder as transient; re-set vmcs02 controls after a lazy
   degradation heals; initialize caches once at vCPU creation
 • New: dirty ring overflow fix, WRITE_HYPERCALL_PAGE attribute,
   the two Xen race selftests

v1: https://git.infradead.org/users/dwmw2/linux.git/shortlog/refs/heads/gpc-srcu
    https://lore.kernel.org/kvm/20260920211920.928306-1-dwmw2@xxxxxxxxxxxxx/

Carsten Stollmaier (1):
  KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status

David Woodhouse (13):
  KVM: x86/xen: Fill the hypercall page with a single guest write
  KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE
  KVM: x86: Request the guest TLB flush from record_steal_time()
  KVM: selftests: Actually deactivate the HVA-based shared_info cache in
    xen_shinfo_test
  KVM: pfncache: Use atomic SRCU for readers instead of a rwlock
  KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor)
  KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates
  KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page
  KVM: nSVM: Cache L1's MSR permissions map pages
  KVM: nSVM: Cache L1's IO permissions map pages
  KVM: selftests: Add nested transition benchmark
  KVM: selftests: Add Xen evtchn vs memslot-update race test
  KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA
    caches

Fred Griffoul (3):
  KVM: nVMX: Implement cache for L1 MSR bitmap
  KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages
  KVM: selftests: Add nested VMX APIC cache invalidation test

Sean Christopherson (3):
  KVM: x86/xen: Extract delivery of event to vCPU into a separate helper
  KVM: x86/xen: Explicitly tag "shared info" page as never being dirty
    tracked
  KVM: x86/xen: Don't dirty track "vCPU info" page

 Documentation/virt/kvm/api.rst                |  27 +-
 arch/x86/include/asm/kvm_host.h               |   2 +-
 arch/x86/include/uapi/asm/kvm.h               |   3 +
 arch/x86/kvm/msrs.c                           |   7 +-
 arch/x86/kvm/svm/nested.c                     | 174 ++++-
 arch/x86/kvm/svm/svm.c                        |  17 +
 arch/x86/kvm/svm/svm.h                        |  29 +
 arch/x86/kvm/vmx/nested.c                     | 517 +++++++++++--
 arch/x86/kvm/vmx/nested.h                     |   1 +
 arch/x86/kvm/vmx/vmx.c                        |  13 +-
 arch/x86/kvm/vmx/vmx.h                        |  21 +-
 arch/x86/kvm/x86.c                            | 209 ++++--
 arch/x86/kvm/x86.h                            |  33 +
 arch/x86/kvm/xen.c                            | 380 ++++++----
 include/linux/kvm_host.h                      | 121 +++-
 include/linux/kvm_types.h                     |  64 +-
 include/linux/srcu.h                          |   7 +
 tools/testing/selftests/kvm/Makefile.kvm      |   4 +
 .../kvm/x86/nested_transition_bench.c         | 210 ++++++
 .../selftests/kvm/x86/vmx_apic_update_test.c  | 323 +++++++++
 .../kvm/x86/xen_evtchn_hva_notify_race_test.c | 188 +++++
 .../kvm/x86/xen_evtchn_memslot_race_test.c    | 232 ++++++
 .../selftests/kvm/x86/xen_shinfo_test.c       |   2 +-
 .../selftests/kvm/x86/xen_vmcall_test.c       | 118 +++
 virt/kvm/kvm_main.c                           |   9 +
 virt/kvm/pfncache.c                           | 677 +++++++++++++++---
 26 files changed, 2962 insertions(+), 426 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/nested_transition_bench.c
 create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_update_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.43.0