[PATCH v4 00/30] KVM: x86/mmu: Overhaul TDP MMU zapping and flushing

[Paolo Bonzini]

Overhaul TDP MMU's handling of zapping and TLB flushing to reduce the
number of TLB flushes, fix soft lockups and RCU stalls, avoid blocking
vCPUs for long durations while zapping paging structure, and to clean up
the zapping code.

The largest cleanup is to separate the flows for zapping roots (zap
_everything_), zapping leaf SPTEs (zap guest mappings for whatever reason),
and zapping a specific SP (NX recovery).  They're currently smushed into a
single zap_gfn_range(), which was a good idea at the time, but became a
mess when trying to handle the different rules, e.g. TLB flushes aren't
needed when zapping a root because KVM can safely zap a root if and only
if it's unreachable.

To solve the soft lockups, stalls, and vCPU performance issues:

 - Defer remote TLB flushes to the caller when zapping TDP MMU shadow
   pages by relying on RCU to ensure the paging structure isn't freed
   until all vCPUs have exited the guest.

 - Allowing yielding when zapping TDP MMU roots in response to the root's
   last reference being put.  This requires a bit of trickery to ensure
   the root is reachable via mmu_notifier, but it's not too gross.

 - Zap roots in two passes to avoid holding RCU for potential hundreds of
   seconds when zapping guest with terabytes of memory that is backed
   entirely by 4kb SPTEs.

 - Zap defunct roots asynchronously via the common work_queue so that a
   vCPU doesn't get stuck doing the work if the vCPU happens to drop the
   last reference to a root.

The selftest at the end allows populating a guest with the max amount of
memory allowed by the underlying architecture.  The most I've tested is
~64tb (MAXPHYADDR=46) as I don't have easy access to a system with
MAXPHYADDR=52.  The selftest compiles on arm64 and s390x, but otherwise
hasn't been tested outside of x86-64.  It will hopefully do something
useful as is, but there's a non-zero chance it won't get past init with
a high max memory.  Running on x86 without the TDP MMU is comically slow.

Testing: passes kvm-unit-tests and guest installation tests on Intel.
Haven't yet run AMD or selftests.

Thanks,

Paolo

v4:
- collected reviews and typo fixes (plus some typo fixes of my own)

- new patches to simplify reader invariants: they are not allowed to
  acquire references to invalid roots

- new version of "Allow yielding when zapping GFNs for defunct TDP MMU
  root", simplifying the atomic a bit by 1) using xchg and relying on
  its implicit memory barriers 2) relying on readers to have the same
  behavior for the three stats refcount=0/valid, refcount=0/invalid,
  refcount=1/invalid (see previous point)

- switch zapping of invalidated roots to asynchronous workers on a
  per-VM workqueue, fixing a bug in v3 where the extra reference added
  by kvm_tdp_mmu_put_root could be given back twice.  This also replaces
  "KVM: x86/mmu: Use common iterator for walking invalid TDP MMU roots"
  in v3, since it gets rid of next_invalidated_root() in a different way.

- because of the previous point, most of the logic in v3's "KVM: x86/mmu:
  Zap defunct roots via asynchronous worker" moves to the earlier patch
  "KVM: x86/mmu: Zap invalidated roots via asynchronous worker"


v3:
- Drop patches that were applied.
- Rebase to latest kvm/queue.
- Collect a review. [David]
- Use helper instead of goto to zap roots in two passes. [David]
- Add patches to disallow REMOVED "old" SPTE when atomically
  setting SPTE.

Paolo Bonzini (5):
  KVM: x86/mmu: only perform eager page splitting on valid roots
  KVM: x86/mmu: do not allow readers to acquire references to invalid roots
  KVM: x86/mmu: Zap invalidated roots via asynchronous worker
  KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU root
  KVM: x86/mmu: Zap defunct roots via asynchronous worker

Sean Christopherson (25):
  KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP MMU
  KVM: x86/mmu: Fix wrong/misleading comments in TDP MMU fast zap
  KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush logic
  KVM: x86/mmu: Document that zapping invalidated roots doesn't need to flush
  KVM: x86/mmu: Require mmu_lock be held for write in unyielding root iter
  KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP removal
  KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier change_spte
  KVM: x86/mmu: Drop RCU after processing each root in MMU notifier hooks
  KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU
  KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path
  KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw values
  KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery
  KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU
  KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page
  KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range
  KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range()
  KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU resched
  KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages
  KVM: x86/mmu: Zap roots in two passes to avoid inducing RCU stalls
  KVM: x86/mmu: Check for a REMOVED leaf SPTE before making the SPTE
  KVM: x86/mmu: WARN on any attempt to atomically update REMOVED SPTE
  KVM: selftests: Move raw KVM_SET_USER_MEMORY_REGION helper to utils
  KVM: selftests: Split out helper to allocate guest mem via memfd
  KVM: selftests: Define cpu_relax() helpers for s390 and x86
  KVM: selftests: Add test to populate a VM with the max possible guest mem

 arch/x86/include/asm/kvm_host.h               |   2 +
 arch/x86/kvm/mmu/mmu.c                        |  49 +-
 arch/x86/kvm/mmu/mmu_internal.h               |  15 +-
 arch/x86/kvm/mmu/tdp_iter.c                   |   6 +-
 arch/x86/kvm/mmu/tdp_iter.h                   |  15 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    | 559 +++++++++++-------
 arch/x86/kvm/mmu/tdp_mmu.h                    |  26 +-
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   3 +
 .../selftests/kvm/include/kvm_util_base.h     |   5 +
 .../selftests/kvm/include/s390x/processor.h   |   8 +
 .../selftests/kvm/include/x86_64/processor.h  |   5 +
 tools/testing/selftests/kvm/lib/kvm_util.c    |  66 ++-
 .../selftests/kvm/max_guest_memory_test.c     | 292 +++++++++
 .../selftests/kvm/set_memory_region_test.c    |  35 +-
 15 files changed, 794 insertions(+), 293 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/max_guest_memory_test.c

-- 
2.31.1