[PATCH v12 00/24] KVM: x86: Cleaning up the KVM clock mess, part 2

[David Woodhouse]

From: David Woodhouse <dwmw@xxxxxxxxxxxx>

I love it when sashiko-bot finds things that my own local sashiko run didn't.

This is the second half of the KVM clock series which has been churning
since about 2024. The first half¹ is upstream as of 7.4-rc1 (yay!).  This is
the remainder — the masterclock overhaul, and new API for fixing migration.

In v12 I've finally removed the last reason why a *guest* can force the KVM
clock out of masterclock mode (by using the old SYSTEM_TIME MSR). Even if we
do still care about that SUSE kernel from 2015, we can satisfy it just by
updating the master clock when it rewrites the MSR.

Tree: https://git.infradead.org/?p=users/dwmw2/linux.git;a=shortlog;h=refs/heads/kvmclock12

Since v11², fixing what the on-list review found, and more that arose out of
that:

KVM_SET_TSC_KHZ:
 • Synchronizing during vCPU creation, before the vCPU is counted in
   online_vcpus, corrupted the VM-wide matching state.
 • Because the user_set_tsc flag was VM-wide, restoring one vCPU's TSC made
   a sibling's *unrestored* value be preserved across a frequency change.
 • Time spent in the ioctl between sampling the guest TSC and reinstalling
   it was silently subtracted from the vCPU's clock.
 • Setting the unchanged frequency perturbed the generation tracking, and a
   failed call committed the rejected frequency anyway.
 • An above-host-rate frequency on scaling-less hardware left the vCPU in
   catchup mode forever.

The master clock:
 • Making the first vCPU match the TSC generation established at VM
   creation was a silent guest-visible change (its TSC became the cycles
   elapsed since KVM_CREATE_VM), and handed it a stale offset if the host
   suspended while the VM had no vCPUs.
 • KVM_GET_CLOCK trusted a master snapshot which nothing refreshes for a
   VM with no vCPUs across a host suspend, returning a discontinuity
   flagged KVM_CLOCK_TSC_STABLE.
 • A TSC synchronization which changed the vCPU's offset left the guest's
   pvclock unrefreshed, now that offset changes no longer turn the master
   clock off.
 • When the last vCPU reached a new common frequency, nothing refreshed
   the masterclock scaling factors.
 • A pending masterclock update on a newly created vCPU wasn't serviced
   until it first ran, and a vCPU whose creation was aborted part-way
   stayed counted as matched forever.
 • Guests using the legacy MSR_KVM_SYSTEM_TIME lost the master clock
   entirely — the 2015 workaround for SUSE's 2.6.16 freshness heuristic,
   which is better satisfied by sliding the reference point forward when
   the legacy MSR is rewritten (new patch 24).

KVM_[GS]ET_CLOCK_GUEST:
 • The guest TSC comparison mishandled wraparound.
 • The frequency sanity check could underflow, rejected two hosts
   calibrated exactly 1kHz apart, and hvclock_to_hz() could wrap on
   absurd user-supplied mul/shift.
 • The -EBUSY case was unreachable, and GET returned from inside the
   seqcount read section.

Elsewhere:
 • kvm_vm_ioctl_set_clock() wrapped if the host clock had legally been
   set to within moments of the 1970 epoch.
 • clang couldn't see that guest_now was always initialized in
   kvm_xen_start_timer() on 32-bit builds (kernel test robot³).
 • Comments and commit messages overclaimed in places, the
   kvm_update_master_clock tracepoint still said "offsetmatched", the
   tools/ uapi header mirror had drifted, and api.rst didn't document the
   new KVM_SET_TSC_KHZ behaviour.
 • Selftests skipped on any errno rather than just -ENODATA, could pass
   on a buggy kernel under adverse scheduling, lost a whole frequency's
   coverage when hardware scaling wasn't engaged, and carried a third
   private copy of the pvclock arithmetic.

¹ https://lore.kernel.org/all/20260826213303.914988-1-seanjc@xxxxxxxxxx
² https://lore.kernel.org/all/20261105154955.2525419-1-dwmw2@xxxxxxxxxxxxx
³ https://lore.kernel.org/all/202611072152.FL5QBzjH-lkp@xxxxxxxxx


David Woodhouse (21):
  KVM: x86: Improve synchronization in kvm_synchronize_tsc()
  KVM: x86: Kill last_tsc_{nsec,write,offset} fields
  KVM: x86: Replace nr_vcpus_matched_tsc count with all_vcpus_matched_tsc bool
  KVM: x86: Allow KVM master clock mode when TSCs are offset from each other
  KVM: x86: Factor out kvm_use_master_clock()
  KVM: x86: Avoid gratuitous global clock updates
  KVM: x86: Avoid redundant masterclock updates from multiple vCPUs
  KVM: x86: Re-synchronize TSC after KVM_SET_TSC_KHZ
  KVM: x86: Cache host vclock_mode for masterclock eligibility checks
  KVM: x86: Remove pvclock_gtod_data and private timekeeping code
  KVM: x86: Activate master clock from kvm_arch_init_vm()
  KVM: selftests: Use UAPI pvclock-abi.h in xen_shinfo_test
  KVM: selftests: Add library helpers for clock and TSC-offset ioctls
  KVM: x86: Add KVM_VCPU_TSC_SCALE and fix the documentation on TSC migration
  KVM: selftests: Add master clock offset test
  KVM: selftests: Add Xen/generic CPUID timing leaf test
  KVM: selftests: Add Xen runstate migration test
  KVM: selftests: Test that the first user TSC write is honoured exactly
  KVM: x86/xen: Drop open-coded fallback clock read in kvm_xen_start_timer()
  KVM: x86: Use a single time snapshot in kvm_vm_ioctl_set_clock()
  KVM: x86: Keep master clock for guests using legacy MSR_KVM_SYSTEM_TIME

Jack Allister (3):
  UAPI: x86: Move pvclock-abi to UAPI for x86 platforms
  KVM: x86: Add KVM_[GS]ET_CLOCK_GUEST for accurate KVM clock migration
  KVM: selftests: Add KVM/PV clock selftest to prove timer correction

 Documentation/virt/kvm/api.rst                |  61 ++
 Documentation/virt/kvm/devices/vcpu.rst       | 137 ++-
 Documentation/virt/kvm/locking.rst            |   2 +-
 MAINTAINERS                                   |   4 +-
 arch/x86/include/asm/kvm_host.h               |  14 +-
 arch/x86/include/uapi/asm/kvm.h               |   7 +
 arch/x86/include/{ => uapi}/asm/pvclock-abi.h |  27 +-
 arch/x86/kvm/msrs.c                           |  28 +-
 arch/x86/kvm/trace.h                          |  12 +-
 arch/x86/kvm/x86.c                            | 957 ++++++++++++++----
 arch/x86/kvm/x86.h                            |   2 +-
 arch/x86/kvm/xen.c                            |  48 +-
 include/uapi/linux/kvm.h                      |   4 +
 scripts/xen-hypercalls.sh                     |   2 +-
 tools/arch/x86/include/asm/pvclock-abi.h      |  27 +-
 tools/arch/x86/include/uapi/asm/kvm.h         |   6 +
 tools/testing/selftests/kvm/Makefile.kvm      |   6 +
 .../selftests/kvm/include/x86/processor.h     |  50 +
 .../kvm/x86/masterclock_offset_test.c         | 203 ++++
 .../kvm/x86/pvclock_migration_test.c          | 521 ++++++++++
 .../testing/selftests/kvm/x86/pvclock_test.c  | 518 ++++++++++
 .../selftests/kvm/x86/tsc_user_set_test.c     | 153 +++
 .../selftests/kvm/x86/xen_cpuid_timing_test.c | 271 +++++
 .../selftests/kvm/x86/xen_migration_test.c    | 222 ++++
 .../selftests/kvm/x86/xen_shinfo_test.c       |  31 +-
 25 files changed, 3010 insertions(+), 303 deletions(-)
 rename arch/x86/include/{ => uapi}/asm/pvclock-abi.h (82%)
 create mode 100644 tools/testing/selftests/kvm/x86/masterclock_offset_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_migration_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/pvclock_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/tsc_user_set_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_cpuid_timing_test.c
 create mode 100644 tools/testing/selftests/kvm/x86/xen_migration_test.c


base-commit: dea4f6380f4ec96e8f3b0aaf2343764bd0d25ef4
-- 
2.43.0