Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers

From: Vitaly Kuznetsov

Date: Mon Nov 16 2026 - 12:13:08 EST


DV <dv@xxxxxxxx> writes:

> Hi Vitaly, Alex, all,
>
> On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote:
>> My bet goes to the clocksource switch, e.g. the guest disables (or
>> just stops using, good luck detecting that :-) ) TSC page and uses
>> raw TSC for some period or something.
>
> I think I've hit a close relative of this, with a reproducer that
> works on every boot. A Windows guest running its own Hyper-V computes
> direct-mode stimer deadlines from the raw TSC, while KVM's reference
> counter lags the TSC because of how QEMU restores kvmclock. Here the
> lag is positive, so the guest's timers fire late instead of storming.
>
> Setup
> -----
>
> Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux
> 7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0.
>
> Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with
> virtualization-based security on. So Windows runs as the root
> partition on its own Hyper-V (L1), nested on KVM.
>
> -cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on,
> hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on,
> hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on,
> hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on,
> hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,...
>
> Symptom
> -------
>
> Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a
> high-resolution waitable timer took up to 466 ms, and an indirect
> display driver paced only by a timer refreshed at 6-8 Hz instead of
> 120. Things only moved when some other interrupt woke a vCPU.
>
> What L1 does
> ------------
>
> With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in
> direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and
> writes absolute counts. Call X the amount by which the reference
> counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are
> X too far out. A timer that should fire right away looks like this:
>
> kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0
> time_now 40095770231 count 40100417164
>
> That count is 464.69 ms ahead of time_now, and X on that boot was
> 464.7 ms. L1 writes the count again on every L2 timer exit, so such a
> timer keeps moving to now + X and only fires once the vCPU is quiet
> for that long.
>
> Without hv-stimer-direct, without hv-frequencies, or without
> hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes,
> a relative countdown), and all timers are on time. I've turned direct
> mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms.
>
> Why the reference counter lags
> ------------------------------
>
> get_time_ref_counter() follows kvmclock on both paths. The fallback
> divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters()
> takes the TSC page's tsc_offset from the pvclock system_time.
>
> QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created.
> kvmclock_vm_state_change() writes the saved value back with
> KVM_SET_CLOCK when the VM starts running, and saves a new value
> whenever it stops. So kvmclock, and with it the reference counter,
> stands still from creation to first run and during every pause. The
> guest TSC keeps counting through both. On six boots X was 297, 465,
> 570, 495, 669 and 512 ms.
>
> Two measurements tie X to this:
>
> - In a trace from QEMU startup, extrapolated back to zero, the guest
> TSC started at host monotonic 265208.1505 s and the reference
> counter at 265208.8205 s, 0.670 s later. X on that boot was
> 669.2 ms.
>
> - virsh suspend for 1.045 s, then resume: X grew from 669.2 to
> 1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by
> the same amount.
>
> hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms)
> didn't change the behaviour. Host load and power management were
> ruled out too.
>
> Connection to your thread
> -------------------------
>
> If the reference counter were ahead of TSC time instead, L1's
> deadlines would land in the past. stimer_start() expires a one-shot
> whose count is in the past at once, L1 arms it again on its next
> exit, and that would look like the expired-stimer storm Alex's patch
> delays.
>
> I haven't been able to try that here, because libvirt is set up to
> start a fresh QEMU whenever the guest reboots. But a reset inside the
> same QEMU process sets the guest TSC back to 0 while kvmclock keeps
> counting, so I'd expect it to flip the sign, possibly by the whole
> previous uptime. A resume path that restores the two clocks
> differently could do the same.
>
> Questions
> ---------
>
> 1. Should KVM keep the Hyper-V reference counter equal to guest TSC /
> TSC frequency when hv-frequencies is exposed? L1 seems to assume
> it is.
>
> 2. If so, where does the fix belong: in QEMU's kvmclock save/restore,
> or in KVM deriving the reference counter (and TSC page) from the
> guest TSC?
>
> 3. If not, should hv-stimer-direct be documented as unsafe for guests
> that run their own Hyper-V?
>

>From what I remember, direct mode synthetic timers were added
specifically to aid Hyper-V-on-KVM case as non-nested Windows was
preferring non-direct version anyway. So I believe we need a fix and not
3).

I'm wondering if the issue is somewhat related to the nasty hack we put
in

commit 0469f2f7ab4c6a6cae4b74c4f981c4da6d909411
Author: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx>
Date: Tue Mar 16 15:37:36 2021 +0100

KVM: x86: hyper-v: Don't touch TSC page values when guest opted for re-enlightenment

but maybe hibernation is somewhat different. Or maybe we misunderstood
the problem in the first place and only fixed one sympthom of it?

> I can share the full traces and the Windows-side timer and frame
> probes if they'd help.
>
> Claude Code, an AI coding assistant, did the tracing and analysis
> while I was looking into why the VM's virtual display ran at a few
> frames per second.
>
> Thanks,
> David
>

--
Vitaly