Re: [PATCH] kvm: hyper-v: Delay firing of expired stimers
From: DV
Date: Sat Nov 07 2026 - 13:37:14 EST
Hi Vitaly, Alex, all,
On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote:
My bet goes to the clocksource switch, e.g. the guest disables (or
just stops using, good luck detecting that :-) ) TSC page and uses
raw TSC for some period or something.
I think I've hit a close relative of this, with a reproducer that
works on every boot. A Windows guest running its own Hyper-V computes
direct-mode stimer deadlines from the raw TSC, while KVM's reference
counter lags the TSC because of how QEMU restores kvmclock. Here the
lag is positive, so the guest's timers fire late instead of storming.
Setup
-----
Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux
7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0.
Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with
virtualization-based security on. So Windows runs as the root
partition on its own Hyper-V (L1), nested on KVM.
-cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on,
hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on,
hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on,
hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on,
hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,...
Symptom
-------
Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a
high-resolution waitable timer took up to 466 ms, and an indirect
display driver paced only by a timer refreshed at 6-8 Hz instead of
120. Things only moved when some other interrupt woke a vCPU.
What L1 does
------------
With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in
direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and
writes absolute counts. Call X the amount by which the reference
counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are
X too far out. A timer that should fire right away looks like this:
kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0
time_now 40095770231 count 40100417164
That count is 464.69 ms ahead of time_now, and X on that boot was
464.7 ms. L1 writes the count again on every L2 timer exit, so such a
timer keeps moving to now + X and only fires once the vCPU is quiet
for that long.
Without hv-stimer-direct, without hv-frequencies, or without
hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes,
a relative countdown), and all timers are on time. I've turned direct
mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms.
Why the reference counter lags
------------------------------
get_time_ref_counter() follows kvmclock on both paths. The fallback
divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters()
takes the TSC page's tsc_offset from the pvclock system_time.
QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created.
kvmclock_vm_state_change() writes the saved value back with
KVM_SET_CLOCK when the VM starts running, and saves a new value
whenever it stops. So kvmclock, and with it the reference counter,
stands still from creation to first run and during every pause. The
guest TSC keeps counting through both. On six boots X was 297, 465,
570, 495, 669 and 512 ms.
Two measurements tie X to this:
- In a trace from QEMU startup, extrapolated back to zero, the guest
TSC started at host monotonic 265208.1505 s and the reference
counter at 265208.8205 s, 0.670 s later. X on that boot was
669.2 ms.
- virsh suspend for 1.045 s, then resume: X grew from 669.2 to
1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by
the same amount.
hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms)
didn't change the behaviour. Host load and power management were
ruled out too.
Connection to your thread
-------------------------
If the reference counter were ahead of TSC time instead, L1's
deadlines would land in the past. stimer_start() expires a one-shot
whose count is in the past at once, L1 arms it again on its next
exit, and that would look like the expired-stimer storm Alex's patch
delays.
I haven't been able to try that here, because libvirt is set up to
start a fresh QEMU whenever the guest reboots. But a reset inside the
same QEMU process sets the guest TSC back to 0 while kvmclock keeps
counting, so I'd expect it to flip the sign, possibly by the whole
previous uptime. A resume path that restores the two clocks
differently could do the same.
Questions
---------
1. Should KVM keep the Hyper-V reference counter equal to guest TSC /
TSC frequency when hv-frequencies is exposed? L1 seems to assume
it is.
2. If so, where does the fix belong: in QEMU's kvmclock save/restore,
or in KVM deriving the reference counter (and TSC page) from the
guest TSC?
3. If not, should hv-stimer-direct be documented as unsafe for guests
that run their own Hyper-V?
I can share the full traces and the Windows-side timer and frame
probes if they'd help.
Claude Code, an AI coding assistant, did the tracing and analysis
while I was looking into why the VM's virtual display ran at a few
frames per second.
Thanks,
David