Re: [PATCH v6 00/28] KVM: combined patchset for MBEC/GMET support

[Jonas Theisen]

Hello everyone,

we ran into an issue we suspect is indirectly related to the patches 
enabling
MBEC support in the current 7.2-rc6.  We first encountered the issue on 
Proxmox
VE, where we have applied the MBEC patches to our downstream kernel, but all
results below are obtained using an unpatched kernel 7.2-rc6 from kernel.org
and unpatched upstream QEMU.

When using Windows 11 / Server 2022 / Server 2025 with Virtualization-Based
Security (VBS) enabled on an Intel CPU with MBEC support and CPU type 
host, the
VM will occasionally freeze and get stuck at 100% vCPU usage. After some
minutes to hours the VM usually un-freezes itself.  A reliable trigger 
for us
is the free software LatencyMon [1]. As soon as the trace is started inside
the software the VM will freeze within seconds.

We have found several potential workarounds:
- Removing the CPU flags 'vmx-apicv-register' or 'vmx-apicv-vid' apparently
  removes the support for APICv inside Windows and we haven't seen a 
freeze in
  such a VM yet.
- Enabling the Hyper-V enlightenment 'hv-evmcs' also removes the 
reported APICv
  support and runs stable so far.
- Disabling MBEC by removing the CPU flag "vmx-mbec" also prevents the 
freezes
  but as expected the VM is much slower overall

To check MBEC/APICv support inside Windows we ran the following command [2]:

(Get-CimInstance -ClassName Win32_DeviceGuard -Namespace 
root\Microsoft\Windows\DeviceGuard).AvailableSecurityProperties

And check whether MBEC/GMET (value 7) and/or APICv (value 8) support are 
present:

- MBEC disabled, VBS disabled: 8 (APICv)           - no freezes
- MBEC disabled, VBS enabled:  none of the above   - no freezes
- MBEC enabled,  VBS disabled: 7 (MBEC), 8 (APICv) - no freezes
- MBEC enabled,  VBS enabled:  7 (MBEC), 8 (APICv) - freezes

Removing the APICv flags or enabling the hv-evmcs enlightenment (see above)
makes the 8 in the AvailableSecurityProperties output disappear and 
fixes the
freezes for us. This is why we suspect that enabling MBEC support indirectly
causes Windows to use APICv if VBS is enabled and somehow trigger the 
observed
freezes.

Any idea how to troubleshoot this further? Happy to provide more debug
information, just let me know.

We can reproduce the freezes on 7.2-rc6 with QEMU 11.0.3 on
an Intel Core Ultra 7 255H with the following commandline:
'/root/qemu-11.0.3/build/qemu-system-x86_64 \
  -enable-kvm \
  -name win25-440fx \
  -chardev 
'socket,id=qmp,path=/var/run/qemu-server/117.qmp,server=on,wait=off' \
  -mon 'chardev=qmp,mode=control' \
  -pidfile /var/run/qemu-server/117.pid \
  -smbios 'type=1,uuid=30769fc5-9e9a-44bb-b232-9e67892a12b8' \
  -blockdev 
'{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.fd"},"node-name":"pflash0","read-only":true}' 
\
  -blockdev 
'{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/pve/vm-117-disk-0","node-name":"e252459adb2974f3ea2c089ceda6c1a","read-only":false},"node-name":"drive-efidisk0","read-only":false,"size":540672}' 
\
  -smp '8,sockets=1,cores=8,maxcpus=8' \
  -vnc 'unix:/var/run/qemu-server/117.vnc,password=on' \
  -cpu 'host,-cet-ibt,-cet-ss,hv_relaxed' \
  -m 8192 \
  -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' \
  -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' \
  -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
  -blockdev 
'{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/pve/vm-117-disk-1","node-name":"e2a0877c37c7b22323a318bd3346607","read-only":false},"node-name":"drive-scsi0","read-only":false}' 
\
  -device 
'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,device_id=drive-scsi0,bootindex=100,write-cache=on' 
\
  -rtc 'driftfix=slew,base=localtime' \
  -machine 
'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=pc-i440fx-11.0'
'

Sidenote - likely unrelated to the freezes: Removing the Hyper-V 
enlightenment
'hv_relaxed' still causes the VM to freeze but instead of staying frozen for
minutes to hours the VM will throw a BSOD with "CLOCK_WATCHDOG_TIMEOUT 
(0x101)"
or "SYNTHETIC_WATCHDOG_TIMEOUT (0x1CA)" after around one minute and reboot
after that.

Thank you!

Best regards
Jonas

[1] https://www.resplendence.com/latencymon
[2] 
https://learn.microsoft.com/en-us/windows/security/hardware-security/enable-virtualization-based-protection-of-code-integrity?tabs=security