Re: [PATCH v2 0/4] KVM: x86: Partially allow KVM_SET_CPUID{,2} after KVM_RUN for CPU hotplug

[Vitaly Kuznetsov]

Igor Mammedov <imammedo@xxxxxxxxxx> writes:

> On Mon, 17 Jan 2022 16:05:38 +0100
> Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> wrote:
>
>> Changes since v1:
>> - Drop the allowlist of items which were allowed to change and just allow
>> the exact same CPUID data [Sean, Paolo]. Adjust selftest accordingly.
>> - Drop PATCH1 as the exact same change got merged upstream.
>> 
>> Recently, KVM made it illegal to change CPUID after KVM_RUN but
>> unfortunately this change is not fully compatible with existing VMMs.
>> In particular, QEMU reuses vCPU fds for CPU hotplug after unplug and it
>> calls KVM_SET_CPUID2. Relax the requirement by implementing an allowing
>> KVM_SET_CPUID{,2} with the exact same data.
>
>
> Can you check following scenario:
>  * on host that has IA32_TSX_CTRL and TSX enabled (RTM/HLE cpuid bits present)
>  * boot 2 vcpus VM with TSX enabled on VMM side but with tsx=off on kernel CLI
>
>      that should cause kernel to set MSR_IA32_TSX_CTRL to 3H from initial 0H
>      and clear RTM+HLE bits in CPUID, check that RTM/HLE cpuid it
>      cleared

Forgive me my ignorance around (not only) TSX :-) I took a "Intel(R)
Xeon(R) CPU E3-1270 v5 @ 3.60GHz" host which seems to have rtm/hle and
booted a guest with 'cpu=host' and with (and without) 'tsx=off' on the
kernel command line. I decided to check what's is MSR_IA32_TSX_CTRL but
I see the following:

# rdmsr 0x122
rdmsr: CPU 0 cannot read MSR 0x00000122

I tried adding 'tsx_ctrl' to my QEMU command line but it complains with
qemu-system-x86_64: warning: host doesn't support requested feature: MSR(10AH).tsx-ctrl [bit 7]

so I think my host is not good enough :-(

Also, I've looked at tsx_clear_cpuid() but it actually writes to
MSR_TSX_FORCE_ABORT MSR (0x10F), not MSR_IA32_TSX_CTRL so I'm confused.

>
>  * hotunplug a VCPU and then replug it again
>     if IA32_TSX_CTRL is reset to initial state, that should re-enable
>     RTM/HLE cpuid bits and KVM_SET_CPUID2 might fail due to difference

Could you please teach me this kung-fu, I mean hot to unplug a
cold-plugged CPU with QMP? Previoulsy, I only did un-plugging for what
I've hotplugged, something like:

(QEMU) device_add driver=host-x86_64-cpu socket-id=0 core-id=2 thread-id=0 id=cpu2
{"return": {}}
(QEMU) device_del id=cpu2
{"return": {}}

What's the ids of the cold-plugged CPUs?

>
> and as Sean pointed out there might be other non constant leafs,
> where exact match check could leave userspace broken.

Indeed, while testing your suggestion I've stumbled upon
CPUID.(EAX=0x12, ECX=1) (SGX) where we mangle ECX from
kvm_vcpu_after_set_cpuid():

        best = kvm_find_cpuid_entry(vcpu, 0x12, 0x1);
	if (best) {
                best->ecx &= vcpu->arch.guest_supported_xcr0 & 0xffffffff;
		best->edx &= vcpu->arch.guest_supported_xcr0 >> 32;
                best->ecx |= XFEATURE_MASK_FPSSE;
        }

In theory, we should just move this to __kvm_update_cpuid_runtime()...
I'll take a look tomorrow.

-- 
Vitaly