Re: [PATCH v2 0/4] KVM: x86: Partially allow KVM_SET_CPUID{,2} after KVM_RUN for CPU hotplug

[Igor Mammedov]

On Tue, 18 Jan 2022 17:34:09 +0100
Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> wrote:

> Igor Mammedov <imammedo@xxxxxxxxxx> writes:
> 
> > On Mon, 17 Jan 2022 16:05:38 +0100
> > Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> wrote:
> >  
> >> Changes since v1:
> >> - Drop the allowlist of items which were allowed to change and just allow
> >> the exact same CPUID data [Sean, Paolo]. Adjust selftest accordingly.
> >> - Drop PATCH1 as the exact same change got merged upstream.
> >> 
> >> Recently, KVM made it illegal to change CPUID after KVM_RUN but
> >> unfortunately this change is not fully compatible with existing VMMs.
> >> In particular, QEMU reuses vCPU fds for CPU hotplug after unplug and it
> >> calls KVM_SET_CPUID2. Relax the requirement by implementing an allowing
> >> KVM_SET_CPUID{,2} with the exact same data.  
> >
> >
> > Can you check following scenario:
> >  * on host that has IA32_TSX_CTRL and TSX enabled (RTM/HLE cpuid bits present)
> >  * boot 2 vcpus VM with TSX enabled on VMM side but with tsx=off on kernel CLI
> >
> >      that should cause kernel to set MSR_IA32_TSX_CTRL to 3H from initial 0H
> >      and clear RTM+HLE bits in CPUID, check that RTM/HLE cpuid it
> >      cleared  
> 
> Forgive me my ignorance around (not only) TSX :-) I took a "Intel(R)
> Xeon(R) CPU E3-1270 v5 @ 3.60GHz" host which seems to have rtm/hle and
> booted a guest with 'cpu=host' and with (and without) 'tsx=off' on the
> kernel command line. I decided to check what's is MSR_IA32_TSX_CTRL but
> I see the following:
> 
> # rdmsr 0x122
> rdmsr: CPU 0 cannot read MSR 0x00000122
> 
> I tried adding 'tsx_ctrl' to my QEMU command line but it complains with
> qemu-system-x86_64: warning: host doesn't support requested feature: MSR(10AH).tsx-ctrl [bit 7]
> 
> so I think my host is not good enough :-(

I've seen it being available on "COOPER LAKE" Xeon

> 
> Also, I've looked at tsx_clear_cpuid() but it actually writes to
> MSR_TSX_FORCE_ABORT MSR (0x10F), not MSR_IA32_TSX_CTRL so I'm confused.

look at tsx_disable()

> >  * hotunplug a VCPU and then replug it again
> >     if IA32_TSX_CTRL is reset to initial state, that should re-enable
> >     RTM/HLE cpuid bits and KVM_SET_CPUID2 might fail due to difference  
> 
> Could you please teach me this kung-fu, I mean hot to unplug a
> cold-plugged CPU with QMP? Previoulsy, I only did un-plugging for what
> I've hotplugged, something like:
> 
> (QEMU) device_add driver=host-x86_64-cpu socket-id=0 core-id=2 thread-id=0 id=cpu2
> {"return": {}}
> (QEMU) device_del id=cpu2
> {"return": {}}
> 
> What's the ids of the cold-plugged CPUs?

it doesn't have to be coldplugged, hot(plug/unplug/plug) sequence is fine as well.
fyi you can use qom_path with device _del from 'info hotpluggable-cpus' output


> > and as Sean pointed out there might be other non constant leafs,
> > where exact match check could leave userspace broken.  
> 
> Indeed, while testing your suggestion I've stumbled upon
> CPUID.(EAX=0x12, ECX=1) (SGX) where we mangle ECX from
> kvm_vcpu_after_set_cpuid():
> 
>         best = kvm_find_cpuid_entry(vcpu, 0x12, 0x1);
> 	if (best) {
>                 best->ecx &= vcpu->arch.guest_supported_xcr0 & 0xffffffff;
> 		best->edx &= vcpu->arch.guest_supported_xcr0 >> 32;
>                 best->ecx |= XFEATURE_MASK_FPSSE;
>         }
> 
> In theory, we should just move this to __kvm_update_cpuid_runtime()...
> I'll take a look tomorrow.
>