Re: [PATCH RFC 2/3] x86/kvm/hyper-v: move VMX controls sanitization out of nested_enable_evmcs()

[Vitaly Kuznetsov]

Sean Christopherson <sean.j.christopherson@xxxxxxxxx> writes:

> On Thu, Jan 23, 2020 at 08:09:03PM +0100, Vitaly Kuznetsov wrote:
>> Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> writes:
>> 
>> > Paolo Bonzini <pbonzini@xxxxxxxxxx> writes:
>> >
>> >> On 22/01/20 17:29, Vitaly Kuznetsov wrote:
>> >>> Yes, in case we're back to the idea to filter things out in QEMU we can
>> >>> do this. What I don't like is that every other userspace which decides
>> >>> to enable eVMCS will have to perform the exact same surgery as in case
>> >>> it sets allow_unsupported_controls=0 it'll have to know (hardcode) the
>> >>> filtering (or KVM_SET_MSRS will fail) and in case it opts for
>> >>> allow_unsupported_controls=1 Windows guests just won't boot without the
>> >>> filtering.
>> >>> 
>> >>> It seems to be 1:1, eVMCSv1 requires the filter.
>> >>
>> >> Yes, that's the point.  It *is* a hack in KVM, but it is generally
>> >> preferrable to have an easier API for userspace, if there's only one way
>> >> to do it.
>> >>
>> >> Though we could be a bit more "surgical" and only remove
>> >> SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES---thus minimizing the impact on
>> >> non-eVMCS guests.  Vitaly, can you prepare a v2 that does that and adds
>> >> a huge "hack alert" comment that explains the discussion?
>> >
>> > Yes, sure. I'd like to do more testing to make sure filtering out
>> > SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES is enough for other Hyper-V
>> > versions too (who knows how many bugs are there :-)
>> 
>> ... and the answer is -- more than one :-)
>> 
>> I've tested Hyper-V 2016/2019 BIOS and UEFI-booted and the minimal
>> viable set seems to be:
>> 
>> MSR_IA32_VMX_PROCBASED_CTLS2: 
>> 	~SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES
>> 
>> MSR_IA32_VMX_ENTRY_CTLS/MSR_IA32_VMX_TRUE_ENTRY_CTLS:
>> 	~VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL
>> 
>> MSR_IA32_VMX_EXIT_CTLS/MSR_IA32_VMX_TRUE_EXIT_CTLS:
>> 	~VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL
>>  
>> with these filtered out all 4 versions are at least able to boot with >1
>> vCPU and run a nested guest (different from Windows management
>> partition).
>> 
>> This still feels a bit fragile as who knows under which circumstances
>> Hyper-V might want to enable additional (missing) controls.
>
> No strong opinion, I'm good either way.
>
>> If there are no objections and if we still think it would be beneficial
>> to minimize the list of controls we filter out (and not go with the full
>> set like my RFC suggests), I'll prepare v2. (v1, actually, this was RFC).
>
> One last idea, can we keep the MSR filtering as is and add the hack in
> vmx_restore_control_msr()?  That way the (userspace) host and guest see
> the same values when reading the affected MSRs, and eVMCS wouldn't need
> it's own hook to do consistency checks.

Yes but (if I'm not mistaken) we'll have then to keep the filtering we
currently do in nested_enable_evmcs(): if userspace doesn't do
KVM_SET_MSR for VMX MSRs (QEMU<4.2) then the filtering in
vmx_restore_control_msr() won't happen and the guest will see the
unfiltered set of controls...

>
> @@ -1181,28 +1181,38 @@ static int
>  vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)
>  {
>         u64 supported;
> -       u32 *lowp, *highp;
> +       u32 *lowp, *highp, evmcs_unsupported;
>
>         switch (msr_index) {
>         case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
>                 lowp = &vmx->nested.msrs.pinbased_ctls_low;
>                 highp = &vmx->nested.msrs.pinbased_ctls_high;
> +               if (vmx->nested.enlightened_vmcs_enabled)
> +                       evmcs_unsupported = EVMCS1_UNSUPPORTED_PINCTRL;
>                 break;
>         case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
>                 lowp = &vmx->nested.msrs.procbased_ctls_low;
>                 highp = &vmx->nested.msrs.procbased_ctls_high;
> +               if (vmx->nested.enlightened_vmcs_enabled)
> +                       evmcs_unsupported = 0;
>                 break;
>         case MSR_IA32_VMX_TRUE_EXIT_CTLS:
>                 lowp = &vmx->nested.msrs.exit_ctls_low;
>                 highp = &vmx->nested.msrs.exit_ctls_high;
> +               if (vmx->nested.enlightened_vmcs_enabled)
> +                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMEXIT_CTRL;
>                 break;
>         case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
>                 lowp = &vmx->nested.msrs.entry_ctls_low;
>                 highp = &vmx->nested.msrs.entry_ctls_high;
> +               if (vmx->nested.enlightened_vmcs_enabled)
> +                       evmcs_unsupported = EVMCS1_UNSUPPORTED_VMENTRY_CTRL;
>                 break;
>         case MSR_IA32_VMX_PROCBASED_CTLS2:
>                 lowp = &vmx->nested.msrs.secondary_ctls_low;
>                 highp = &vmx->nested.msrs.secondary_ctls_high;
> +               if (vmx->nested.enlightened_vmcs_enabled)
> +                       evmcs_unsupported = EVMCS1_UNSUPPORTED_2NDEXEC;
>                 break;
>         default:
>                 BUG();
> @@ -1210,6 +1220,9 @@ vmx_restore_control_msr(struct vcpu_vmx *vmx, u32 msr_index, u64 data)
>
>         supported = vmx_control_msr(*lowp, *highp);
>
> +       /* HACK! */
> +       data &= ~(u64)evmcs_unsupported << 32;
> +
>         /* Check must-be-1 bits are still 1. */
>         if (!is_bitwise_subset(data, supported, GENMASK_ULL(31, 0)))
>

-- 
Vitaly