Re: [PATCH 2/2] KVM: SVM: Add support for Virtual SPEC_CTRL

[Babu Moger]

On 12/10/20 3:36 PM, Jim Mattson wrote:
> On Thu, Dec 10, 2020 at 1:26 PM Babu Moger <babu.moger@xxxxxxx> wrote:
>>
>> Hi Jim,
>>
>>> -----Original Message-----
>>> From: Jim Mattson <jmattson@xxxxxxxxxx>
>>> Sent: Monday, December 7, 2020 5:06 PM
>>> To: Moger, Babu <Babu.Moger@xxxxxxx>
>>> Cc: Paolo Bonzini <pbonzini@xxxxxxxxxx>; Thomas Gleixner
>>> <tglx@xxxxxxxxxxxxx>; Ingo Molnar <mingo@xxxxxxxxxx>; Borislav Petkov
>>> <bp@xxxxxxxxx>; Yu, Fenghua <fenghua.yu@xxxxxxxxx>; Tony Luck
>>> <tony.luck@xxxxxxxxx>; Wanpeng Li <wanpengli@xxxxxxxxxxx>; kvm list
>>> <kvm@xxxxxxxxxxxxxxx>; Lendacky, Thomas <Thomas.Lendacky@xxxxxxx>;
>>> Peter Zijlstra <peterz@xxxxxxxxxxxxx>; Sean Christopherson
>>> <seanjc@xxxxxxxxxx>; Joerg Roedel <joro@xxxxxxxxxx>; the arch/x86
>>> maintainers <x86@xxxxxxxxxx>; kyung.min.park@xxxxxxxxx; LKML <linux-
>>> kernel@xxxxxxxxxxxxxxx>; Krish Sadhukhan <krish.sadhukhan@xxxxxxxxxx>; H .
>>> Peter Anvin <hpa@xxxxxxxxx>; mgross@xxxxxxxxxxxxxxx; Vitaly Kuznetsov
>>> <vkuznets@xxxxxxxxxx>; Phillips, Kim <kim.phillips@xxxxxxx>; Huang2, Wei
>>> <Wei.Huang2@xxxxxxx>
>>> Subject: Re: [PATCH 2/2] KVM: SVM: Add support for Virtual SPEC_CTRL
>>>
>>> On Mon, Dec 7, 2020 at 2:38 PM Babu Moger <babu.moger@xxxxxxx> wrote:
>>>>
>>>> Newer AMD processors have a feature to virtualize the use of the
>>>> SPEC_CTRL MSR. When supported, the SPEC_CTRL MSR is automatically
>>>> virtualized and no longer requires hypervisor intervention.
>>>>
>>>> This feature is detected via CPUID function 0x8000000A_EDX[20]:
>>>> GuestSpecCtrl.
>>>>
>>>> Hypervisors are not required to enable this feature since it is
>>>> automatically enabled on processors that support it.
>>>>
>>>> When this feature is enabled, the hypervisor no longer has to
>>>> intercept the usage of the SPEC_CTRL MSR and no longer is required to
>>>> save and restore the guest SPEC_CTRL setting when switching
>>>> hypervisor/guest modes.  The effective SPEC_CTRL setting is the guest
>>>> SPEC_CTRL setting or'ed with the hypervisor SPEC_CTRL setting. This
>>>> allows the hypervisor to ensure a minimum SPEC_CTRL if desired.
>>>>
>>>> This support also fixes an issue where a guest may sometimes see an
>>>> inconsistent value for the SPEC_CTRL MSR on processors that support
>>>> this feature. With the current SPEC_CTRL support, the first write to
>>>> SPEC_CTRL is intercepted and the virtualized version of the SPEC_CTRL
>>>> MSR is not updated. When the guest reads back the SPEC_CTRL MSR, it
>>>> will be 0x0, instead of the actual expected value. There isn’t a
>>>> security concern here, because the host SPEC_CTRL value is or’ed with
>>>> the Guest SPEC_CTRL value to generate the effective SPEC_CTRL value.
>>>> KVM writes with the guest's virtualized SPEC_CTRL value to SPEC_CTRL
>>>> MSR just before the VMRUN, so it will always have the actual value
>>>> even though it doesn’t appear that way in the guest. The guest will
>>>> only see the proper value for the SPEC_CTRL register if the guest was
>>>> to write to the SPEC_CTRL register again. With Virtual SPEC_CTRL
>>>> support, the MSR interception of SPEC_CTRL is disabled during
>>>> vmcb_init, so this will no longer be an issue.
>>>>
>>>> Signed-off-by: Babu Moger <babu.moger@xxxxxxx>
>>>> ---
>>>
>>> Shouldn't there be some code to initialize a new "guest SPEC_CTRL"
>>> value in the VMCB, both at vCPU creation, and at virtual processor reset?
>>
>> Yes, I think so. I will check on this.
>>
>>>
>>>>  arch/x86/kvm/svm/svm.c |   17 ++++++++++++++---
>>>>  1 file changed, 14 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index
>>>> 79b3a564f1c9..3d73ec0cdb87 100644
>>>> --- a/arch/x86/kvm/svm/svm.c
>>>> +++ b/arch/x86/kvm/svm/svm.c
>>>> @@ -1230,6 +1230,14 @@ static void init_vmcb(struct vcpu_svm *svm)
>>>>
>>>>         svm_check_invpcid(svm);
>>>>
>>>> +       /*
>>>> +        * If the host supports V_SPEC_CTRL then disable the interception
>>>> +        * of MSR_IA32_SPEC_CTRL.
>>>> +        */
>>>> +       if (boot_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               set_msr_interception(&svm->vcpu, svm->msrpm,
>>> MSR_IA32_SPEC_CTRL,
>>>> +                                    1, 1);
>>>> +
>>>>         if (kvm_vcpu_apicv_active(&svm->vcpu))
>>>>                 avic_init_vmcb(svm);
>>>>
>>>> @@ -3590,7 +3598,8 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct
>>> kvm_vcpu *vcpu)
>>>>          * is no need to worry about the conditional branch over the wrmsr
>>>>          * being speculatively taken.
>>>>          */
>>>> -       x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl);
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               x86_spec_ctrl_set_guest(svm->spec_ctrl,
>>>> + svm->virt_spec_ctrl);
>>>
>>> Is this correct for the nested case? Presumably, there is now a "guest
>>> SPEC_CTRL" value somewhere in the VMCB. If L1 does not intercept this MSR,
>>> then we need to transfer the "guest SPEC_CTRL" value from the
>>> vmcb01 to the vmcb02, don't we?
>>
>> Here is the text from to be published documentation.
>> "When in host mode, the host SPEC_CTRL value is in effect and writes
>> update only the host version of SPEC_CTRL. On a VMRUN, the processor loads
>> the guest version of SPEC_CTRL from the VMCB. For non- SNP enabled guests,
>> processor behavior is controlled by the logical OR of the two registers.
>> When the guest writes SPEC_CTRL, only the guest version is updated. On a
>> VMEXIT, the guest version is saved into the VMCB and the processor returns
>> to only using the host SPEC_CTRL for speculation control. The guest
>> SPEC_CTRL is located at offset 0x2E0 in the VMCB."  This offset is into
>> the save area of the VMCB (i.e. 0x400 + 0x2E0).
>>
>> The feature X86_FEATURE_V_SPEC_CTRL will not be advertised to guests.
>> So, the guest will use the same mechanism as today where it will save and
>> restore the value into/from svm->spec_ctrl. If the value saved in the VMSA
>> is left untouched, both an L1 and L2 guest will get the proper value.
>> Thing that matters is the initial setup of vmcb01 and vmcb02 when this
>> feature is available in host(bare metal). I am going to investigate that
>> part. Do you still think I am missing something here?
> 
> It doesn't matter whether X86_FEATURE_V_SPEC_CTRL is advertised to L1
> or not. If L1 doesn't virtualize MSR_SPEC_CTRL for L2, then L1 and L2
> share the same value for that MSR. With this change, the current value
> in vmcb01 is only in vmcb01, and doesn't get propagated anywhere else.
> Hence, if L1 changes the value of MSR_SPEC_CTRL, that change is not
> visible to L2.
> 
> Thinking about what Sean said about live migration, I think the
> correct solution here is that the authoritative value for this MSR
> should continue to live in svm->spec_ctrl. When the CPU supports
> X86_FEATURE_V_SPEC_CTRL, we should just transfer the value into the
> VMCB prior to VMRUN and out of the VMCB after #VMEXIT.

Ok. Got it. I will try this approach. Thanks for the suggestion.

> 
>>
>>>
>>>>         svm_vcpu_enter_exit(vcpu, svm);
>>>>
>>>> @@ -3609,12 +3618,14 @@ static __no_kcsan fastpath_t
>>> svm_vcpu_run(struct kvm_vcpu *vcpu)
>>>>          * If the L02 MSR bitmap does not intercept the MSR, then we need to
>>>>          * save it.
>>>>          */
>>>> -       if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)))
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL) &&
>>>> +           unlikely(!msr_write_intercepted(vcpu,
>>>> + MSR_IA32_SPEC_CTRL)))
>>>>                 svm->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL);
>>>
>>> Is this correct for the nested case? If L1 does not intercept this MSR, then it
>>> might have changed while L2 is running. Presumably, the hardware has stored
>>> the new value somewhere in the vmcb02 at #VMEXIT, but now we need to move
>>> that value into the vmcb01, don't we?
>>>
>>>>         reload_tss(vcpu);
>>>>
>>>> -       x86_spec_ctrl_restore_host(svm->spec_ctrl, svm->virt_spec_ctrl);
>>>> +       if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
>>>> +               x86_spec_ctrl_restore_host(svm->spec_ctrl,
>>>> + svm->virt_spec_ctrl);
>>>>
>>>>         vcpu->arch.cr2 = svm->vmcb->save.cr2;
>>>>         vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax;
>>>>
>>>
>>> It would be great if you could add some tests to kvm-unit-tests.
>>
>> Yes. I will check on this part.
>>
>> Thanks
>> Babu