Re: [PATCH v3 1/4] KVM: nVMX: Don't use Enlightened MSR Bitmap for L3

[Maxim Levitsky]

On Fri, 2021-11-05 at 17:38 +0200, Maxim Levitsky wrote:
> On Fri, 2021-11-05 at 00:52 +0000, Sean Christopherson wrote:
> > On Wed, Oct 13, 2021, Vitaly Kuznetsov wrote:
> > > 3-level nesting is also not a very common setup nowadays.
> > 
> > Says who? :-D
> 
> I regularly test 4 level nesting :P
> It's KVM all the way down.... 
>  
> But jokes aside 3 level nesting will start to happen occasionally more and more often,
> IMHO with windows guests which have accidently/or on purpose enabled HypoerV/Core isolation/WSL3 inside,

*insert some joke about coffee here* 

I mean HyperV/Core Isolation/WSL2. There is no WSL3 yet :)

Best regards,
	Maxim Levitsky
> and that are run nested on KVM.
>  
> Just FYI. I have a patch series pending (reviews are welcome!) which implement nested vVMLOAD/vVMSAVE and
> vGIF which allows L1 to use these optional SVM features to run its nested guests (that is L3s) faster.
> (This series is the reason I was recently stress testing 3/4 level nesting. 
>  
> 4 levels usually work so slow that VM doesn't boot and timeouts in various systemd settings).
> 3rd level works not that bad IMHO.
> 
> All that said I don't have any objections to the patch itself.
> 
> 
> Best regards,
> 	Maxim Levitsky
> > > Don't enable 'Enlightened MSR Bitmap' feature for KVM's L2s (real L3s) for
> > > now.
> > > 
> > > Signed-off-by: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx>
> > > Reviewed-by: Maxim Levitsky <mlevitsk@xxxxxxxxxx>
> > > ---
> > >  arch/x86/kvm/vmx/vmx.c | 21 ++++++++++++---------
> > >  1 file changed, 12 insertions(+), 9 deletions(-)
> > > 
> > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> > > index 1c8b2b6e7ed9..e82cdde58119 100644
> > > --- a/arch/x86/kvm/vmx/vmx.c
> > > +++ b/arch/x86/kvm/vmx/vmx.c
> > > @@ -2655,15 +2655,6 @@ int alloc_loaded_vmcs(struct loaded_vmcs *loaded_vmcs)
> > >  		if (!loaded_vmcs->msr_bitmap)
> > >  			goto out_vmcs;
> > >  		memset(loaded_vmcs->msr_bitmap, 0xff, PAGE_SIZE);
> > > -
> > > -		if (IS_ENABLED(CONFIG_HYPERV) &&
> > > -		    static_branch_unlikely(&enable_evmcs) &&
> > > -		    (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) {
> > > -			struct hv_enlightened_vmcs *evmcs =
> > > -				(struct hv_enlightened_vmcs *)loaded_vmcs->vmcs;
> > > -
> > > -			evmcs->hv_enlightenments_control.msr_bitmap = 1;
> > > -		}
> > >  	}
> > >  
> > >  	memset(&loaded_vmcs->host_state, 0, sizeof(struct vmcs_host_state));
> > > @@ -6903,6 +6894,18 @@ static int vmx_create_vcpu(struct kvm_vcpu *vcpu)
> > >  
> > >  	vmx->loaded_vmcs = &vmx->vmcs01;
> > >  
> > > +	/*
> > > +	 * Use Hyper-V 'Enlightened MSR Bitmap' feature when KVM runs as a
> > > +	 * nested (L1) hypervisor and Hyper-V in L0 supports it.
> > 
> > And maybe call out specifically that KVM intentionally uses this only for vmcs02?
> > 
> > > +	 */
> > > +	if (IS_ENABLED(CONFIG_HYPERV) && static_branch_unlikely(&enable_evmcs)
> > > +	    && (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) {
> > 
> > && on the previous line, I think we'll survive the 82 char line :-)
> > 
> > > +		struct hv_enlightened_vmcs *evmcs =
> > > +			(struct hv_enlightened_vmcs *)vmx->loaded_vmcs->vmcs;
> > 
> > Hmm, what about landing this right after vmcs01's VMCS is allocated?  It's kinda
> > weird, but it makes it more obvious that ->vmcs is not NULL.  And if the cast is
> > simply via a "void *" it all fits on one line.
> > 
> > 	err = alloc_loaded_vmcs(&vmx->vmcs01);
> > 	if (err < 0)
> > 		goto free_pml;
> > 
> > 	/*
> > 	 * Use Hyper-V 'Enlightened MSR Bitmap' feature when KVM runs as a
> > 	 * nested (L1) hypervisor and Hyper-V in L0 supports it.  Enable an
> > 	 * enlightened bitmap only for vmcs01, KVM currently isn't equipped to
> > 	 * realize any performance benefits from enabling it for vmcs02.
> > 	 */ 
> > 	if (IS_ENABLED(CONFIG_HYPERV) && static_branch_unlikely(&enable_evmcs) &&
> > 	    (ms_hyperv.nested_features & HV_X64_NESTED_MSR_BITMAP)) {
> > 		struct hv_enlightened_vmcs *evmcs = (void *)vmx->vmcs01.vmcs;
> > 
> > 		evmcs->hv_enlightenments_control.msr_bitmap = 1;
> > 	}
> > 
> > > +
> > > +		evmcs->hv_enlightenments_control.msr_bitmap = 1;
> > > +	}
> > > +
> > >  	if (cpu_need_virtualize_apic_accesses(vcpu)) {
> > >  		err = alloc_apic_access_page(vcpu->kvm);
> > >  		if (err)
> > > -- 
> > > 2.31.1
> > >