Re: [PATCH v6 7/8] KVM: x86: Load Guest fpu state when accessing MSRs managed by XSAVES

[Yang Weijiang]

On Mon, Aug 12, 2019 at 04:29:21PM -0700, Sean Christopherson wrote:
> On Mon, Aug 12, 2019 at 04:02:03PM -0700, Sean Christopherson wrote:
> > On Thu, Jul 25, 2019 at 11:12:45AM +0800, Yang Weijiang wrote:
> > > From: Sean Christopherson <sean.j.christopherson@xxxxxxxxx>
> > > 
> > > A handful of CET MSRs are not context switched through "traditional"
> > > methods, e.g. VMCS or manual switching, but rather are passed through
> > > to the guest and are saved and restored by XSAVES/XRSTORS, i.e. the
> > > guest's FPU state.
> > > 
> > > Load the guest's FPU state if userspace is accessing MSRs whose values
> > > are managed by XSAVES so that the MSR helper, e.g. vmx_{get,set}_msr(),
> > > can simply do {RD,WR}MSR to access the guest's value.
> > > 
> > > Note that guest_cpuid_has() is not queried as host userspace is allowed
> > > to access MSRs that have not been exposed to the guest, e.g. it might do
> > > KVM_SET_MSRS prior to KVM_SET_CPUID2.
> > > 
> > > Signed-off-by: Sean Christopherson <sean.j.christopherson@xxxxxxxxx>
> > > Co-developed-by: Yang Weijiang <weijiang.yang@xxxxxxxxx>
> > > Signed-off-by: Yang Weijiang <weijiang.yang@xxxxxxxxx>
> > > ---
> > >  arch/x86/kvm/x86.c | 29 ++++++++++++++++++++++++++++-
> > >  1 file changed, 28 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > index fafd81d2c9ea..c657e6a56527 100644
> > > --- a/arch/x86/kvm/x86.c
> > > +++ b/arch/x86/kvm/x86.c
> > > @@ -102,6 +102,8 @@ static void enter_smm(struct kvm_vcpu *vcpu);
> > >  static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
> > >  static void store_regs(struct kvm_vcpu *vcpu);
> > >  static int sync_regs(struct kvm_vcpu *vcpu);
> > > +static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu);
> > > +static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu);
> > >  
> > >  struct kvm_x86_ops *kvm_x86_ops __read_mostly;
> > >  EXPORT_SYMBOL_GPL(kvm_x86_ops);
> > > @@ -2959,6 +2961,12 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
> > >  }
> > >  EXPORT_SYMBOL_GPL(kvm_get_msr_common);
> > >  
> > > +static bool is_xsaves_msr(u32 index)
> > > +{
> > > +	return index == MSR_IA32_U_CET ||
> > > +	       (index >= MSR_IA32_PL0_SSP && index <= MSR_IA32_PL3_SSP);
> > > +}
> > > +
> > >  /*
> > >   * Read or write a bunch of msrs. All parameters are kernel addresses.
> > >   *
> > > @@ -2969,11 +2977,30 @@ static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs,
> > >  		    int (*do_msr)(struct kvm_vcpu *vcpu,
> > >  				  unsigned index, u64 *data))
> > >  {
> > > +	bool fpu_loaded = false;
> > >  	int i;
> > > +	u64 cet_bits = XFEATURE_MASK_CET_USER | XFEATURE_MASK_CET_KERNEL;
> > 
> > Dunno if the compiler will actually generate different code, but this can be a
> > const.
> > 
> > > +	u64 host_xss = 0;
> > > +
> > > +	for (i = 0; i < msrs->nmsrs; ++i) {
> > > +		if (!fpu_loaded && is_xsaves_msr(entries[i].index)) {
> > > +			if (!kvm_x86_ops->xsaves_supported() ||
> > > +			    !kvm_x86_ops->supported_xss())
> > 
> > The "!kvm_x86_ops->supported_xss()" is redundant with the host_xss check
> > below.
> > 
> > > +				continue;
> > 
> > Hmm, vmx_set_msr() should be checking host_xss, arguably we should call
> > do_msr() and let it handle the bad MSR access.  I don't have a strong
> > opinion either way, practically speaking the end result will be the same.
> > 
> > If we do want to handle a misbehaving userspace here, this should be
> > 'break' instead of 'continue'.
> > 
> > > +
> > > +			host_xss = kvm_x86_ops->supported_xss();
> > >  
> > > -	for (i = 0; i < msrs->nmsrs; ++i)
> > > +			if ((host_xss & cet_bits) != cet_bits)
> > 
> > I'm pretty sure this should check for either CET bit being set, not both,
> > e.g. I assume it's possible to enable and expose XFEATURE_MASK_CET_USER
> > but not XFEATURE_MASK_CET_KERNEL.
> > 
> > So something like
> > 
> > 	const u64 cet_bits = XFEATURE_MASK_CET_USER | XFEATURE_MASK_CET_KERNEL;
> > 	const bool cet_supported = kvm_x86_ops->xsaves_supported() &&
> > 				   (kvm_x86_ops->supported_xss() & cet_bits);
> > 
> > 	for (i = 0; i < msrs->nmsrs; ++i) {
> > 		if (!fpu_loaded && cet_supported &&
> > 		    is_xsaves_msr(entries[i].index)) {
> > 			kvm_load_guest_fpu(vcpu);
> > 			fpu_loaded = true;
> > 		}
> > 		if (do_msr(vcpu, entries[i].index, &entries[i].data))
> > 			break;	
> > 	}
> 
> After looking at patch 8/8, and assuming KVM can actually virtualize
> USER and KERNEL independently, we should go with this version that defers
> to do_msr(), otherwise this code would also need to differentiate between
> USER and KERNEL MSRs.  In other words, have __msr_io() load the guest fpu
> if CET is support and any CET MSRs is being accessed, and let vmx_set_msr()
> do the fine grained fault/error handling.
>
Sure, will follow that.
> > or
> > 
> > 	const u64 cet_bits = XFEATURE_MASK_CET_USER | XFEATURE_MASK_CET_KERNEL;
> > 
> > 	for (i = 0; i < msrs->nmsrs; ++i) {
> > 		if (!fpu_loaded && is_xsaves_msr(entries[i].index)) {
> > 			if (!kvm_x86_ops->supported_xss() ||
> > 			    !(kvm_x86_ops->supported_xss() & cet_bits))
> > 				break;
> > 			kvm_load_guest_fpu(vcpu);
> > 			fpu_loaded = true;
> > 		}
> > 		if (do_msr(vcpu, entries[i].index, &entries[i].data))
> > 			break;	
> > 	}
> > 
> > 
> > > +				continue;
> > > +
> > > +			kvm_load_guest_fpu(vcpu);
> > > +			fpu_loaded = true;
> > > +		}
> > >  		if (do_msr(vcpu, entries[i].index, &entries[i].data))
> > >  			break;
> > > +	}
> > > +	if (fpu_loaded)
> > > +		kvm_put_guest_fpu(vcpu);
> > >  
> > >  	return i;
> > >  }
> > > -- 
> > > 2.17.2
> > >