Re: [PATCH kernel v3] x86/compressed/64: reduce #VC nesting for intercepted CPUID for SEV-SNP guest

[Tom Lendacky]

On 10/3/23 18:22, Alexey Kardashevskiy wrote:
> On 4/10/23 04:21, Tom Lendacky wrote:
> > On 10/3/23 02:31, Alexey Kardashevskiy wrote:
> > > For certain intercepts an SNP guest uses the GHCB protocol to talk to
> > > the hypervisor from the #VC handler. The protocol requires a shared 
> > > page so
> > > there is one per vCPU. In case NMI arrives in a middle of #VC or the NMI
> > > handler triggers a #VC, there is another "backup" GHCB page which stores
> > > the content of the first one while SVM_VMGEXIT_NMI_COMPLETE is sent.
> > > The vc_raw_handle_exception() handler manages main and backup GHCB pages
> > > via __sev_get_ghcb/__sev_put_ghcb.
> > >
> > > This works fine for #VC and occasional NMIs but not so fine when the #VC
> > > handler causes intercept + another #VC. If NMI arrives during
> > > the second #VC, there are no more pages for SVM_VMGEXIT_NMI_COMPLETE.
> > > The problem place is the #VC CPUID handler which reads an MSR which
> > > triggers another #VC and if "perf" was running, panic happens:
> > >
> > > Kernel panic - not syncing: Unable to handle #VC exception! GHCB and 
> > > Backup GHCB are already in use
> > >
> > > Add a helper similar to native_read_msr_safe() for making a direct 
> > > hypercall
> > > in the SEV-ES environment. Use the new helper instead of the raw 
> > > "rdmsr" to
> > > avoid the extra #VC event.
> > >
> > > Fixes: ee0bfa08a345 ("x86/compressed/64: Add support for SEV-SNP CPUID 
> > > table in #VC handlers")
> > > Signed-off-by: Alexey Kardashevskiy <aik@xxxxxxx>
> > > ---
> > >
> > > Based on:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/log/?h=tip-x86-urgent
> > > which top at the time was:
> > > 62d5e970d022 "x86/sev: Change npages to unsigned long in 
> > > snp_accept_memory()"
> > >
> > > ---
> > > Changes:
> > > v3:
> > > * made it a function, mimic native_read_msr_safe() which 1) returns 
> > > value 2) returns an error
> > > * removed debug backtraces the commit log as these were added for 
> > > debugging and never
> > > appear with actual kernels
> > >
> > >
> > > v2:
> > > * de-uglify by defining rdmsr_safe_GHCB()
> > > ---
> > >   arch/x86/kernel/sev-shared.c | 27 +++++++++++++++++---
> > >   1 file changed, 23 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c
> > > index dcf325b7b022..494d92a71986 100644
> > > --- a/arch/x86/kernel/sev-shared.c
> > > +++ b/arch/x86/kernel/sev-shared.c
> > > @@ -241,6 +241,25 @@ static enum es_result sev_es_ghcb_hv_call(struct 
> > > ghcb *ghcb,
> > >       return verify_exception_info(ghcb, ctxt);
> > >   }
> > > +
> > > +/* Paravirt SEV-ES rdmsr which avoids extra #VC event */
> > > +static unsigned long long ghcb_prot_read_msr(unsigned int msr, struct 
> > > ghcb *ghcb,
> > > +                         struct es_em_ctxt *ctxt, int *err)
> >
> > Alternatively you could return enum es_result and take xss as a 
> > parameter... six of one, half dozen of another I guess.
>
> How do we decide on this? :)
>
> and yeah, I need to s/int/enum es_result/
>
> > > +{
> > > +    unsigned long long ret = 0;
> > > +
> > > +    ghcb_set_rcx(ghcb, msr);
> > > +
> > > +    *err = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, 0, 0);
> > > +    if (*err == ES_OK)
> > > +        ret = (ghcb->save.rdx << 32) | ghcb->save.rax;
> >
> > You should check ghcb_rax_is_valid(ghcb) and ghcb_rdx_is_valid(ghcb) 
> > before using the values.
>
> Huh. v4 is coming then. Although what are the chances of *err == ES_OK and 
> !ghcb_rax_is_valid() at the same time? What if *err == ES_OK and 
> ghcb_rdx_is_valid()==true but ghcb_rax_is_valid()==false?
>
> return ((ghcb_rdx_is_valid(ghcb)?(ghcb->save.rdx << 32):0) |
>      (ghcb_rax_is_valid(ghcb)?ghcb->save.rax:0;
>
> Or I can just drop *err, invalidate ghcb before sev_es_ghcb_hv_call() and 
> only rely on (ghcb_rdx_is_valid() && ghcb_rax_is_valid)?
>
> Where should I stop with this? :)

No, you can't drop *err. The GHCB protocol specifically calls out how 
errors can be returned and how register state is returned.

In this case, sev_es_ghcb_hv_call() will check for general errors being 
returned from the hypervisor, e.g. non-zero SW_EXITINFO1[31:0] and that is 
why you need to check *err.

Then you need to validate that the hypervisor set the proper registers, 
hence the check for ghcb_rax/rdx_is_valid() (see __sev_cpuid_hv_ghcb() as 
an example).

Thanks,
Tom

> > > +
> > > +    /* Invalidate qwords for likely another following GHCB call */
> > > +    vc_ghcb_invalidate(ghcb);
> >
> > We should probably call this on entry to the function, too, right? Not 
> > sure it really matters though.
>
> The SVM_EXIT_MSR's handler in SVM/KVM only cares if RCX is valid in 
> sev_es_validate_vmgexit() and the guest's ghcb_set_rcx() does that. 
> Nothing in SVM enforces that other (unused) registers are not valid 
> though. Thanks,
>
>
> > Thanks,
> > Tom
> >
> > > +
> > > +    return ret;
> > > +}
> > > +
> > >   static int __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg)
> > >   {
> > >       u64 val;
> > > @@ -477,11 +496,11 @@ static int snp_cpuid_postprocess(struct ghcb 
> > > *ghcb, struct es_em_ctxt *ctxt,
> > >           if (leaf->subfn == 1) {
> > >               /* Get XSS value if XSAVES is enabled. */
> > >               if (leaf->eax & BIT(3)) {
> > > -                unsigned long lo, hi;
> > > +                int err = 0;
> > > -                asm volatile("rdmsr" : "=a" (lo), "=d" (hi)
> > > -                             : "c" (MSR_IA32_XSS));
> > > -                xss = (hi << 32) | lo;
> > > +                xss = ghcb_prot_read_msr(MSR_IA32_XSS, ghcb, ctxt, &err);
> > > +                if (err != ES_OK)
> > > +                    return -EINVAL;
> > >               }
> > >               /*