Re: [PATCH 1/2] KVM: VMX: Move IRQ invocation to assembly subroutine

[Uros Bizjak]

On Mon, Sep 14, 2020 at 11:07 PM Sean Christopherson
<sean.j.christopherson@xxxxxxxxx> wrote:
>
> On Mon, Sep 14, 2020 at 03:40:24PM -0500, Josh Poimboeuf wrote:
> > On Mon, Sep 14, 2020 at 12:56:33PM -0700, Sean Christopherson wrote:
> > > Move the asm blob that invokes the appropriate IRQ handler after VM-Exit
> > > into a proper subroutine.  Slightly rework the blob so that it plays
> > > nice with objtool without any additional hints (existing hints aren't
> > > able to handle returning with a seemingly modified stack size).
> > >
> > > Suggested-by: Josh Poimboeuf <jpoimboe@xxxxxxxxxx>
> > > Cc: Uros Bizjak <ubizjak@xxxxxxxxx>
> > > Signed-off-by: Sean Christopherson <sean.j.christopherson@xxxxxxxxx>
> > > ---
> > >  arch/x86/kvm/vmx/vmenter.S | 28 ++++++++++++++++++++++++++++
> > >  arch/x86/kvm/vmx/vmx.c     | 33 +++------------------------------
> > >  2 files changed, 31 insertions(+), 30 deletions(-)
> > >
> > > diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S
> > > index 799db084a336..baec1e0fefc5 100644
> > > --- a/arch/x86/kvm/vmx/vmenter.S
> > > +++ b/arch/x86/kvm/vmx/vmenter.S
> > > @@ -4,6 +4,7 @@
> > >  #include <asm/bitsperlong.h>
> > >  #include <asm/kvm_vcpu_regs.h>
> > >  #include <asm/nospec-branch.h>
> > > +#include <asm/segment.h>
> > >
> > >  #define WORD_SIZE (BITS_PER_LONG / 8)
> > >
> > > @@ -294,3 +295,30 @@ SYM_FUNC_START(vmread_error_trampoline)
> > >
> > >     ret
> > >  SYM_FUNC_END(vmread_error_trampoline)
> > > +
> > > +SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff)
> > > +   /*
> > > +    * Unconditionally create a stack frame.  RSP needs to be aligned for
> > > +    * x86-64, getting the correct RSP on the stack (for x86-64) would take
> > > +    * two instructions anyways, and it helps make objtool happy (see below).
> > > +    */
> > > +   push %_ASM_BP
> > > +   mov %rsp, %_ASM_BP
> >
> > RSP needs to be aligned to what?  How would this align the stack, other
> > than by accident?
>
> Ah, yeah, that's lacking info.
>
> 16-byte aligned to correctly mimic CPU behavior when vectoring an IRQ/NMI.
> When not changing stack, the CPU aligns RSP before pushing the frame.
>
> The above shenanigans work because the x86-64 ABI also requires RSP to be
> 16-byte aligned prior to CALL.  RSP is thus 8-byte aligned due to CALL
> pushing the return IP, and so creating the stack frame by pushing RBP makes
> it 16-byte aliagned again.

IIRC, the kernel violates x86_64 ABI and aligns RSP to 8 bytes prior
to CALL. Please note -mpreferred-stack-boundary=3 in the compile
flags.

Uros.