Re: [Question] int3 instruction generates a #UD in SEV VM

[Tom Lendacky]

On 8/2/23 09:33, Tom Lendacky wrote:
> On 8/2/23 09:25, Tom Lendacky wrote:
> > On 8/2/23 09:01, Sean Christopherson wrote:
> > > On Wed, Aug 02, 2023, Wu Zongyo wrote:
> > > > On Mon, Jul 31, 2023 at 11:45:29PM +0800, wuzongyong wrote:
> > > > > On 2023/7/31 23:03, Tom Lendacky wrote:
> > > > > > On 7/31/23 09:30, Sean Christopherson wrote:
> > > > > > > On Sat, Jul 29, 2023, wuzongyong wrote:
> > > > > > > > Hi,
> > > > > > > > I am writing a firmware in Rust to support SEV based on project 
> > > > > > > > td-shim[1].
> > > > > > > > But when I create a SEV VM (just SEV, no SEV-ES and no SEV-SNP) 
> > > > > > > > with the firmware,
> > > > > > > > the linux kernel crashed because the int3 instruction in 
> > > > > > > > int3_selftest() cause a
> > > > > > > > #UD.
> > > > > > >
> > > > > > > ...
> > > > > > >
> > > > > > > > BTW, if a create a normal VM without SEV by qemu & OVMF, the int3 
> > > > > > > > instruction always generates a
> > > > > > > > #BP.
> > > > > > > > So I am confused now about the behaviour of int3 instruction, 
> > > > > > > > could anyone help to explain the behaviour?
> > > > > > > > Any suggestion is appreciated!
> > > > > > >
> > > > > > > Have you tried my suggestions from the other thread[*]?
> > > > > Firstly, I'm sorry for sending muliple mails with the same content. I 
> > > > > thought the mails I sent previously
> > > > > didn't be sent successfully.
> > > > > And let's talk the problem here.
> > > > > > >     : > > I'm curious how this happend. I cannot find any condition 
> > > > > > > that would
> > > > > > >     : > > cause the int3 instruction generate a #UD according to 
> > > > > > > the AMD's spec.
> > > > > > >     :
> > > > > > >     : One possibility is that the value from memory that gets 
> > > > > > > executed diverges from the
> > > > > > >     : value that is read out be the #UD handler, e.g. due to 
> > > > > > > patching (doesn't seem to
> > > > > > >     : be the case in this test), stale cache/tlb entries, etc.
> > > > > > >     :
> > > > > > >     : > > BTW, it worked nomarlly with qemu and ovmf.
> > > > > > >     : >
> > > > > > >     : > Does this happen every time you boot the guest with your 
> > > > > > > firmware? What
> > > > > > >     : > processor are you running on?
> > > > > > >     :
> > > > > Yes, every time.
> > > > > The processor I used is EPYC 7T83.
> > > > > > >     : And have you ruled out KVM as the culprit?  I.e. verified 
> > > > > > > that KVM is NOT injecting
> > > > > > >     : a #UD.  That obviously shouldn't happen, but it should be 
> > > > > > > easy to check via KVM
> > > > > > >     : tracepoints.
> > > > > >
> > > > > > I have a feeling that KVM is injecting the #UD, but it will take 
> > > > > > instrumenting KVM to see which path the #UD is being injected from.
> > > > > >
> > > > > > Wu Zongyo, can you add some instrumentation to figure that out if 
> > > > > > the trace points towards KVM injecting the #UD?
> > > > > Ok, I will try to do that.
> > > > You're right. The #UD is injected by KVM.
> > > >
> > > > The path I found is:
> > > >      svm_vcpu_run
> > > >          svm_complete_interrupts
> > > >         kvm_requeue_exception // vector = 3
> > > >             kvm_make_request
> > > >
> > > >      vcpu_enter_guest
> > > >          kvm_check_and_inject_events
> > > >         svm_inject_exception
> > > >             svm_update_soft_interrupt_rip
> > > >             __svm_skip_emulated_instruction
> > > >                 x86_emulate_instruction
> > > >                 svm_can_emulate_instruction
> > > >                     kvm_queue_exception(vcpu, UD_VECTOR)
> > > >
> > > > Does this mean a #PF intercept occur when the guest try to deliver a
> > > > #BP through the IDT? But why?
> > >
> > > I doubt it's a #PF.  A #NPF is much more likely, though it could be 
> > > something
> > > else entirely, but I'm pretty sure that would require bugs in both the 
> > > host and
> > > guest.
> > >
> > > What is the last exit recorded by trace_kvm_exit() before the #UD is 
> > > injected?
> >
> > I'm guessing it was a #NPF, too. Could it be related to the changes that
> > went in around svm_update_soft_interrupt_rip()?
> >
> > 6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the 
> > instruction")
>
> Sorry, that should have been:
>
> 7e5b5ef8dca3 ("KVM: SVM: Re-inject INTn instead of retrying the insn on 
> "failure"")

Doh! I was right the first time... sigh

6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")

Thanks,
Tom

> > Before this the !nrips check would prevent the call into
> > svm_skip_emulated_instruction(). But now, there is a call to:
> >
> >    svm_update_soft_interrupt_rip()
> >      __svm_skip_emulated_instruction()
> >        kvm_emulate_instruction()
> >          x86_emulate_instruction() (passed a NULL insn pointer)
> >            kvm_can_emulate_insn() (passed a NULL insn pointer)
> >              svm_can_emulate_instruction() (passed NULL insn pointer)
> >
> > Because it is an SEV guest, it ends up in the "if (unlikely(!insn))" path
> > and injects the #UD.
> >
> > Thanks,
> > Tom