Re: [PATCH 1/1] arch/x86/kvm/vmx.c: Fix external interrupts inject directly bug with guestos RFLAGS.IF=0

[Li Kaihang]

From:	Radim KrÄmÃÅ <rkrcmar@xxxxxxxxxx>
To:	Li Kaihang <li.kaihang@xxxxxxxxxx>,
Cc:	gleb@xxxxxxxxxx, pbonzini@xxxxxxxxxx, tglx@xxxxxxxxxxxxx, mingo@xxxxxxxxxx, hpa@xxxxxxxxx, x86@xxxxxxxxxx, kvm@xxxxxxxxxxxxxxx, linux-kernel@xxxxxxxxxxxxxxx
Date:	2015-01-17 äå 02:36
Subject:	Re: [PATCH 1/1] arch/x86/kvm/vmx.c: Fix external interrupts inject directly bug with guestos RFLAGS.IF=0



2015-01-16 16:07+0800, Li Kaihang:
> > GuestOS is running and handling some interrupt with RFLAGS.IF = 0 while a external interrupt coming,
> > then can lead to a vm exit,in this case,we must avoid inject this external interrupt or it will generate
> > a processor hardware exception causing virtual machine crash.
>
> What is the source of this exception?  (Is there a reproducer?)
>
> Li Kaihang: exception is produced by intel processor hardware because injecting a external interrupt vector is forbidden by intel processor when GuestOS RFLAGS.IF = 0,
>             this need to be ensured by hypervisor software according to Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3.

(Found it, happens on VMENTRY ... 26.3.1.4 Checks on Guest RIP and RFLAGS
   The IF flag (RFLAGS[bit 9]) must be 1 if the valid bit (bit 31) in the
   VM-entry interruption-information field is 1 and the interruption type
   (bits 10:8) is external interrupt.)

>             This bug has a certain probability, if code is designed to be very short between cli and sti in a guestos's interrupt processing, probability of occurrence
>             is very low, this event is like moving trap, bug is produced that guestos is running between cli and sti instruction while a external interrupt coming, it
>             may be verified by constructing a special guestos interrupt code. General OS running on kvm vm has also probability to hit this bug.

Is APICv enabled?  (Does togging it avoid the bug?)

Li kaihang: APICv doesn't completely solve the problem. Firstly, interrupts delivered by APICv must be recognized by virtual apic controller. Secondly, APIC sw is
            enabled. We know that some interrupts are not recognized by virtual apic controller(those may be what vm need or not),and some hardware interrupts emuluated
            by QEMU are needed by virtual bios before guestos starts, at this stage APIC sw is disabled. Thirdly, APICv is only supported by intel ivybridge or
            processors after it, it is not before that.

> > Step 1:
> >      a ext intr gen a vm_exit --> vmx_complete_interrupts --> __vmx_complete_interrupts --> case INTR_TYPE_EXT_INR: kvm_queue_interrupt(vcpu, vector, type == INTR_TYPE_SOFT_INTR);
> >
> > Step 2:
> >      kvm_x86_ops->handle_external_intr(vcpu);
> > for the above steps, step 4 and 5 will be a processor hardware exception if step1 happen while guestos RFLAGS.IF = 0, that is to say, guestos interrupt is disabled.
> > So we should add a logic to judge in step 1 whether a external interrupt need to be pended then inject directly, in the process, we don't need to worry about
> > this external interrupt lost because the next Step 2 will handle and choose a best chance to inject it by virtual interrupt controller.
>
> Can you explain the relation between vectored events (Step 1) and
> external interrupts (Step 2)?
> (The bug happens when external interrupt arrives during event delivery?)
>
> Li Kaihang: a external interrupt to running vm can trigger a vm_exit event handled in step 1, then this interrupt vector can be processed in step2
>             kvm_x86_ops->handle_external_intr(vcpu) and this function can jump to HOSTOS IDT to complete external interrupt handling,external interrupt handler in HOSTOS
>             IDT may inject the external interrupt into virtual interrupt controller if it has been registered to be needed by virtual machine.

Proposed non-accidental relation between first two steps confuses me ...

1) External interrupt in Step 1 should be from incomplete event delivery
   to the guest,
2) External interrupt in Step 2 from an interrupt for the host.
3) Interrupts should be unrelated, because if not, then we either
   deliver host's interrupts directly to the guest, or the other way
   around.  (Both cases are buggy, regardless of CLI.)
   - What is stored in "VM-exit interruption information" (Step 2) and
     "IDT-vectoring information" (Step 1)?

=> Interrupts in first two steps are independent and we have interrupted
   external interrupt delivery to the guest.

Can you provide a KVM trace of what is happening before the crash?
(Or just point out where I made a mistake.)

Li kaihang: There exist two cases:
            First, external interrupts registered by vm, such as timer,shared devices,pci passthrough devices interrupts, are handled by host idt to notify one virtual
            apic interrupt controller which manages all general interrupts owned by a vm, inject_pending_event will complete a interrupt vector injecting, if virtual
            interrupt delivery on APICv not supported, after virtual apic interrupt controller updating and computing. This code is looked at inject_pending_event
            fragment below:

            else if (kvm_cpu_has_injectable_intr(vcpu)) {              ==>  find a injecting vector from virtual apic interrupt controller
		/*
		 * Because interrupts can be injected asynchronously, we are
		 * calling check_nested_events again here to avoid a race condition.
		 * See https://lkml.org/lkml/2014/7/2/60 for discussion about this
		 * proposal and current concerns.  Perhaps we should be setting
		 * KVM_REQ_EVENT only on certain events and not unconditionally?
		 */
		if (is_guest_mode(vcpu) && kvm_x86_ops->check_nested_events) {
			r = kvm_x86_ops->check_nested_events(vcpu, req_int_win);
			if (r != 0)
				return r;
		}
		if (kvm_x86_ops->interrupt_allowed(vcpu)) {         ==>  inject the vector if current guestos RFLAGS.IF = 1, else do not.
			kvm_queue_interrupt(vcpu, kvm_cpu_get_interrupt(vcpu),
					    false);
			kvm_x86_ops->set_irq(vcpu);
		}
	    }

           Second, external interrupts needed by vm are not registered in host idt (this may be a wrong or some older interrupts are not recognized by host idt virtual
           apic interrupt controller), in this case, external interrupts handling have no relations between Step 1 and Step 2,but injecting a external interrupt vector
           also must judge current guestos RFLAGS.IF state like above.

           And this bug will be happened under certain probability condition,so it is difficult to use a very accurate model to reproduce that, we may construct a
           special interrupt code to test it. I show a possible steps:

           1. Emulate a A interrupt vector to inject to guestos:

           2. A interrupt handler test part in guestos:

              cli

              do enough waits or delays to not enter next step sti for RFLAGS.IF = 0

              sti

           3. Send a B interrupt to guestos in runing A interrupt handler with RFLAGS.IF = 0 by a hardware device or another way

           4. Bug will appear if all goes well

Thank you.

>             The Bug has never happened in step 1 and 2, but vcpu->arch.interrupt.pending is set in step 1, if this pending should not be injected, it also will be passed
>             to step4 to complete the dangerous external interrupt injecting. Please see the above answer about what is "pending should not be injected"? Our solution
>             is that clearing invalid external interrupt pending to prevent error inject pass by adding a logical judge in step 1.

(I agree that we shouldn't inject events that fail VMENTRY,
 I'm just not sure that this solution is correct.)

Li kaihang: Your consideration is reasonable, in fact we may add a judge in step 4, below, but there are not only pendings on external interrupt here,also including
            software interrupt, and it is not safe in somewhere else, so we consider add a judge in step 1.

            Step 4:
            if (vcpu->arch.interrupt.pending) {
 		 		 kvm_x86_ops->set_irq(vcpu);
 		 		 return 0;
 		 }

> Why isn't the delivered event lost?
> (It should be different from the external interrupt.)
>
> Li Kaihang: please refer to the above answer, a external interrupt in step1 only can get to case INTR_TYPE_EXT_INR branch in patch code, so it should not affect other

(It is the same type of interrupt, but with a different meaning ...
 You don't save the incomplete event delivery in Step 1, so it has to be
 recorded somewhere, otherwise it is lost -- where is it?)

 Li kaihang: To this problem, we should first establish a basic rule in hypervisor design that all interrupts needed by vm must come from virtual interrupt controller
             just like real hardware interrupt controller behavior, then injecting it or not to a vm is decided by whether the external interrupt are recognized by
             virtual interrupt controller(it may be invalid one or not defined to current vm if not recognized) to be notified by host idt handler in step2, even if
             there are different meanings with same type interrupt, those are recognized by virtual interrupt controller in step2. Unfortunately, virtual
             interrupt controller can't do that, some interrupts needed by vm may not be still recognized by virtual interrupt controller so we must pend those interrupt
             flag in step 1 then inject it in step 4 in case of loss, these interrupt bypass virtual interrupt controller checking and bug is hidden in the process.

>             type events delivering, but there is another possibility that the external interrupt needed by running vm is not registered in hostos idt handler chain,of
>             course, this situation is another problem, even so it is dangerous action to inject the external interrupt directly if not judge current guestos RFLAGS.IF
>             state.

(This would be a major blunder -- does it happen?)

Li kaihang: Same as above, it must be guaranteed in virtual interrupt controller design, a external interrupt not recognized by virtual interrupt controller should not
            be injected to guestos. Just like if no virtualization, Should a interrupt not recognized by hard interrupt controller be responded by os?
            We consider solve vm injecting exception with RFLAGS.IF = 0 at first.

> > Signed-off-by: Li kaihang <li.kaihang@xxxxxxxxxx>
> > ---
> > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
> > index d4c58d8..e8311ee 100644
> > --- a/arch/x86/kvm/vmx.c
> > +++ b/arch/x86/kvm/vmx.c
> > @@ -7711,10 +7711,26 @@ static void __vmx_complete_interrupts(struct kvm_vcpu *vcpu,
[...]
> > +       case INTR_TYPE_EXT_INTR:
> > +               /*
> > +               * GuestOS is running and handling some interrupt with
> > +               * RFLAGS.IF = 0 while a external interrupt coming,
> > +               * then can lead a vm exit getting here,in this case,
> > +               * we must avoid inject this external interrupt or it will
> > +               * generate a processor hardware exception causing vm crash.
> > +               */
> > +               if (kvm_x86_ops->interrupt_allowed(vcpu))

(I missed this on the first reading ...
 in vmx.c, it is better to use vmx_interrupt_allowed() directly.)

Li kaihang: OK--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.
N‹§²æ¸›yú²X¬¶ÇvØ–)Þ{.nlj·¥Š{±‘êX§¶›¡Ü}©ž²ÆzÚj:+v‰¨¾«‘êZ+€Êzf£¢·hšˆ§~†­†Ûÿû®w¥¢¸?™¨è&¢)ßf”ùy§m…á«a¶Úÿ0¶ìå