RE: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for oneshot interrupts

From: Liu, Rong L
Date: Tue Aug 09 2022 - 18:03:57 EST

Next message: Daniel Lezcano: "[PATCH v2 02/26] thermal/sysfs: Do not make get_trip_hyst optional"
Previous message: Lyude Paul: "Re: [PATCH v2 1/3] drm/dp_mst: add passthrough_aux to struct drm_dp_mst_port"
Next in thread: Dmytro Maluka: "Re: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for oneshot interrupts"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi Dmytro,

> -----Original Message-----
> From: Dmytro Maluka <dmy@xxxxxxxxxxxx>
> Sent: Saturday, July 30, 2022 7:35 AM
> To: Liu, Rong L <rong.l.liu@xxxxxxxxx>; Christopherson,, Sean
> <seanjc@xxxxxxxxxx>; Paolo Bonzini <pbonzini@xxxxxxxxxx>;
> kvm@xxxxxxxxxxxxxxx
> Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>; Ingo Molnar
> <mingo@xxxxxxxxxx>; Borislav Petkov <bp@xxxxxxxxx>; Dave Hansen
> <dave.hansen@xxxxxxxxxxxxxxx>; x86@xxxxxxxxxx; H. Peter Anvin
> <hpa@xxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; Eric Auger
> <eric.auger@xxxxxxxxxx>; Alex Williamson
> <alex.williamson@xxxxxxxxxx>; Zhenyu Wang
> <zhenyuw@xxxxxxxxxxxxxxx>; Tomasz Nowicki <tn@xxxxxxxxxxxx>;
> Grzegorz Jaszczyk <jaz@xxxxxxxxxxxx>; Dmitry Torokhov
> <dtor@xxxxxxxxxx>
> Subject: Re: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for
> oneshot interrupts
>
> On 7/29/22 10:48 PM, Liu, Rong L wrote:
> > Hi Dmytro,
> >
> >> -----Original Message-----
> >> From: Dmytro Maluka <dmy@xxxxxxxxxxxx>
> >> Sent: Tuesday, July 26, 2022 7:08 AM
> >> To: Liu, Rong L <rong.l.liu@xxxxxxxxx>; Christopherson,, Sean
> >> <seanjc@xxxxxxxxxx>; Paolo Bonzini <pbonzini@xxxxxxxxxx>;
> >> kvm@xxxxxxxxxxxxxxx
> >> Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>; Ingo Molnar
> >> <mingo@xxxxxxxxxx>; Borislav Petkov <bp@xxxxxxxxx>; Dave Hansen
> >> <dave.hansen@xxxxxxxxxxxxxxx>; x86@xxxxxxxxxx; H. Peter Anvin
> >> <hpa@xxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; Eric Auger
> >> <eric.auger@xxxxxxxxxx>; Alex Williamson
> >> <alex.williamson@xxxxxxxxxx>; Zhenyu Wang
> >> <zhenyuw@xxxxxxxxxxxxxxx>; Tomasz Nowicki <tn@xxxxxxxxxxxx>;
> >> Grzegorz Jaszczyk <jaz@xxxxxxxxxxxx>; Dmitry Torokhov
> >> <dtor@xxxxxxxxxx>
> >> Subject: Re: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for
> >> oneshot interrupts
> >>
> >> Hi Rong,
> >>
> >> On 7/26/22 01:44, Liu, Rong L wrote:
> >>> Hi Dmytro,
> >>>
> >>>> -----Original Message-----
> >>>> From: Dmytro Maluka <dmy@xxxxxxxxxxxx>
> >>>> Sent: Friday, July 15, 2022 8:59 AM
> >>>> To: Christopherson,, Sean <seanjc@xxxxxxxxxx>; Paolo Bonzini
> >>>> <pbonzini@xxxxxxxxxx>; kvm@xxxxxxxxxxxxxxx
> >>>> Cc: Thomas Gleixner <tglx@xxxxxxxxxxxxx>; Ingo Molnar
> >>>> <mingo@xxxxxxxxxx>; Borislav Petkov <bp@xxxxxxxxx>; Dave
> Hansen
> >>>> <dave.hansen@xxxxxxxxxxxxxxx>; x86@xxxxxxxxxx; H. Peter Anvin
> >>>> <hpa@xxxxxxxxx>; linux-kernel@xxxxxxxxxxxxxxx; Eric Auger
> >>>> <eric.auger@xxxxxxxxxx>; Alex Williamson
> >>>> <alex.williamson@xxxxxxxxxx>; Liu, Rong L <rong.l.liu@xxxxxxxxx>;
> >>>> Zhenyu Wang <zhenyuw@xxxxxxxxxxxxxxx>; Tomasz Nowicki
> >>>> <tn@xxxxxxxxxxxx>; Grzegorz Jaszczyk <jaz@xxxxxxxxxxxx>;
> Dmitry
> >>>> Torokhov <dtor@xxxxxxxxxx>; Dmytro Maluka
> <dmy@xxxxxxxxxxxx>
> >>>> Subject: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for
> >> oneshot
> >>>> interrupts
> >>>>
> >>>> The existing KVM mechanism for forwarding of level-triggered
> >> interrupts
> >>>> using resample eventfd doesn't work quite correctly in the case of
> >>>> interrupts that are handled in a Linux guest as oneshot interrupts
> >>>> (IRQF_ONESHOT). Such an interrupt is acked to the device in its
> >>>> threaded irq handler, i.e. later than it is acked to the interrupt
> >>>> controller (EOI at the end of hardirq), not earlier.
> >>>>
> >>>> Linux keeps such interrupt masked until its threaded handler
> finishes,
> >>>> to prevent the EOI from re-asserting an unacknowledged interrupt.
> >>>> However, with KVM + vfio (or whatever is listening on the
> resamplefd)
> >>>> we don't check that the interrupt is still masked in the guest at the
> >>>> moment of EOI. Resamplefd is notified regardless, so vfio
> prematurely
> >>>> unmasks the host physical IRQ, thus a new (unwanted) physical
> >> interrupt
> >>>> is generated in the host and queued for injection to the guest.
> >>>>
> >>>> The fact that the virtual IRQ is still masked doesn't prevent this new
> >>>> physical IRQ from being propagated to the guest, because:
> >>>>
> >>>> 1. It is not guaranteed that the vIRQ will remain masked by the time
> >>>> when vfio signals the trigger eventfd.
> >>>> 2. KVM marks this IRQ as pending (e.g. setting its bit in the virtual
> >>>> IRR register of IOAPIC on x86), so after the vIRQ is unmasked, this
> >>>> new pending interrupt is injected by KVM to the guest anyway.
> >>>>
> >>>> There are observed at least 2 user-visible issues caused by those
> >>>> extra erroneous pending interrupts for oneshot irq in the guest:
> >>>>
> >>>> 1. System suspend aborted due to a pending wakeup interrupt from
> >>>> ChromeOS EC (drivers/platform/chrome/cros_ec.c).
> >>>> 2. Annoying "invalid report id data" errors from ELAN0000
> touchpad
> >>>> (drivers/input/mouse/elan_i2c_core.c), flooding the guest dmesg
> >>>> every time the touchpad is touched.
> >>>>
> >>>> This patch fixes the issue on x86 by checking if the interrupt is
> >>>> unmasked when we receive irq ack (EOI) and, in case if it's masked,
> >>>> postponing resamplefd notify until the guest unmasks it.
> >>>>
> >>>> Important notes:
> >>>>
> >>>> 1. It doesn't fix the issue for other archs yet, due to some missing
> >>>> KVM functionality needed by this patch:
> >>>> - calling mask notifiers is implemented for x86 only
> >>>> - irqchip ->is_masked() is implemented for x86 only
> >>>>
> >>>> 2. It introduces an additional spinlock locking in the resample notify
> >>>> path, since we are no longer just traversing an RCU list of irqfds
> >>>> but also updating the resampler state. Hopefully this locking won't
> >>>> noticeably slow down anything for anyone.
> >>>>
> >>>
> >>> Instead of using a spinlock waiting for the unmask event, is it
> possible
> >> to call
> >>> resampler notify directly when unmask event happens, instead of
> >> calling it on
> >>> EOI?
> >>
> >> In this patch, resampler notify is already called directly when unmask
> >> happens: e.g. with IOAPIC, when the guest unmasks the interrupt by
> >> writing to IOREDTBLx register, ioapic_write_indirect() calls
> >> kvm_fire_mask_notifiers() which calls irqfd_resampler_mask() which
> >> notifies the resampler. On EOI we postpone it just by setting
> >> resampler->pending to true, not by waiting. The spinlock is needed
> >> merely to synchronize reading & updating resampler->pending and
> >> resampler->masked values between possibly concurrently running
> >> instances
> >> of irqfd_resampler_ack() and/or irqfd_resampler_mask().
> >>
> >> Thanks,
> >> Dmytro
> >>
> >
> > I mean the organization of the code. In current implementation,
> > kvm_ioapic_update_eoi_one() calls kvm_notify_acked_irq(), in your
> patch, why not
> > call kvm_notify_acked_irq() from ioapic_write_indirect() (roughly at
> the same
> > place where kvm_fire_mask_notifiers is called), instead of calling it
> from
> > kvm_ioapic_update_eoi_one, since what your intention here is to
> notify
> > vfio of the end of interrupt at the event of ioapic unmask, instead of
> > EOI?
>
> Ah ok, got your point.
>
> That was my initial approach in my PoC patch posted in [1]. But then I
> dropped it, for 2 reasons:
>
> 1. Upon feedback from Sean I realized that kvm_notify_acked_irq() is
> also doing some other important things besides notifying vfio. In
> particular, in irqfd_resampler_ack() we also de-assert the vIRQ via
> kvm_set_irq(). In case of IOAPIC it means clearing its bit in IRR
> register. If we delay that until unmasking, it means that we change
> the way how KVM emulates the interrupt controller. That would seem
> inconsistent.
>

Thanks for clarification. I totally agree that it is important to keep the way
how KVM emulates the interrupt controller.

> Also kvm_notify_acked_irq() notifies the emulated PIT timer via
> kvm_pit_ack_irq(). I haven't analyzed how exactly that PIT stuff
> works, so I'm not sure if delaying that until unmask wouldn't cause
> any unwanted effects.
>
> So the idea is to postpone eventfd_signal() only, to fix interaction
> with vfio while keeping the rest of the KVM behavior intact. Because
> the KVM job is to emulate the interrupt controller (which it already
> does correctly), not the external device which is the job of vfio*.
>

I made a mistake in my last post. I mean just to delay the notification of
vfio, but keep the rest of the code as intact as possible.

> 2. kvm_notify_acked_irq() can't be called under ioapic->lock, so in [1]
> I was unlocking ioapic->lock in ioapic_write_indirect() with a naive
> assumption that it was as safe as doing it in
> kvm_ioapic_update_eoi_one(). That was probably racy, and I hadn't
> figured out how to rework it in a race-free way.
>
> [1] https://lore.kernel.org/kvm/31420943-8c5f-125c-a5ee-
> d2fde2700083@xxxxxxxxxxxx/
>
> [*] By "vfio" I always mean "vfio or any other resamplefd user".
>
> Thanks,
> Dmytro
>
> >
> >>>
> >>>> Regarding #2, there may be an alternative solution worth
> considering:
> >>>> extend KVM irqfd (userspace) API to send mask and unmask
> >> notifications
> >>>> directly to vfio/whatever, in addition to resample notifications, to
> >>>> let vfio check the irq state on its own. There is already locking on
> >>>> vfio side (see e.g. vfio_platform_unmask()), so this way we would
> >> avoid
> >>>> introducing any additional locking. Also such mask/unmask
> >> notifications
> >>>> could be useful for other cases.
> >>>>
> >>>> Link: https://lore.kernel.org/kvm/31420943-8c5f-125c-a5ee-
> >>>> d2fde2700083@xxxxxxxxxxxx/
> >>>> Suggested-by: Sean Christopherson <seanjc@xxxxxxxxxx>
> >>>> Signed-off-by: Dmytro Maluka <dmy@xxxxxxxxxxxx>
> >>>> ---
> >>>> include/linux/kvm_irqfd.h | 14 ++++++++++++
> >>>> virt/kvm/eventfd.c | 45
> >>>> +++++++++++++++++++++++++++++++++++++++
> >>>> 2 files changed, 59 insertions(+)
> >>>>
> >>>> diff --git a/include/linux/kvm_irqfd.h b/include/linux/kvm_irqfd.h
> >>>> index dac047abdba7..01754a1abb9e 100644
> >>>> --- a/include/linux/kvm_irqfd.h
> >>>> +++ b/include/linux/kvm_irqfd.h
> >>>> @@ -19,6 +19,16 @@
> >>>> * resamplefd. All resamplers on the same gsi are de-asserted
> >>>> * together, so we don't need to track the state of each individual
> >>>> * user. We can also therefore share the same irq source ID.
> >>>> + *
> >>>> + * A special case is when the interrupt is still masked at the
> moment
> >>>> + * an irq ack is received. That likely means that the interrupt has
> >>>> + * been acknowledged to the interrupt controller but not
> >> acknowledged
> >>>> + * to the device yet, e.g. it might be a Linux guest's threaded
> >>>> + * oneshot interrupt (IRQF_ONESHOT). In this case notifying
> through
> >>>> + * resamplefd is postponed until the guest unmasks the interrupt,
> >>>> + * which is detected through the irq mask notifier. This prevents
> >>>> + * erroneous extra interrupts caused by premature re-assert of an
> >>>> + * unacknowledged interrupt by the resamplefd listener.
> >>>> */
> >>>> struct kvm_kernel_irqfd_resampler {
> >>>> struct kvm *kvm;
> >>>> @@ -28,6 +38,10 @@ struct kvm_kernel_irqfd_resampler {
> >>>> */
> >>>> struct list_head list;
> >>>> struct kvm_irq_ack_notifier notifier;
> >>>> + struct kvm_irq_mask_notifier mask_notifier;
> >>>> + bool masked;
> >>>> + bool pending;
> >>>> + spinlock_t lock;
> >>>> /*
> >>>> * Entry in list of kvm->irqfd.resampler_list. Use for sharing
> >>>> * resamplers among irqfds on the same gsi.
> >>>> diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
> >>>> index 50ddb1d1a7f0..9ff47ac33790 100644
> >>>> --- a/virt/kvm/eventfd.c
> >>>> +++ b/virt/kvm/eventfd.c
> >>>> @@ -75,6 +75,44 @@ irqfd_resampler_ack(struct
> >> kvm_irq_ack_notifier
> >>>> *kian)
> >>>> kvm_set_irq(kvm, KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID,
> >>>> resampler->notifier.gsi, 0, false);
> >>>>
> >>>> + spin_lock(&resampler->lock);
> >>>> + if (resampler->masked) {
> >>>> + resampler->pending = true;
> >>>> + spin_unlock(&resampler->lock);
> >>>> + return;
> >>>> + }
> >>>> + spin_unlock(&resampler->lock);
> >>>> +
> >>>> + idx = srcu_read_lock(&kvm->irq_srcu);
> >>>> +
> >>>> + list_for_each_entry_srcu(irqfd, &resampler->list,
> resampler_link,
> >>>> + srcu_read_lock_held(&kvm->irq_srcu))
> >>>> + eventfd_signal(irqfd->resamplefd, 1);
> >>>> +
> >>>> + srcu_read_unlock(&kvm->irq_srcu, idx);
> >>>> +}
> >>>> +
> >>>> +static void
> >>>> +irqfd_resampler_mask(struct kvm_irq_mask_notifier *kimn, bool
> >>>> masked)
> >>>> +{
> >>>> + struct kvm_kernel_irqfd_resampler *resampler;
> >>>> + struct kvm *kvm;
> >>>> + struct kvm_kernel_irqfd *irqfd;
> >>>> + int idx;
> >>>> +
> >>>> + resampler = container_of(kimn,
> >>>> + struct kvm_kernel_irqfd_resampler,
> mask_notifier);
> >>>> + kvm = resampler->kvm;
> >>>> +
> >>>> + spin_lock(&resampler->lock);
> >>>> + resampler->masked = masked;
> >>>> + if (masked || !resampler->pending) {
> >>>> + spin_unlock(&resampler->lock);
> >>>> + return;
> >>>> + }
> >>>> + resampler->pending = false;
> >>>> + spin_unlock(&resampler->lock);
> >>>> +
> >>>> idx = srcu_read_lock(&kvm->irq_srcu);
> >>>>
> >>>> list_for_each_entry_srcu(irqfd, &resampler->list,
> resampler_link,
> >>>> @@ -98,6 +136,8 @@ irqfd_resampler_shutdown(struct
> >>>> kvm_kernel_irqfd *irqfd)
> >>>> if (list_empty(&resampler->list)) {
> >>>> list_del(&resampler->link);
> >>>> kvm_unregister_irq_ack_notifier(kvm, &resampler-
> >notifier);
> >>>> + kvm_unregister_irq_mask_notifier(kvm, resampler-
> >>>>> mask_notifier.irq,
> >>>> + &resampler->mask_notifier);
> >>>> kvm_set_irq(kvm,
> KVM_IRQFD_RESAMPLE_IRQ_SOURCE_ID,
> >>>> resampler->notifier.gsi, 0, false);
> >>>> kfree(resampler);
> >>>> @@ -367,11 +407,16 @@ kvm_irqfd_assign(struct kvm *kvm,
> struct
> >>>> kvm_irqfd *args)
> >>>> INIT_LIST_HEAD(&resampler->list);
> >>>> resampler->notifier.gsi = irqfd->gsi;
> >>>> resampler->notifier.irq_acked =
> irqfd_resampler_ack;
> >>>> + resampler->mask_notifier.func =
> irqfd_resampler_mask;
> >>>> + kvm_irq_is_masked(kvm, irqfd->gsi, &resampler-
> >>>>> masked);
> >>>> + spin_lock_init(&resampler->lock);
> >>>> INIT_LIST_HEAD(&resampler->link);
> >>>>
> >>>> list_add(&resampler->link, &kvm-
> >irqfds.resampler_list);
> >>>> kvm_register_irq_ack_notifier(kvm,
> >>>> &resampler->notifier);
> >>>> + kvm_register_irq_mask_notifier(kvm, irqfd->gsi,
> >>>> + &resampler->mask_notifier);
> >>>> irqfd->resampler = resampler;
> >>>> }
> >>>>
> >>>> --
> >>>> 2.37.0.170.g444d1eabd0-goog
> >>>

Next message: Daniel Lezcano: "[PATCH v2 02/26] thermal/sysfs: Do not make get_trip_hyst optional"
Previous message: Lyude Paul: "Re: [PATCH v2 1/3] drm/dp_mst: add passthrough_aux to struct drm_dp_mst_port"
Next in thread: Dmytro Maluka: "Re: [PATCH 3/3] KVM: irqfd: Postpone resamplefd notify for oneshot interrupts"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]