RE: [RFC v2 0/9] KVM-VFIO IRQ forward control

[Wu, Feng]

> -----Original Message-----
> From: kvm-owner@xxxxxxxxxxxxxxx [mailto:kvm-owner@xxxxxxxxxxxxxxx] On
> Behalf Of Eric Auger
> Sent: Monday, November 17, 2014 9:42 PM
> To: Wu, Feng; Alex Williamson; Christoffer Dall
> Cc: eric.auger@xxxxxx; marc.zyngier@xxxxxxx;
> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx;
> kvm@xxxxxxxxxxxxxxx; joel.schopp@xxxxxxx; kim.phillips@xxxxxxxxxxxxx;
> paulus@xxxxxxxxx; gleb@xxxxxxxxxx; pbonzini@xxxxxxxxxx;
> linux-kernel@xxxxxxxxxxxxxxx; patches@xxxxxxxxxx; will.deacon@xxxxxxx;
> a.motakis@xxxxxxxxxxxxxxxxxxxxxx; a.rigo@xxxxxxxxxxxxxxxxxxxxxx;
> john.liuli@xxxxxxxxxx
> Subject: Re: [RFC v2 0/9] KVM-VFIO IRQ forward control
> 
> Hi Feng,
> 
> I will submit a PATCH v3 release end of this week.
> 
> Best Regards
> 
> Eric

Thanks for the update, Eric!

Thanks,
Feng

> 
> On 11/17/2014 12:25 PM, Wu, Feng wrote:
> >
> >
> >> -----Original Message-----
> >> From: linux-kernel-owner@xxxxxxxxxxxxxxx
> >> [mailto:linux-kernel-owner@xxxxxxxxxxxxxxx] On Behalf Of Alex Williamson
> >> Sent: Thursday, September 11, 2014 1:10 PM
> >> To: Christoffer Dall
> >> Cc: Eric Auger; eric.auger@xxxxxx; marc.zyngier@xxxxxxx;
> >> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx;
> >> kvm@xxxxxxxxxxxxxxx; joel.schopp@xxxxxxx; kim.phillips@xxxxxxxxxxxxx;
> >> paulus@xxxxxxxxx; gleb@xxxxxxxxxx; pbonzini@xxxxxxxxxx;
> >> linux-kernel@xxxxxxxxxxxxxxx; patches@xxxxxxxxxx; will.deacon@xxxxxxx;
> >> a.motakis@xxxxxxxxxxxxxxxxxxxxxx; a.rigo@xxxxxxxxxxxxxxxxxxxxxx;
> >> john.liuli@xxxxxxxxxx
> >> Subject: Re: [RFC v2 0/9] KVM-VFIO IRQ forward control
> >>
> >> On Thu, 2014-09-11 at 05:10 +0200, Christoffer Dall wrote:
> >>> On Tue, Sep 02, 2014 at 03:05:41PM -0600, Alex Williamson wrote:
> >>>> On Mon, 2014-09-01 at 14:52 +0200, Eric Auger wrote:
> >>>>> This RFC proposes an integration of "ARM: Forwarding physical
> >>>>> interrupts to a guest VM" (http://lwn.net/Articles/603514/) in
> >>>>> KVM.
> >>>>>
> >>>>> It enables to transform a VFIO platform driver IRQ into a forwarded
> >>>>> IRQ. The direct benefit is that, for a level sensitive IRQ, a VM
> >>>>> switch can be avoided on guest virtual IRQ completion. Before this
> >>>>> patch, a maintenance IRQ was triggered on the virtual IRQ completion.
> >>>>>
> >>>>> When the IRQ is forwarded, the VFIO platform driver does not need to
> >>>>> disable the IRQ anymore. Indeed when returning from the IRQ handler
> >>>>> the IRQ is not deactivated. Only its priority is lowered. This means
> >>>>> the same IRQ cannot hit before the guest completes the virtual IRQ
> >>>>> and the GIC automatically deactivates the corresponding physical IRQ.
> >>>>>
> >>>>> Besides, the injection still is based on irqfd triggering. The only
> >>>>> impact on irqfd process is resamplefd is not called anymore on
> >>>>> virtual IRQ completion since this latter becomes "transparent".
> >>>>>
> >>>>> The current integration is based on an extension of the KVM-VFIO
> >>>>> device, previously used by KVM to interact with VFIO groups. The
> >>>>> patch serie now enables KVM to directly interact with a VFIO
> >>>>> platform device. The VFIO external API was extended for that purpose.
> >>>>>
> >>>>> Th KVM-VFIO device can get/put the vfio platform device, check its
> >>>>> integrity and type, get the IRQ number associated to an IRQ index.
> >>>>>
> >>>>> The IRQ forward programming is architecture specific (virtual interrupt
> >>>>> controller programming basically). However the whole infrastructure is
> >>>>> kept generic.
> >>>>>
> >>>>> from a user point of view, the functionality is provided through new
> >>>>> KVM-VFIO device commands,
> >> KVM_DEV_VFIO_DEVICE_(UN)FORWARD_IRQ
> >>>>> and the capability can be checked with KVM_HAS_DEVICE_ATTR.
> >>>>> Assignment can only be changed when the physical IRQ is not active.
> >>>>> It is the responsability of the user to do this check.
> >>>>>
> >>>>> This patch serie has the following dependencies:
> >>>>> - "ARM: Forwarding physical interrupts to a guest VM"
> >>>>>   (http://lwn.net/Articles/603514/) in
> >>>>> - [PATCH v3] irqfd for ARM
> >>>>> - and obviously the VFIO platform driver serie:
> >>>>>   [RFC PATCH v6 00/20] VFIO support for platform devices on ARM
> >>>>>
> https://www.mail-archive.com/kvm@xxxxxxxxxxxxxxx/msg103247.html
> >>>>>
> >>>>> Integrated pieces can be found at
> >>>>> ssh://git.linaro.org/people/eric.auger/linux.git
> >>>>> on branch 3.17rc3_irqfd_forward_integ_v2
> >>>>>
> >>>>> This was was tested on Calxeda Midway, assigning the xgmac main IRQ.
> >>>>>
> >>>>> v1 -> v2:
> >>>>> - forward control is moved from architecture specific file into generic
> >>>>>   vfio.c module.
> >>>>>   only kvm_arch_set_fwd_state remains architecture specific
> >>>>> - integrate Kim's patch which enables KVM-VFIO for ARM
> >>>>> - fix vgic state bypass in vgic_queue_hwirq
> >>>>> - struct kvm_arch_forwarded_irq moved from
> >> arch/arm/include/uapi/asm/kvm.h
> >>>>>   to include/uapi/linux/kvm.h
> >>>>>   also irq_index renamed into index and guest_irq renamed into gsi
> >>>>> - ASSIGN/DEASSIGN renamed into FORWARD/UNFORWARD
> >>>>> - vfio_external_get_base_device renamed into
> vfio_external_base_device
> >>>>> - vfio_external_get_type removed
> >>>>> - kvm_vfio_external_get_base_device renamed into
> >> kvm_vfio_external_base_device
> >>>>> - __KVM_HAVE_ARCH_KVM_VFIO renamed into
> >> __KVM_HAVE_ARCH_KVM_VFIO_FORWARD
> >>>>>
> >>>>> Eric Auger (8):
> >>>>>   KVM: ARM: VGIC: fix multiple injection of level sensitive forwarded
> >>>>>     IRQ
> >>>>>   KVM: ARM: VGIC: add forwarded irq rbtree lock
> >>>>>   VFIO: platform: handler tests whether the IRQ is forwarded
> >>>>>   KVM: KVM-VFIO: update user API to program forwarded IRQ
> >>>>>   VFIO: Extend external user API
> >>>>>   KVM: KVM-VFIO: add new VFIO external API hooks
> >>>>>   KVM: KVM-VFIO: generic KVM_DEV_VFIO_DEVICE command and IRQ
> >> forwarding
> >>>>>     control
> >>>>>   KVM: KVM-VFIO: ARM forwarding control
> >>>>>
> >>>>> Kim Phillips (1):
> >>>>>   ARM: KVM: Enable the KVM-VFIO device
> >>>>>
> >>>>>  Documentation/virtual/kvm/devices/vfio.txt |  26 ++
> >>>>>  arch/arm/include/asm/kvm_host.h            |   7 +
> >>>>>  arch/arm/kvm/Kconfig                       |   1 +
> >>>>>  arch/arm/kvm/Makefile                      |   4 +-
> >>>>>  arch/arm/kvm/kvm_vfio_arm.c                |  85 +++++
> >>>>>  drivers/vfio/platform/vfio_platform_irq.c  |   7 +-
> >>>>>  drivers/vfio/vfio.c                        |  24 ++
> >>>>>  include/kvm/arm_vgic.h                     |   1 +
> >>>>>  include/linux/kvm_host.h                   |  27 ++
> >>>>>  include/linux/vfio.h                       |   3 +
> >>>>>  include/uapi/linux/kvm.h                   |   9 +
> >>>>>  virt/kvm/arm/vgic.c                        |  59 +++-
> >>>>>  virt/kvm/vfio.c                            | 497
> >> ++++++++++++++++++++++++++++-
> >>>>>  13 files changed, 733 insertions(+), 17 deletions(-)
> >>>>>  create mode 100644 arch/arm/kvm/kvm_vfio_arm.c
> >>>>>
> >>>>
> >>>> Have we ventured too far in the other direction?  I suppose what I was
> >>>> hoping to see was something more like:
> >>>>
> >>>> 	case KVM_DEV_VFIO_DEVICE_FORWARD_IRQ:{
> >>>>
> >>>> 		/* get vfio_device */
> >>>>
> >>>> 		/* get mutex */
> >>>>
> >>>> 		/* verify device+irq isn't already forwarded */
> >>>>
> >>>> 		/* allocate device/forwarded irq */
> >>>>
> >>>> 		/* get struct device */
> >>>>
> >>>> 		/* callout to arch code passing struct device, gsi, ... */
> >>>>
> >>>> 		/* if success, add to kv, else free and error */
> >>>>
> >>>> 		/* mutex unlock */
> >>>> 	}
> >>>
> >>> I think that's essentially what this patch set is trying to do, but
> >>> there are just too many complicated intertwining cases right now that
> >>> makes the code hard to read.
> >>>
> >>>>
> >>>> Exposing the internal mutex out to arch code, as in v1, was an
> >>>> indication that we were pushing too much out to arch code, but including
> >>>> platform_device.h into virt/kvm/vfio.c tells me we're still not
> >>>> abstracting at the right point.  Thanks,
> >>>>
> >>> I raised my eyebrows over the platform device bus thingy here as well,
> >>> but on the other hand, there's nothing ARM-specific about referring to
> >>> the platform device bus.
> >>>
> >>> I think perhaps it just has to be made more clear that the generic code
> >>> deals with translating the device resources in the necessary way, and
> >>> currently it only supports vfio-platform devices?
> >>
> >> Ok, you're probably right, looking at it again it is closer than I
> >> thought.  At the same time, the use of platform device in
> >> virt/kvm/vfio.c is pointless and can easily be pushed out to the arch
> >> code as just another error return case.  vfio.c doesn't need to be aware
> >> of hwirq.  The rest of the code is just overly complicated, with three
> >> different cleanup functions and validation function bloat.  Thanks,
> >>
> >> Alex
> >
> >
> > Hi Alex, Could you please tell what is the current status of this patch set.
> > As you mentioned in another thread, something(such as,
> kvm_vfio_device_get_external_user(), etc.)
> > in this patch set can be leveraged for VT-d Posted-interrtups.
> >
> > Thanks,
> > Feng
> >
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> Please read the FAQ at  http://www.tux.org/lkml/
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/