RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding

[Wu, Feng]

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx]
> Sent: Tuesday, June 16, 2015 12:45 AM
> To: Eric Auger
> Cc: Avi Kivity; Wu, Feng; kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> pbonzini@xxxxxxxxxx; mtosatti@xxxxxxxxxx
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> 
> On Mon, 2015-06-15 at 18:17 +0200, Eric Auger wrote:
> > Hi Alex, all,
> > On 06/12/2015 09:03 PM, Alex Williamson wrote:
> > > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote:
> > >> On 06/12/2015 06:41 PM, Alex Williamson wrote:
> > >>> On Fri, 2015-06-12 at 00:23 +0000, Wu, Feng wrote:
> > >>>>> -----Original Message-----
> > >>>>> From: Avi Kivity [mailto:avi.kivity@xxxxxxxxx]
> > >>>>> Sent: Friday, June 12, 2015 3:59 AM
> > >>>>> To: Wu, Feng; kvm@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx
> > >>>>> Cc: pbonzini@xxxxxxxxxx; mtosatti@xxxxxxxxxx;
> > >>>>> alex.williamson@xxxxxxxxxx; eric.auger@xxxxxxxxxx
> > >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> > >>>>>
> > >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote:
> > >>>>>> From: Eric Auger <eric.auger@xxxxxxxxxx>
> > >>>>>>
> > >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE
> group
> > >>>>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
> > >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be
> able
> > >>>>>> to set a VFIO device IRQ as forwarded or not forwarded.
> > >>>>>> the command takes as argument a handle to a new struct named
> > >>>>>> kvm_vfio_dev_irq.
> > >>>>> Is there no way to do this automatically?  After all, vfio knows that a
> > >>>>> device interrupt is forwarded to some eventfd, and kvm knows that
> some
> > >>>>> eventfd is forwarded to a guest interrupt.  If they compare notes
> > >>>>> through a central registry, they can figure out that the interrupt needs
> > >>>>> to be forwarded.
> > >>>> Oh, just like Eric mentioned in his reply, this description is out of context
> of
> > >>>> this series, I will remove them in the next version.
> > >>>
> > >>> I suspect Avi's question was more general.  While forward/unforward is
> > >>> out of context for this series, it's very similar in nature to
> > >>> enabling/disabling posted interrupts.  So I think the question remains
> > >>> whether we really need userspace to participate in creating this
> > >>> shortcut or if kvm and vfio can some how orchestrate figuring it out
> > >>> automatically.
> > >>>
> > >>> Personally I don't know how we could do it automatically.  We've always
> > >>> relied on userspace to independently setup vfio and kvm such that
> > >>> neither have any idea that the other is there and update each side
> > >>> independently when anything changes.  So it seems consistent to
> continue
> > >>> that here.  It doesn't seem like there's much to gain performance-wise
> > >>> either, updates should be a relatively rare event I'd expect.
> > >>>
> > >>> There's really no metadata associated with an eventfd, so "comparing
> > >>> notes" automatically might imply some central registration entity.  That
> > >>> immediately sounds like a much more complex solution, but maybe Avi
> has
> > >>> some ideas to manage it.  Thanks,
> > >>>
> > >>
> > >> The idea is to have a central registry maintained by a posted interrupts
> > >> manager.  Both vfio and kvm pass the filp (along with extra information)
> > >> to the posted interrupts manager, which, when it detects a filp match,
> > >> tells each of them what to do.
> > >>
> > >> The advantages are:
> > >> - old userspace gains the optimization without change
> > >> - a userspace API is more expensive to maintain than internal kernel
> > >> interfaces (CVEs, documentation, maintaining backwards compatibility)
> > >> - if you can do it without a new interface, this indicates that all the
> > >> information in the new interface is redundant.  That means you have to
> > >> check it for consistency with the existing information, so it's extra
> > >> work (likely, it's exactly what the posted interrupt manager would be
> > >> doing anyway).
> > >
> > > Yep, those all sound like good things and I believe that's similar in
> > > design to the way we had originally discussed this interaction at
> > > LPC/KVM Forum several years ago.  I'd be in favor of that approach.
> >
> > I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO
> > IRQ forward control" series? Or is that "central registry maintained by
> > a posted interrupts manager" something more specific to x86?
> 
> I'd think we'd want it for any sort of offload and supporting both
> posted-interrupts and irq-forwarding would be a good validation.  I
> imagine there would be registration/de-registration callbacks separate
> for interrupt producers vs interrupt consumers.  Each registration
> function would likely provide a struct of callbacks, probably similar to
> the get_symbol callbacks proposed for the kvm-vfio device on the IRQ
> producer side.  The eventfd would be the token that the manager would
> use to match producers and consumers.  The hard part is probably
> figuring out what information to retrieve from the producer and provide
> to the consumer in a generic way between pci and platform, but as an
> internal interface, it's not a big deal if we screw it up a few times to
> start.  Thanks,

On posted-interrupts side, the main purpose of the new APIs is to update
the IRTE when guest changes vMSI/vMSIx configuration. Alex, do you have
any detailed ideas for the new solution to achieve this purpose? It should
be helpful if you can share some!

Thanks,
Feng

> 
> Alex

N‹§²æ¸›yú²X¬¶ÇvØ–)Þ{.nlj·¥Š{±‘êX§¶›¡Ü}©ž²ÆzÚj:+v‰¨¾«‘êZ+€Êzf£¢·hšˆ§~†­†Ûÿû®w¥¢¸?™¨è&¢)ßf”ùy§m…á«a¶Úÿ0¶ìå