Re: [PATCH V5 00/11] vfio/pci: Support dynamic allocation of MSI-X interrupts

[YangHang Liu]

Running regression tests in the following test matrix:

    i40e PF + INTx interrupt + RHEL9 guest -- PASS
    bnx2x PF + MSI-X + RHEL9 guest -- PASS
    iavf VF + MSI-X + Win2019 guest  -- PASS
    mlx5_core VF + MSI-X  + Win2022 guest -- PASS
    ixgbe PF + INTx + RHEL9 guest -- PASS
    i40e PF + MSIX + Win2019 guest -- PASS
    qede VF + MSIX + RHEL9 guest -- PASS

Tested-by: YangHang Liu <yanghliu@xxxxxxxxxx>



On Wed, May 24, 2023 at 6:46 AM Alex Williamson
<alex.williamson@xxxxxxxxxx> wrote:
>
> On Thu, 11 May 2023 08:44:27 -0700
> Reinette Chatre <reinette.chatre@xxxxxxxxx> wrote:
>
> > Changes since V4:
> > - V4: https://lore.kernel.org/lkml/cover.1682615447.git.reinette.chatre@xxxxxxxxx/
> > - Add Kevin's Reviewed-by tag as applicable.
> > - Treat non-existing INTx interrupt context as kernel bug with WARN. This
> >   exposed an issue in the scenario where INTx mask/unmask may occur without
> >   INTx enabled. This is fixed by obtaining the interrupt context later
> >   (right before use) within impacted functions: vfio_pci_intx_mask() and
> >   vfio_pci_intx_unmask_handler(). (Kevin)
> > - Treat pci_irq_vector() returning '0' for a MSI/MSI-X interrupt as a kernel
> >   bug via a WARN instead of ignoring this value. (Kevin)
> > - Improve accuracy of comments. (Kevin)
> > - Please refer to individual patches for local changes.
> >
> > Changes since V3:
> > - V3: https://lore.kernel.org/lkml/cover.1681837892.git.reinette.chatre@xxxxxxxxx/
> > - Be considerate about layout and size with changes to
> >   struct vfio_pci_core_device. Keep flags together and transition all to
> >   use bitfields. (Alex and Jason)
> > - Do not free dynamically allocated interrupts on error path. (Alex)
> > - Please refer to individual patches for localized changes.
> >
> > Changes since V2:
> > - V2: https://lore.kernel.org/lkml/cover.1680038771.git.reinette.chatre@xxxxxxxxx/
> > - During testing of V2 "kernel test robot" reported issues resulting from
> >   include/linux/pci.h missing a stub for pci_msix_can_alloc_dyn() when
> >   CONFIG_PCI_MSI=n. A separate fix was sent to address this. The fix can
> >   be found in the kernel (since v6.3-rc7) as
> >   commit 195d8e5da3ac ("PCI/MSI: Provide missing stub for pci_msix_can_alloc_dyn()")
> > - Biggest change is the transition to "active contexts" for both MSI and MSI-X.
> >   Interrupt contexts have always been allocated when the interrupts are
> >   allocated while they are only used while interrupts are
> >   enabled. In this series interrupt contexts are made dynamic, while doing
> >   so their allocation is moved to match how they are used: allocated when
> >   interrupts are enabled. Whether a Linux interrupt number exists determines
> >   whether an interrupt can be enabled.
> >   Previous policy (up to V2) that an allocated interrupt has an interrupt
> >   context no longer applies. Instead, an interrupt context has a
> >   handler/trigger, aka "active contexts". (Alex)
> > - Re-ordered patches in support of "active contexts".
> > - Only free interrupts on MSI-X teardown and otherwise use the
> >   allocated interrupts as a cache. (Alex)
> > - Using unsigned int for the vector broke the unwind loop within
> >   vfio_msi_set_block(). (Alex)
> > - Introduce new "has_dyn_msix" property of virtual device instead of
> >   querying support every time. (Alex)
> > - Some smaller changes, please refer to individual patches.
> >
> > Changes since RFC V1:
> > - RFC V1: https://lore.kernel.org/lkml/cover.1678911529.git.reinette.chatre@xxxxxxxxx/
> > - Improved changelogs.
> > - Simplify interface so that vfio_irq_ctx_alloc_single() returns pointer to
> >   allocated context. (Alex)
> > - Remove vfio_irq_ctx_range_allocated() and associated attempts to maintain
> >   invalid error path behavior. (Alex and Kevin)
> > - Add pointer to interrupt context as function parameter to
> >   vfio_irq_ctx_free(). (Alex)
> > - Ensure variables are initialized. (Dan Carpenter)
> > - Only support dynamic allocation if device supports it. (Alex)
> >
> > Qemu allocates interrupts incrementally at the time the guest unmasks an
> > interrupt, for example each time a Linux guest runs request_irq().
> >
> > Dynamic allocation of MSI-X interrupts was not possible until v6.2 [1].
> > This prompted Qemu to, when allocating a new interrupt, first release all
> > previously allocated interrupts (including disable of MSI-X) followed
> > by re-allocation of all interrupts that includes the new interrupt.
> > Please see [2] for a detailed discussion about this issue.
> >
> > Releasing and re-allocating interrupts may be acceptable if all
> > interrupts are unmasked during device initialization. If unmasking of
> > interrupts occur during runtime this may result in lost interrupts.
> > For example, consider an accelerator device with multiple work queues,
> > each work queue having a dedicated interrupt. A work queue can be
> > enabled at any time with its associated interrupt unmasked while other
> > work queues are already active. Having all interrupts released and MSI-X
> > disabled to enable the new work queue will impact active work queues.
> >
> > This series builds on the recent interrupt sub-system core changes
> > that added support for dynamic MSI-X allocation after initial MSI-X
> > enabling.
> >
> > Add support for dynamic MSI-X allocation to vfio-pci. A flag
> > indicating lack of support for dynamic allocation already exist:
> > VFIO_IRQ_INFO_NORESIZE and has always been set for MSI and MSI-X. With
> > support for dynamic MSI-X the flag is cleared for MSI-X when supported,
> > enabling Qemu to modify its behavior.
> >
> > Any feedback is appreciated
> >
> > Reinette
> >
> > [1] commit 34026364df8e ("PCI/MSI: Provide post-enable dynamic allocation interfaces for MSI-X")
> > [2] https://lore.kernel.org/kvm/MWHPR11MB188603D0D809C1079F5817DC8C099@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/#t
> >
> > Reinette Chatre (11):
> >   vfio/pci: Consolidate irq cleanup on MSI/MSI-X disable
> >   vfio/pci: Remove negative check on unsigned vector
> >   vfio/pci: Prepare for dynamic interrupt context storage
> >   vfio/pci: Move to single error path
> >   vfio/pci: Use xarray for interrupt context storage
> >   vfio/pci: Remove interrupt context counter
> >   vfio/pci: Update stale comment
> >   vfio/pci: Use bitfield for struct vfio_pci_core_device flags
> >   vfio/pci: Probe and store ability to support dynamic MSI-X
> >   vfio/pci: Support dynamic MSI-X
> >   vfio/pci: Clear VFIO_IRQ_INFO_NORESIZE for MSI-X
> >
> >  drivers/vfio/pci/vfio_pci_core.c  |   8 +-
> >  drivers/vfio/pci/vfio_pci_intrs.c | 305 ++++++++++++++++++++----------
> >  include/linux/vfio_pci_core.h     |  26 +--
> >  include/uapi/linux/vfio.h         |   3 +
> >  4 files changed, 229 insertions(+), 113 deletions(-)
> >
> >
> > base-commit: 457391b0380335d5e9a5babdec90ac53928b23b4
>
> Applied to vfio next branch for v6.5.  Thanks!
>
> Alex
>