Re: [RFC PATCH 1/5] PCI: Refuse function reset of an SR-IOV PF with enabled VFs

[Samiullah Khawaja]

On Thu, Aug 27, 2026 at 05:35:17AM +0000, Tian, Kevin wrote:
> > From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> > Sent: Wednesday, August 19, 2026 4:39 AM
> >
> > On Tue, 18 Aug 2026 11:03:04 -0300
> > Jason Gunthorpe <jgg@xxxxxxxx> wrote:
> >
> > > On Tue, Aug 18, 2026 at 07:07:20AM +0000, Tian, Kevin wrote:
> > > > 3) resetting a PF due to ad-hoc FW error
> > > >
> > > > e.g. pdsc_check_pci_health() calls pci_reset_function() when FW
> > > > reports broken PCI. Sounds a similar condition to AER.
> > > >
> > > > but as it calls pci_reset_function() this series causes a regression
> > > > in case of active VFs. Is it desired?
> > >
> > > mlx5 does something similar. RAS flows require detecting the PF has
> > > blown up and resetting it to bring it back.
> > >
> > > This has to work while VFs exist and bound to kernel drivers or vfio.
> > >
> > > In the VFIO KVM case I would expect the VM to experiance a RAS error
> > > on its vPCI device and trigger a VF flr
> >
> > This would need to be a device level RAS flow as we're sorely lacking
> > on forwarding reset notification to VF drivers or providing any
> > mechanism for recovery from the vfio error eventfd.
> >
> > > But we have to sort all this out through the hypervisor to not crash
> > > the machine or otherwise :)
> >
> > In general guarding the high level pci_reset_function() family on
> > (pci_num_vfs() == 0) seems to have more pros than cons and the
> > restriction matches the scoped intent of the interface.
> >
> > The low level __pci_reset_function_locked() interface remains
> > untouched, and as the prefix implies, relies on the caller to do
> > necessary verification and preparation.  A "reset the PF regardless of
> > VFs" seems like an underscore prefix sort of operation.  We'll need to
> > export the pci_dev_save_and_disable() and pci_dev_restore() functions
> > to provide handlers the full suite of tools they need to replace the
> > existing function and port those we find (maybe also questioning the
> > author's intent on VF reset and recovery).
> >
> > Of course making the guard parameterized with wrappers to expand this
> > family of calls is also an option,
> > pci_reset_function_locked_with_vfs()...
>
> +1 for wrappers. Maybe call it pci_force_reset_function_locked() which
> is slightly forward-looking if in future other conditions may reject the
> reset request using the base function.
>
> > The mlx5 driver doesn't actually seem to be affected by these changes,
> > doing its own link toggle or a PCI bus reset, not a
> > pci_reset_function() call.
> >
> > On the other improvements, triggering a pci_reset_function_locked() on
> > unbind after SR-IOV is disabled, when needs_reset is indicated, looks
> > pretty trivial.
> >
> > The blocking domain operation looks like it might be simplest to
> > implement in the IOMMU core.  We can set a flag for a default blocking
> > domain on the IOMMU group when we take_dma_ownership of the group.
> > Then
> > release_dma_ownership picks the blocking rather than default domain.
>
> IMHO we should change release_dma_ownership() to always picks the
> blocking. No new flag.
>
> this take/release interfaces are invented for drivers which manage dma
> on their own, so it sounds reasonable to not assume the operated device
> is in a good shape after the ownership is released.
>
> Then only a clean reset can unwind it.
>
> > This is then unwound in use_default_domain, called via dma_configure,
> > attaching the device to the default domain in probe of the next driver.
> > Therefore until probe by another driver, a device used by vfio would
> > remain in a blocking domain even while unused and unbound.
>
> so unwinding it blindly at dma_configure time is also problematic, as the
> device may still have a bad state to hurt the kernel.
>
> Instead use_default_domain should fail in such case (in blocked with
> owner_cnt being '0') then a reset is expected to unwind it (Nicolin's
> work will carry the resetting status to pci_dev_reset_iommu_done())

Thinking about this more, it might be tricky in multiple ways,

- This will only work if drivers do reset after dma ownership is
  released. Looking at the code, vfio currently resets the device first
  and then dma ownership is released.
- For some devices with no reset mechanism, vfio-pci uses bus or slot
  resets. This will need extra care to propagate the
  reset_iommu_prepare/reset_iommu_done to subordinate devices to bring
  the device out of blocking domain.
- AFAIU looking at vfio-pci code, some devices do not have proper reset
  mechanism, not even slot/bus, so those devices would be stuck in the
  blocking domain state. Probably need a mechanism to let IOMMU know
  that there is no reset mechanism?

Thanks,
Sami