RE: [RFC PATCH 1/5] PCI: Refuse function reset of an SR-IOV PF with enabled VFs

[Tian, Kevin]

> From: Alex Williamson <alex.williamson@xxxxxxxxxx>
> Sent: Wednesday, August 19, 2026 4:39 AM
> 
> On Tue, 18 Aug 2026 11:03:04 -0300
> Jason Gunthorpe <jgg@xxxxxxxx> wrote:
> 
> > On Tue, Aug 18, 2026 at 07:07:20AM +0000, Tian, Kevin wrote:
> > > 3) resetting a PF due to ad-hoc FW error
> > >
> > > e.g. pdsc_check_pci_health() calls pci_reset_function() when FW
> > > reports broken PCI. Sounds a similar condition to AER.
> > >
> > > but as it calls pci_reset_function() this series causes a regression
> > > in case of active VFs. Is it desired?
> >
> > mlx5 does something similar. RAS flows require detecting the PF has
> > blown up and resetting it to bring it back.
> >
> > This has to work while VFs exist and bound to kernel drivers or vfio.
> >
> > In the VFIO KVM case I would expect the VM to experiance a RAS error
> > on its vPCI device and trigger a VF flr
> 
> This would need to be a device level RAS flow as we're sorely lacking
> on forwarding reset notification to VF drivers or providing any
> mechanism for recovery from the vfio error eventfd.
> 
> > But we have to sort all this out through the hypervisor to not crash
> > the machine or otherwise :)
> 
> In general guarding the high level pci_reset_function() family on
> (pci_num_vfs() == 0) seems to have more pros than cons and the
> restriction matches the scoped intent of the interface.
> 
> The low level __pci_reset_function_locked() interface remains
> untouched, and as the prefix implies, relies on the caller to do
> necessary verification and preparation.  A "reset the PF regardless of
> VFs" seems like an underscore prefix sort of operation.  We'll need to
> export the pci_dev_save_and_disable() and pci_dev_restore() functions
> to provide handlers the full suite of tools they need to replace the
> existing function and port those we find (maybe also questioning the
> author's intent on VF reset and recovery).
> 
> Of course making the guard parameterized with wrappers to expand this
> family of calls is also an option,
> pci_reset_function_locked_with_vfs()...

+1 for wrappers. Maybe call it pci_force_reset_function_locked() which
is slightly forward-looking if in future other conditions may reject the
reset request using the base function.

> 
> The mlx5 driver doesn't actually seem to be affected by these changes,
> doing its own link toggle or a PCI bus reset, not a
> pci_reset_function() call.
> 
> On the other improvements, triggering a pci_reset_function_locked() on
> unbind after SR-IOV is disabled, when needs_reset is indicated, looks
> pretty trivial.
> 
> The blocking domain operation looks like it might be simplest to
> implement in the IOMMU core.  We can set a flag for a default blocking
> domain on the IOMMU group when we take_dma_ownership of the group.
> Then
> release_dma_ownership picks the blocking rather than default domain.

IMHO we should change release_dma_ownership() to always picks the
blocking. No new flag.

this take/release interfaces are invented for drivers which manage dma
on their own, so it sounds reasonable to not assume the operated device
is in a good shape after the ownership is released.

Then only a clean reset can unwind it.

> 
> This is then unwound in use_default_domain, called via dma_configure,
> attaching the device to the default domain in probe of the next driver.
> Therefore until probe by another driver, a device used by vfio would
> remain in a blocking domain even while unused and unbound.
> 

so unwinding it blindly at dma_configure time is also problematic, as the
device may still have a bad state to hurt the kernel.

Instead use_default_domain should fail in such case (in blocked with
owner_cnt being '0') then a reset is expected to unwind it (Nicolin's
work will carry the resetting status to pci_dev_reset_iommu_done())

Once we follow Jason's suggestion to do reset on vfio unbind, user vfio
experience is intact across binding/unbinding drivers.

Then when a 2nd driver claiming dma comes but it forgets to do reset
on unbind, only manual reset can unblock the device to bind to a new
driver.