Re: [PATCH v16 5/5] s390/pci: Fix circular/recursive deadlocks in PCI-bus and -device release

From: Niklas Schnelle

Date: Wed Nov 18 2026 - 05:10:43 EST


On Fri, 2026-10-02 at 14:55 +0200, Benjamin Block wrote:
> When removing PCI device or PCI bus objects there are a couple of
> call-chains where it is possible that the kernel runs into a circular
> deadlock involving the central `pci_rescan_remove_lock`.
>
> Two examples:
>
> (A) Thread α receives a PCI event notifying the kernel that a PCI
> virtual function has been moved into Reserved state, and so the PCI
> subsystem will try to remove that PCI function. The call-chain for that
> looks like this:
> __zpci_event_availability()
> -> zpci_zdev_put() # will lock(zpci_add_remove_lock),
> # and lock(zpci_list_lock)
> -> zpci_release_device() # will unlock(zpci_list_lock)
> -> zpci_cleanup_bus_resources() # will lock(pci_rescan_remove_lock)
>
> Thread β is triggered by userspace writing 0 into the SysFS attribute
> `sriov_numvfs` of the parent PCI physical function of the same function
> we just try to remove. This will also try to release the PCI virtual
> function; but this time the call-chain looks like this:
> sriov_numvfs_store()
> -> ... (device driver dependent)
> -> sriov_disable()
> -> sriov_del_vfs() # will lock(pci_rescan_remove_lock)
> -> ... (deep chain)
> -> pci_release_dev()
> -> pcibios_release_device()
> -> zpci_zdev_put() # will lock(zpci_add_remove_lock)
>
> If thread α and β coincide, this will result in a cyclic deadlock.
>
> (B) Thread γ receives a PCI event notifying the kernel that one or more
> PCI functions got hot plugged, and need to be configured. A possible
> call-chain that might happen while the PCI subsystem is trying to add
> those new function looks like this:
> __zpci_event_availability()
> -> ... (multiple ways in which a device gets added)
> -> zpci_add_device() # will lock(zpci_add_remove_lock)
> -> zpci_bus_device_register()
> -> zpci_bus_put() # will lock(zbus_list_lock)
> -> zpci_bus_release() # will unlock(zbus_list_lock)
> # will lock(pci_rescan_remove_lock)
>
> Now the same thread β as above in (A) might coincide, and again could
> result in a cyclic deadlock.
>
> `pci_rescan_remove_lock` has to be and is taken at a "high level" in
> most call-chains since it is intended to protect/mutual exclude all
> rescan and/or removal actions taken in the PCI subsystem. So to prevent
> the outlined deadlock scenarios above remove it instead from the "low
> level" release functions for both the PCI device and PCI bus objects.
>
> Instead, lock `pci_rescan_remove_lock` in all call-chains leading to
> those release functions:
> * initialization of the PCI subsystem;
> * processing of availability events (CRWs) for PCI functions;
> * processing of error events (CRWs) for PCI functions;
> * architecture specific release PCI device implementation.
>
> Additionally, remove `pci_rescan_remove_lock` from zpci_bus_scan_bus()
> since its only caller zpci_scan_devices() is now always called with
> `pci_rescan_remove_lock` already held.
>
> Lastly, document the new locking expectations after these changes. Add
> sparse and lockdep annotations to functions that previously locked
> `pci_rescan_remove_lock` explicitly, making sure the lock is now
> already held when called. Additionally also add the annotations to
> zpci_zdev_put() and zpci_bus_put() to make sure that every function that
> potentially drops the last reference already holds the lock to prevent
> surprises.
>
> Fixes: 05bc1be6db4b2 ("s390/pci: create zPCI bus")
> Fixes: ab909509850b2 ("PCI: s390: Fix use-after-free of PCI resources with per-function hotplug")
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Benjamin Block <bblock@xxxxxxxxxxxxx>

Thank you for the very detailed analysis. I agree with it and the fix
approach.

> ---
> arch/s390/pci/pci.c | 10 ++++++++--
> arch/s390/pci/pci_bus.c | 15 ++++++++-------
> arch/s390/pci/pci_event.c | 29 +++++++++++++++++++----------
> arch/s390/pci/pci_iov.c | 3 +--
> arch/s390/pci/pci_sysfs.c | 25 ++++++++++++-------------
> 5 files changed, 48 insertions(+), 34 deletions(-)
--- snip ---
>
> diff --git a/arch/s390/pci/pci_bus.c b/arch/s390/pci/pci_bus.c
> index 36a4807285fa..c1b48b572e86 100644
> --- a/arch/s390/pci/pci_bus.c
> +++ b/arch/s390/pci/pci_bus.c
> @@ -82,9 +82,8 @@ int zpci_bus_scan_device(struct zpci_dev *zdev)
> if (!pdev)
> return -ENODEV;
>
> - pci_lock_rescan_remove();
> + guard(pci_rescan_remove)();
> pci_bus_add_device(pdev);
> - pci_unlock_rescan_remove();

This is a pre-existing issue not introduced by your patches, but I
think pci_scan_single_device() further up in zpci_bus_scan_device()
should also be done under the pci_rescan_remove lock.

>
> return 0;
> }
> @@ -132,10 +131,13 @@ void zpci_bus_remove_device(struct zpci_dev *zdev, bool set_error)
> * Return: 0 on success, an error value otherwise
> */
> int zpci_bus_scan_bus(struct zpci_bus *zbus)
> + __must_hold(&pci_rescan_remove_lock)
> {
> struct zpci_dev *zdev;
> int devfn, rc, ret = 0;
>
> + lockdep_assert_held(&pci_rescan_remove_lock);
> +
> for (devfn = 0; devfn < ZPCI_FUNCTIONS_PER_BUS; devfn++) {
> zdev = zbus->function[devfn];
> if (zdev && zdev->state == ZPCI_FN_STATE_CONFIGURED) {
> @@ -145,10 +147,8 @@ int zpci_bus_scan_bus(struct zpci_bus *zbus)
> }
> }
>
> - pci_lock_rescan_remove();
> pci_scan_child_bus(zbus->bus);
> pci_bus_add_devices(zbus->bus);
> - pci_unlock_rescan_remove();
>
> return ret;
> }
--- snip ---

Apart from the pre-existing issue this all looks good to me. The
remaining issue is close enough in nature that I think it should be
part of this patch and it does become simpler due to the guard(). The
addition is simple enough that you can go ahead and add my:

Reviewed-by: Niklas Schnelle <schnelle@xxxxxxxxxxxxx>

Thanks,
Niklas