Re: [PATCH v4 2/5] xen/pciback: Don't deadlock when unbinding.

From: Konrad Rzeszutek Wilk
Date: Mon Jul 14 2014 - 11:43:09 EST


On Mon, Jul 14, 2014 at 10:30:13AM -0400, Boris Ostrovsky wrote:
> On 07/14/2014 10:13 AM, Konrad Rzeszutek Wilk wrote:
> >On Fri, Jul 11, 2014 at 05:02:01PM -0400, Konrad Rzeszutek Wilk wrote:
> >>>>--- a/drivers/xen/xen-pciback/pci_stub.c
> >>>>+++ b/drivers/xen/xen-pciback/pci_stub.c
> >>>>@@ -250,6 +250,8 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
> >>>> * - 'echo BDF > unbind' with a guest still using it. See pcistub_remove
> >>>> *
> >>>> * As such we have to be careful.
> >>>>+ *
> >>>>+ * To make this easier, the caller has to hold the device lock.
> >>>Should we assert that the lock is being held?
> >>Yes of course we should. Thank you!
> >How about this:
> >
> > From 388a03c598218dac8bfeb6c5bf3992e0d1e37d1e Mon Sep 17 00:00:00 2001
> >From: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> >Date: Tue, 8 Jul 2014 11:12:02 -0400
> >Subject: [PATCH] xen/pciback: Don't deadlock when unbinding.
> >
> >As commit 0a9fd0152929db372ff61b0d6c280fdd34ae8bdb
> >'xen/pciback: Document the entry points for 'pcistub_put_pci_dev''
> >explained there are four entry points in this function.
> >Two of them are when the user fiddles in the SysFS to
> >unbind a device which might be in use by a guest or not.
> >
> >Both 'unbind' states will cause a deadlock as the the PCI lock has
> >already been taken, which then pci_device_reset tries to take.
> >
> >We can simplify this by requiring that all callers of
> >pcistub_put_pci_dev MUST hold the device lock. And then
> >we can just call the lockless version of pci_device_reset.
> >
> >To make it even simpler we will modify xen_pcibk_release_pci_dev
> >to quality whether it should take a lock or not - as it ends
> >up calling xen_pcibk_release_pci_dev and needs to hold the lock.
> >
> >Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>
> >---
> >[v2: Per David Vrabel's suggestion - use lockless version of reset]
> >[v3: Per Boris suggestion add assertion mechanism]
> >---
> > drivers/xen/xen-pciback/passthrough.c | 9 +++++++--
> > drivers/xen/xen-pciback/pci_stub.c | 12 ++++++------
> > drivers/xen/xen-pciback/pciback.h | 7 ++++---
> > drivers/xen/xen-pciback/vpci.c | 9 +++++++--
> > drivers/xen/xen-pciback/xenbus.c | 2 +-
> > 5 files changed, 25 insertions(+), 14 deletions(-)
> >
> >diff --git a/drivers/xen/xen-pciback/passthrough.c b/drivers/xen/xen-pciback/passthrough.c
> >index 828dddc..d0c3fb4 100644
> >--- a/drivers/xen/xen-pciback/passthrough.c
> >+++ b/drivers/xen/xen-pciback/passthrough.c
> >@@ -69,7 +69,7 @@ static int __xen_pcibk_add_pci_dev(struct xen_pcibk_device *pdev,
> > }
> > static void __xen_pcibk_release_pci_dev(struct xen_pcibk_device *pdev,
> >- struct pci_dev *dev)
> >+ struct pci_dev *dev, bool lock)
> > {
> > struct passthrough_dev_data *dev_data = pdev->pci_dev_data;
> > struct pci_dev_entry *dev_entry, *t;
> >@@ -87,8 +87,13 @@ static void __xen_pcibk_release_pci_dev(struct xen_pcibk_device *pdev,
> > mutex_unlock(&dev_data->lock);
> >- if (found_dev)
> >+ if (found_dev) {
> >+ if (lock)
> >+ device_lock(&found_dev->dev);
> > pcistub_put_pci_dev(found_dev);
> >+ if (lock)
> >+ device_unlock(&found_dev->dev);
> >+ }
> > }
> > static int __xen_pcibk_init_devices(struct xen_pcibk_device *pdev)
> >diff --git a/drivers/xen/xen-pciback/pci_stub.c b/drivers/xen/xen-pciback/pci_stub.c
> >index d57a173..8293fbb 100644
> >--- a/drivers/xen/xen-pciback/pci_stub.c
> >+++ b/drivers/xen/xen-pciback/pci_stub.c
> >@@ -250,6 +250,8 @@ struct pci_dev *pcistub_get_pci_dev(struct xen_pcibk_device *pdev,
> > * - 'echo BDF > unbind' with a guest still using it. See pcistub_remove
> > *
> > * As such we have to be careful.
> >+ *
> >+ * To make this easier, the caller has to hold the device lock.
> > */
> > void pcistub_put_pci_dev(struct pci_dev *dev)
> > {
> >@@ -276,11 +278,8 @@ void pcistub_put_pci_dev(struct pci_dev *dev)
> > /* Cleanup our device
> > * (so it's ready for the next domain)
> > */
> >-
> >- /* This is OK - we are running from workqueue context
> >- * and want to inhibit the user from fiddling with 'reset'
> >- */
> >- pci_reset_function(dev);
> >+ lockdep_assert_held(&dev->dev.mutex);
> >+ __pci_reset_function_locked(dev);
> > pci_restore_state(dev);
>
> Reviewed-by: Boris Ostrovsky <boris.ostrovsky@xxxxxxxxxx>
>
> (Although I wonder about the fact that we are exposing the mutex which is
> typically hidden by device_lock()/unlock() inlines. Have you considered
> adding something like is_device_locked() to device.h?)

I did, but this is a bug-fix (which can be backported to stable) so I thought
it would not be nice - as that is more of an API change.

Instead I split it up and there is another patch that makes it
an 'device_lock_assert' function.

And thanks to your idea - I did find two instances where we did
call without a mutex held.

Reposting shortly (will retain your Reviewed-by - please scream if you
prefer that I drop it).
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/