Re: [PATCH kernel] PCI: Disable IOV before pcibios_sriov_disable()

From: Alexey Kardashevskiy
Date: Wed Aug 30 2017 - 23:08:45 EST


On 31/08/17 05:02, Bjorn Helgaas wrote:
> On Fri, Aug 11, 2017 at 06:19:33PM +1000, Alexey Kardashevskiy wrote:
>> From: Gavin Shan <gwshan@xxxxxxxxxxxxxxxxxx>
>>
>> The PowerNV platform is the only user of pcibios_sriov_disable().
>> The IOV BAR could be shifted by pci_iov_update_resource(). The
>> warning message in the function is printed if the IOV capability
>> is in enabled (PCI_SRIOV_CTRL_VFE && PCI_SRIOV_CTRL_MSE) state.
>>
>> This is the backtrace of what is happening:
>> pci_disable_sriov
>> sriov_disable
>> pnv_pci_sriov_disable
>> pnv_pci_vf_resource_shift
>> pci_update_resource
>> pci_iov_update_resource
>>
>> This fixes the issue by disabling IOV capability before calling
>> pcibios_sriov_disable(). With it, the disabling path matches
>> the enabling path: pcibios_sriov_enable() is called before the
>> IOV capability is enabled.
>>
>> Cc: shan.gavin@xxxxxxxxx
>> Cc: Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>
>> Cc: Paul Mackerras <paulus@xxxxxxxxx>
>> Reported-by: Carol L Soto <clsoto@xxxxxxxxxx>
>> Signed-off-by: Gavin Shan <gwshan@xxxxxxxxxxxxxxxxxx>
>> Tested-by: Carol L Soto <clsoto@xxxxxxxxxx>
>> Signed-off-by: Alexey Kardashevskiy <aik@xxxxxxxxx>
>> ---
>>
>> This is repost. Since Gavin left the team, I am trying to push it out.
>> The previos converstion is here: https://patchwork.ozlabs.org/patch/732653/
>
> I gave up on the previous issue. I think this patch makes sense as-is
> at least as far as the fact that we can't update a struct resource
> while the device is still consuming it. I reworked the changelog to
> emphasize that.
>
> I assume the fact that pci_iov_update_resource() dropped the resource
> update caused some user-visible issue later on, and I might mention
> that, too, if I knew what it was.

I could not identify any issue so far in my test setup - I recreated VFs
several times, run some traffic through them on one of mellanox'es so the
message+backtrace seems to be the only issue for now.


> Here's what I would consider putting on pci/virtualization (the diff
> is unchanged from your post):


This sounds good to me, thanks for updating the commit log.

I'll still try and finish my homework with updating that comment about the
hole in arch/powerpc/platforms/powernv/pci-ioda.c.

Cheers.

>
>
> commit 08132e7759b3929bea0ccdf8afe81ebf05351389
> Author: Gavin Shan <gwshan@xxxxxxxxxxxxxxxxxx>
> Date: Fri Aug 11 18:19:33 2017 +1000
>
> PCI: Disable VF decoding before updating resources in pcibios_sriov_disable()
>
> A struct resource represents the address space consumed by a device. We
> should not modify that resource while the device is actively using the
> address space. For VFs, pci_iov_update_resource() enforces this by
> printing a warning and doing nothing if the VFE (VF Enable) and MSE (VF
> Memory Space Enable) bits are set.
>
> Previously, both sriov_enable() and sriov_disable() called the
> pcibios_sriov_disable() arch hook, which may update the struct resource,
> while VFE and MSE were enabled. This effectively dropped the resource
> update pcibios_sriov_disable() intended to do.
>
> Disable VF memory decoding before calling pcibios_sriov_disable().
>
> Reported-by: Carol L Soto <clsoto@xxxxxxxxxx>
> Tested-by: Carol L Soto <clsoto@xxxxxxxxxx>
> Signed-off-by: Gavin Shan <gwshan@xxxxxxxxxxxxxxxxxx>
> Signed-off-by: Alexey Kardashevskiy <aik@xxxxxxxxx>
> [bhelgaas: changelog]
> Signed-off-by: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>
> Cc: shan.gavin@xxxxxxxxx
> Cc: Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx>
> Cc: Paul Mackerras <paulus@xxxxxxxxx>

>
> diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c
> index 120485d6f352..ac41c8be9200 100644
> --- a/drivers/pci/iov.c
> +++ b/drivers/pci/iov.c
> @@ -331,7 +331,6 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> while (i--)
> pci_iov_remove_virtfn(dev, i, 0);
>
> - pcibios_sriov_disable(dev);
> err_pcibios:
> iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
> pci_cfg_access_lock(dev);
> @@ -339,6 +338,8 @@ static int sriov_enable(struct pci_dev *dev, int nr_virtfn)
> ssleep(1);
> pci_cfg_access_unlock(dev);
>
> + pcibios_sriov_disable(dev);
> +
> if (iov->link != dev->devfn)
> sysfs_remove_link(&dev->dev.kobj, "dep_link");
>
> @@ -357,14 +358,14 @@ static void sriov_disable(struct pci_dev *dev)
> for (i = 0; i < iov->num_VFs; i++)
> pci_iov_remove_virtfn(dev, i, 0);
>
> - pcibios_sriov_disable(dev);
> -
> iov->ctrl &= ~(PCI_SRIOV_CTRL_VFE | PCI_SRIOV_CTRL_MSE);
> pci_cfg_access_lock(dev);
> pci_write_config_word(dev, iov->pos + PCI_SRIOV_CTRL, iov->ctrl);
> ssleep(1);
> pci_cfg_access_unlock(dev);
>
> + pcibios_sriov_disable(dev);
> +
> if (iov->link != dev->devfn)
> sysfs_remove_link(&dev->dev.kobj, "dep_link");
>
>


--
Alexey