Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform

From: Alex Williamson
Date: Fri May 12 2017 - 10:43:21 EST

On Fri, 12 May 2017 04:51:43 +0000
"Cheng, Collins" <Collins.Cheng@xxxxxxx> wrote:

> Hi Williamson,
>
> I verified the patch is working for both AMD SR-IOV GPU and Intel SR-IOV NIC. I don't think it is redundant to check the VF BAR valid before call sriov_init(), it is safe and saving boot time, also there is no a better method to know if system BIOS has correctly initialized the SR-IOV capability or not.

It also masks an underlying bug and creates a maintenance issue that we
won't know when it's safe to remove this workaround. I don't think
faster boot is valid rationale, in one case SR-IOV is completely
disabled, the other we attempt to allocate the resources the BIOS
failed to provide. I expect this is also a corner case, the BIOS
should typically support SR-IOV, therefore this situation should be an
exception.

> I did not try to fix the issue from the kernel resource allocation perspective, it is because:
> 1. I am not very familiar with the PCI resource allocation scheme in kernel. For example, in sriov_init(), kernel will re-assign the PCI resource for both VF and PF. I don't understand why kernel allocates resource for VF firstly, then PF. If it is PF firstly, then this issue could be avoided.
> 2. I am not sure if kernel has error handler if PCI resource allocation failed. In this case, kernel cannot allocate enough resource to PF. It should trigger some error handler to either just keep original BAR values set by system BIOS, or disable this device and log errors.

I think these are the issues we should be trying to solve and I'm sure
folks on the linux-pci list can help us identify the bug. Minimally,
failure to allocate VF resources should leave the device in no worse
condition than before it tried. Perhaps you could post more details
about the issue, boot with pci=earlydump, post dmesg of a boot where
the PF resources are incorrectly re-allocated, and include lspci -vvv
for the SR-IOV device. Also, please test with the latest upstream
kernel, upstream only patches old kernels through stable backports of
commits to the latest kernel. Adding Yinghai as a resource allocation
expert. Thanks,

Alex

> -----Original Message-----
> From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx]
> Sent: Friday, May 12, 2017 12:01 PM
> To: Cheng, Collins <Collins.Cheng@xxxxxxx>
> Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>; linux-pci@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Deucher, Alexander <Alexander.Deucher@xxxxxxx>; Zytaruk, Kelly <Kelly.Zytaruk@xxxxxxx>
> Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the SR-IOV incapable platform
>
> On Fri, 12 May 2017 03:42:46 +0000
> "Cheng, Collins" <Collins.Cheng@xxxxxxx> wrote:
>
> > Hi Williamson,
> >
> > GPU card needs more BAR aperture resource than other PCI devices. For example, Intel SR-IOV network card only require 512KB memory resource for all VFs. AMD SR-IOV GPU card needs 256MB x16 VF = 4GB memory resource for frame buffer BAR aperture.
> >
> > If the system BIOS supports SR-IOV, it will reserve enough resource for all VF BARs. If the system BIOS doesn't support SR-IOV or cannot allocate the enough resource for VF BARs, only PF BAR will be assigned and VF BARs are empty. Then system boots to Linux kernel and kernel doesn't check if the VF BARs are empty or valid. Kernel will re-assign the BAR resources for PF and all VFs. The problem I saw is that kernel will fail to allocate PF BAR resource because some resources are assigned to VF, this is not expected. So kernel might need to do some check before re-assign the PF/VF resource, so that PF device will be correctly assigned BAR resource and user can use PF device.
>
> So the problem is that something bad happens when the kernel is trying to reallocate resources in order to fulfill the requirements of the VFs, leaving the PF resources incorrectly programmed? Why not just fix that bug rather than creating special handling for this vendor/class of device which disables any attempt to fixup resources for SR-IOV? IOW, this patch just avoids the problem for your devices rather than fixing the bug. I'd suggest fixing the bug such that the PF is left in a functional state if the kernel is unable to allocate sufficient resources for the VFs. Thanks,
>
> Alex
>
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson@xxxxxxxxxx]
> > Sent: Friday, May 12, 2017 11:21 AM
> > To: Cheng, Collins <Collins.Cheng@xxxxxxx>
> > Cc: Bjorn Helgaas <bhelgaas@xxxxxxxxxx>; linux-pci@xxxxxxxxxxxxxxx;
> > linux-kernel@xxxxxxxxxxxxxxx; Deucher, Alexander
> > <Alexander.Deucher@xxxxxxx>; Zytaruk, Kelly <Kelly.Zytaruk@xxxxxxx>
> > Subject: Re: [PATCH] PCI: Make SR-IOV capable GPU working on the
> > SR-IOV incapable platform
> >
> > On Fri, 12 May 2017 02:50:32 +0000
> > "Cheng, Collins" <Collins.Cheng@xxxxxxx> wrote:
> >
> > > Hi Helgaas,
> > >
> > > Some AMD GPUs have hardware support for graphics SR-IOV.
> > > If the SR-IOV capable GPU is plugged into the SR-IOV incapable
> > > platform. It would cause a problem on PCI resource allocation in
> > > current Linux kernel.
> > >
> > > Therefore in order to allow the PF (Physical Function) device of
> > > SR-IOV capable GPU to work on the SR-IOV incapable platform, it is
> > > required to verify conditions for initializing BAR resources on AMD
> > > SR-IOV capable GPUs.
> > >
> > > If the device is an AMD graphics device and it supports SR-IOV it
> > > will require a large amount of resources.
> > > Before calling sriov_init() must ensure that the system BIOS also
> > > supports SR-IOV and that system BIOS has been able to allocate
> > > enough resources.
> > > If the VF BARs are zero then the system BIOS does not support SR-IOV
> > > or it could not allocate the resources and this platform will not
> > > support AMD graphics SR-IOV.
> > > Therefore do not call sriov_init().
> > > If the system BIOS does support SR-IOV then the VF BARs will be
> > > properly initialized to non-zero values.
> > >
> > > Below is the patch against to Kernel 4.8 & 4.9. Please review.
> > >
> > > I checked the drivers/pci/quirks.c, it looks the workarounds/fixes
> > > in quirks.c are for specific devices and one or more device ID are
> > > defined for the specific devices. However my patch is for all AMD
> > > SR-IOV capable GPUs, that includes all existing and future AMD server GPUs.
> > > So it doesn't seem like a good fit to put the fix in quirks.c.
> >
> >
> > Why is an AMD graphics card unique here? Doesn't sriov_init() always
> > need to be able to deal with devices of any type where the BIOS hasn't
> > initialized the SR-IOV capability? Some SR-IOV devices can fit their
> > VFs within a minimum bridge aperture, most cannot. I don't understand
> > why the VF resource requirements being exceptionally large dictates
> > that they receive special handling. Thanks,
> >
> > Alex
> >
> > > Signed-off-by: Collins Cheng <collins.cheng@xxxxxxx>
> > > ---
> > > drivers/pci/iov.c | 63
> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++---
> > > 1 file changed, 60 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/pci/iov.c b/drivers/pci/iov.c index
> > > e30f05c..e4f1405 100644
> > > --- a/drivers/pci/iov.c
> > > +++ b/drivers/pci/iov.c
> > > @@ -523,6 +523,45 @@ static void sriov_restore_state(struct pci_dev *dev)
> > > msleep(100);
> > > }
> > >
> > > +/*
> > > + * pci_vf_bar_valid - check if VF BARs have resource allocated
> > > + * @dev: the PCI device
> > > + * @pos: register offset of SR-IOV capability in PCI config space
> > > + * Returns true any VF BAR has resource allocated, false
> > > + * if all VF BARs are empty.
> > > + */
> > > +static bool pci_vf_bar_valid(struct pci_dev *dev, int pos) {
> > > + int i;
> > > + u32 bar_value;
> > > + u32 bar_size_mask = ~(PCI_BASE_ADDRESS_SPACE |
> > > + PCI_BASE_ADDRESS_MEM_TYPE_64 |
> > > + PCI_BASE_ADDRESS_MEM_PREFETCH);
> > > +
> > > + for (i = 0; i < PCI_SRIOV_NUM_BARS; i++) {
> > > + pci_read_config_dword(dev, pos + PCI_SRIOV_BAR + i * 4, &bar_value);
> > > + if (bar_value & bar_size_mask)
> > > + return true;
> > > + }
> > > +
> > > + return false;
> > > +}
> > > +
> > > +/*
> > > + * is_amd_display_adapter - check if it is an AMD/ATI GPU device
> > > + * @dev: the PCI device
> > > + *
> > > + * Returns true if device is an AMD/ATI display adapter,
> > > + * otherwise return false.
> > > + */
> > > +
> > > +static bool is_amd_display_adapter(struct pci_dev *dev) {
> > > + return (((dev->class >> 16) == PCI_BASE_CLASS_DISPLAY) &&
> > > + (dev->vendor == PCI_VENDOR_ID_ATI ||
> > > + dev->vendor == PCI_VENDOR_ID_AMD)); }
> > > +
> > > /**
> > > * pci_iov_init - initialize the IOV capability
> > > * @dev: the PCI device
> > > @@ -537,9 +576,27 @@ int pci_iov_init(struct pci_dev *dev)
> > > return -ENODEV;
> > >
> > > pos = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_SRIOV);
> > > - if (pos)
> > > - return sriov_init(dev, pos);
> > > -
> > > + if (pos) {
> > > + /*
> > > + * If the device is an AMD graphics device and it supports
> > > + * SR-IOV it will require a large amount of resources.
> > > + * Before calling sriov_init() must ensure that the system
> > > + * BIOS also supports SR-IOV and that system BIOS has been
> > > + * able to allocate enough resources.
> > > + * If the VF BARs are zero then the system BIOS does not
> > > + * support SR-IOV or it could not allocate the resources
> > > + * and this platform will not support AMD graphics SR-IOV.
> > > + * Therefore do not call sriov_init().
> > > + * If the system BIOS does support SR-IOV then the VF BARs
> > > + * will be properly initialized to non-zero values.
> > > + */
> > > + if (is_amd_display_adapter(dev)) {
> > > + if (pci_vf_bar_valid(dev, pos))
> > > + return sriov_init(dev, pos);
> > > + } else {
> > > + return sriov_init(dev, pos);
> > > + }
> > > + }
> > > return -ENODEV;
> > > }
> > >
> >
>