[PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Jennifer Chan]

On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
downstream port.  "nvidia-smi -r" resets all GPUs: NVML unbinds the
switches from the nvidia-nvswitch driver, resets each one through sysfs
(reset_method "bus", a Secondary Bus Reset of its port) and binds them
again.  The PCI core waits 100 ms after the link comes back, the switch
answers config reads and the reset completes.  When nvidia-nvswitch then
probes the switches, one fails in pci_enable_device() with

  Unable to change power state from D3cold to D0, device inaccessible

four others fail to initialize with -EIO, and the NVSwitch fabric does
not come back.

The same systems recover every time on a kernel that sleeps 1 s after
each Secondary Bus Reset, as pci_reset_secondary_bus() did before commit
ac91e6980563 ("PCI: Unify delay handling for reset and resume").  Adding
a 1 s sleep before the link wait on the PEX 8725 ports of the failing
kernel fixes it too.  Config reads already succeed at that point, so
pci_dev_wait() does not cover it, and the switch's state there is not
known.

Give LR10 a 1 s d3cold_delay, so that pci_bridge_wait_for_secondary_bus()
waits 1 s after the downstream link comes up before the switch is used.
This adds 1 s to each LR10 reset (about 6 s to an "nvidia-smi -r" on
these systems) and to D3cold resume of its port.  Resets issued further
up the hierarchy keep the default delay.

Fixes: ac91e6980563 ("PCI: Unify delay handling for reset and resume")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Jennifer Chan <jenniferchan@xxxxxxxx>
---
 drivers/pci/quirks.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 7f1f47ea2ee4..4a84095e736f 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -6569,12 +6569,14 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_TOSHIBA, 0x0220, of_pci_make_dev_node);
  * after reset recovery or resume from D3cold:
  *
  * VideoPropulsion (aka Genroco) Torrent QN16e MPEG QAM Modulator
+ * NVIDIA LR10 NVSwitch (HGX A100)
  */
 static void pci_fixup_d3cold_delay_1sec(struct pci_dev *pdev)
 {
 	pdev->d3cold_delay = 1000;
 }
 DECLARE_PCI_FIXUP_FINAL(0x5555, 0x0004, pci_fixup_d3cold_delay_1sec);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x1af1, pci_fixup_d3cold_delay_1sec);
 
 #ifdef CONFIG_PCIEAER
 static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
-- 
2.53.0-Meta