Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch

[Bjorn Helgaas]

[+cc Lukas, author of ac91e6980563, Jason in case NVIDIA knows
anything]

On Thu, Nov 05, 2026 at 12:10:10PM -0800, Jennifer Chan wrote:
> On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
> NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
> downstream port.  "nvidia-smi -r" resets all GPUs: NVML unbinds the
> switches from the nvidia-nvswitch driver, resets each one through sysfs
> (reset_method "bus", a Secondary Bus Reset of its port) and binds them
> again.  The PCI core waits 100 ms after the link comes back, the switch
> answers config reads and the reset completes.  When nvidia-nvswitch then
> probes the switches, one fails in pci_enable_device() with
> 
>   Unable to change power state from D3cold to D0, device inaccessible
> 
> four others fail to initialize with -EIO, and the NVSwitch fabric does
> not come back.

Do you know where the -EIO is from?

> The same systems recover every time on a kernel that sleeps 1 s after
> each Secondary Bus Reset, as pci_reset_secondary_bus() did before commit
> ac91e6980563 ("PCI: Unify delay handling for reset and resume").  Adding
> a 1 s sleep before the link wait on the PEX 8725 ports of the failing
> kernel fixes it too.  Config reads already succeed at that point, so
> pci_dev_wait() does not cover it, and the switch's state there is not
> known.

Ugh, a second is a long time to wait.  Does the kernel you're testing
include e8df1d194c39 ("PCI: Don't treat ~0 as ready in pci_dev_wait()
with RRS SV"), by chance?  It only appeared in v7.3, so it's pretty
new.

> Give LR10 a 1 s d3cold_delay, so that pci_bridge_wait_for_secondary_bus()
> waits 1 s after the downstream link comes up before the switch is used.
> This adds 1 s to each LR10 reset (about 6 s to an "nvidia-smi -r" on
> these systems) and to D3cold resume of its port.  Resets issued further
> up the hierarchy keep the default delay.
> 
> Fixes: ac91e6980563 ("PCI: Unify delay handling for reset and resume")
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Jennifer Chan <jenniferchan@xxxxxxxx>
> ---
>  drivers/pci/quirks.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index 7f1f47ea2ee4..4a84095e736f 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -6569,12 +6569,14 @@ DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_TOSHIBA, 0x0220, of_pci_make_dev_node);
>   * after reset recovery or resume from D3cold:
>   *
>   * VideoPropulsion (aka Genroco) Torrent QN16e MPEG QAM Modulator
> + * NVIDIA LR10 NVSwitch (HGX A100)
>   */
>  static void pci_fixup_d3cold_delay_1sec(struct pci_dev *pdev)
>  {
>  	pdev->d3cold_delay = 1000;
>  }
>  DECLARE_PCI_FIXUP_FINAL(0x5555, 0x0004, pci_fixup_d3cold_delay_1sec);
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_NVIDIA, 0x1af1, pci_fixup_d3cold_delay_1sec);
>  
>  #ifdef CONFIG_PCIEAER
>  static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
> -- 
> 2.53.0-Meta
>