Re: [PATCH] PCI: Add d3cold delay quirk for NVIDIA LR10 NVSwitch
From: Lukas Wunner
Date: Fri Nov 06 2026 - 06:05:38 EST
On Thu, Nov 05, 2026 at 03:51:48PM -0600, Bjorn Helgaas wrote:
> On Thu, Nov 05, 2026 at 12:10:10PM -0800, Jennifer Chan wrote:
> > On a Supermicro HGX A100 system (X12DGO-6), each of the six LR10
> > NVSwitches (10de:1af1) sits alone behind a PLX PEX 8725 (10b5:8725)
> > downstream port. "nvidia-smi -r" resets all GPUs: NVML unbinds the
> > switches from the nvidia-nvswitch driver, resets each one through sysfs
> > (reset_method "bus", a Secondary Bus Reset of its port) and binds them
> > again. The PCI core waits 100 ms after the link comes back, the switch
> > answers config reads and the reset completes.
Broadcom PEX 89xxx switches support a "virtual switch" feature whereby
the switch spoofs responses from downstream devices when they're
inaccessible. I just checked the datasheet of the PEX 8725 and it
seems to support this feature as well.
This feature makes the kernel believe that the downstream device is
ready even though it actually isn't.
I was made aware of this internally because a Supermicro server used
for testing contained two PEX 89xxx switches which had that feature
enabled. I see you're using a Supermicro server as well, so perhaps
it has the same problem.
I developed a patch (included below, goes on top of v7.4-rc1) to work
around this misbehavior. You may want to give it a spin and see if
it helps. I never upstreamed the patch because after a lengthy
back and forth with Broadcom it turned out that the Supermicro server
shipped with an outdated firmware on the PEX 89xxx switches. Updating
to a newer firmware made the issue go away. The BIOS of the Supermicro
server was up to date, but it does not include the switch firmware.
I had to obtain a separate firmware update specifically for the switches
and install it through an EFI shell.
Obviously, a blanket wait of 1 second will work around Broadcom's virtual
switch feature as well, but it's not a proper fix.
The amdgpu driver contains a workaround for the virtual switch feature,
see 1dd2fa0e00f1. Obviously, this only helps amdgpu and other drivers
don't benefit. So we'd need to solve this for everyone in the PCI core,
as my patch below does. I kind of hate putting a workaround for this
vendor-specific misbehavior in core code, but problem reports caused
by it keep landing on my desk, so maybe I should amend the patch with
a commit message and submit it. But please test it first because perhaps
the issue you're seeing has some entirely different cause.
Thanks,
Lukas
-- >8 --