Re: [PATCH v3 00/18] PCI: rcar-gen4: Recover from link down and route Root Port interrupts

[Koichiro Den]

On Tue, Oct 06, 2026 at 05:46:20PM +0900, Koichiro Den wrote:
> Hi,
> 
> This series improves error handling in the pcie-rcar-gen4 driver, which
> supports PCIe controllers in R-Car Gen4 SoCs and the PCIe4 controller
> in R-Car Gen5 SoCs. It fixes unexpected link-down handling so the host
> doesn't hang, and wires up missing Root Port interrupts (AER, PME,
> bandwidth notifications) so port services actually work. I tested on
> an R-Car S4 Spider (r8a779f0); Marek tested the link state fix and the
> Root Port AER and bandwidth paths on an R-Car V4H Sparrow Hawk.
> 
> A few hardware quirks made this tricky:
> 
> 1. PCIEINTSTS0 link-up bits don't track the actual link state. The
>    driver never noticed when the link dropped and kept trying config
>    accesses on the dead link. Patch 3 combines the APP link-up event
>    check from Figure 104.5 with the PORT_LINK_DEBUG1 live link check,
>    which patch 2 factors out of the DWC core. Startup clears the APP
>    latches before enabling LTSSM, and the DWC core polls the combined
>    condition on the RC side.
> 
> 2. On S4, a DBI access immediately after an unexpected link down can
>    hang the host. Commit 0056d29f8c1b ("PCI: rcar-gen4: Assure reset
>    occurs before DBI access") describes an SError on V4H after reset
>    deassertion, but whether the S4 hang shares the same underlying cause
>    has not been verified.
> 
>    Adding a delay before the DBI access avoided the hang in my tests,
>    but that alone would not provide link-down recovery. Also, the reset
>    request shares intreq_pcim_sub with iMSI-RX, while AER arrives on
>    another IRQ. Delaying AER dispatch alone would therefore leave the
>    MSI handler exposed, and both paths would still need coordination
>    with the controller reset.
> 
>    The driver handles the reset request itself and schedules
>    pci_host_handle_link_down(), as the rockchip and qcom drivers do.
>    This also provides recovery with older DTs that have no "aer"
>    interrupt, without relying on AER to initiate it. The reset callback
>    shares the reset sequence used at probe, including the reset-status
>    readback and delay added by 0056d29f8c1b.
> 
>    This covers the interrupt paths that fire on link down. Port service
>    threads or work already queued or running at that point are not
>    stopped.
> 
> 3. Root Port interrupts only trigger APP status bits on platform IRQs.
>    The controller appears to lack SII2MSI, so Root Port MSIs never reach
>    the GIC ITS. The Root Port's INTx is routed to intreq_pcim_sub as
>    well, which the driver holds, so port services can't request it
>    either. This series works around it by hiding Root Port MSI caps
>    across the board and emulating INTx using a virtual IRQ domain, fed
>    by the "aer" IRQ and intreq_pcim_sub.
> 
> Patch 1 is only loosely related: it adds Renesas to the RAS DES VSEC
> list so the DWC debugfs error injection works on R-Car. I used it to
> test the Root Port AER path (see below) and included it here for that
> reason. Happy to send it separately if preferred.
> 
> Based on next-20261002, with Marek's PM ops v5 patch applied first:
> https://lore.kernel.org/r/20261004011851.866833-1-marek.vasut+renesas@xxxxxxxxxxx/

Turns out patch 4 ("dt-bindings: PCI: rcar-gen4: Add optional "aer" interrupt")
also needs to be rebased on top of Marek's patch [2].
(Thanks for the off-list heads-up, Marek!)

I think I'll do that when another respin is needed, or once the series is
otherwise acked. At least Sashiko hasn't complained about v3.

[2] https://lore.kernel.org/r/20261003215710.792518-1-marek.vasut+renesas@xxxxxxxxxxx/

Best regards,
Koichiro Den

> 
> Note: backward compatibility with older DTs is kept. Without the "aer"
>       interrupt, only Root Port AER remains unavailable. See the Testing
>       section below.
> 
> Retesting with v3
> -----------------
> 
> Setup: R-Car S4 Spider (RC) linked to another S4 Spider running the
>        pci-epf-test endpoint. pci_endpoint_test is bound on the RC side.
> 
>   1. Link down / recovery. On the EP side, toggle the endpoint controller
>      off and on. The short pause keeps the endpoint away long enough for
>      the RC to notice, but brings it back within the reset window so
>      recovery can succeed. Adopted the test approach from [1]. Make sure
>      that the unused function 0000:01:00.1 was removed on the RC before
>      testing.
> 
>      # cd /sys/kernel/config/pci_ep
>      # echo 0 > controllers/e65d0000.pcie-ep/start
>      # sleep 0.1
>      # echo 1 > controllers/e65d0000.pcie-ep/start
> 
>      Expected on the RC dmesg:
>        pcieport 0000:00:00.0: Recovering Root Port due to Link Down
>        pcieport 0000:00:00.0: Root Port has been reset
>        pcieport 0000:00:00.0: AER: device recovery successful
> 
>      and the "msi" (intreq_pcim_sub) interrupt count going up in
>      /proc/interrupts. Without this series nothing shows up here: the
>      link comes back on its own once the endpoint returns, but the RC
>      never notices the outage and the endpoint is left unconfigured
>      (see 4). Config accesses issued while the link is down hang the
>      host.
> 
>      [1] https://lore.kernel.org/r/abFMa6DCGGLUHddA@fedora/
> 
>   2. Bandwidth notification. On the RC, retrain the link:
> 
>      # setpci -s 00:00.0 CAP_EXP+0x10.w=0x0c23
> 
>      Expected:
>      - the virtual Root Port IRQ (rcar-gen4-rp in /proc/interrupts,
>        shared by PCIe PME, aerdrv and PCIe bwctrl) fires once
>      - bwctrl clears LnkSta.LBMS
>        (setpci -s 00:00.0 CAP_EXP+0x12.w reads 0x2024 again).
> 
>      Before the series LnkSta read 0xe024 afterwards, LBMS and LABS
>      stuck.
> 
>      This test does not generate a PME. The PME service shares the
>      virtual IRQ, but its path was not exercised separately.
> 
>   3. Root Port AER. On the RC, inject an LCRC error with the DWC debugfs
>      (patch 1) and issue one config read so a TLP actually goes out:
> 
>      # cd /sys/kernel/debug/dwc_pcie_e65d0000.pcie/rasdes_err_inj
>      # echo 1 > rx_lcrc            # error detected by the Root Port
>      # setpci -s 01:00.0 VENDOR_ID.w
>      # echo 1 > tx_lcrc            # error detected by the endpoint,
>      # setpci -s 01:00.0 VENDOR_ID.w    # reported back with ERR_COR
> 
>      Expected on the RC dmesg, respectively:
>       pcieport 0000:00:00.0: PCIe Bus Error: severity=Correctable
>       pcieport 0000:00:00.0:   [ 6] BadTLP | Receiver | Data Link Layer
> 
>       pcieport 0000:00:00.0: AER: Correctable Error message received from 0000:01:00.0
>       pci-endpoint-test 0000:01:00.0: PCIe Bus Error: severity=Correctable
>       pci-endpoint-test 0000:01:00.0:   [ 6] BadTLP | Receiver | Data Link Layer
> 
>      plus the virtual Root Port IRQ count and aer_rootport_total_err_cor
>      going up by one each time. The link stays up throughout, the DLL
>      retry recovers the TLP. Before the series nothing is reported.
> 
>   4. Regression check. Run pci_endpoint_test after step 1. PASS/FAIL/SKIP
>      counts match a run without step 1.
> 
> Configurations:
> 
>   a. Without this series**
>   b. GIC ITS, DT with the new "aer" interrupt (this series)
>   c. GIC ITS, DT without "aer" (b43aa6a6ebe8 ("arm64: dts: renesas:
>      r8a779f0: Add GICv3 ITS and update PCIe nodes") or later)
>   d. iMSI-RX, DT before b43aa6a6ebe8 (no msi-parent, no "aer")
> 
> Result:
>              recovery   bwctrl       RP AER    pcitest
>   ---------------------------------------------------------
>   a.           none         no         no      all FAIL
>   b.            ok          ok         ok      no change
>   c.            ok          ok         n/a*    no change
>   d.            ok          ok         n/a*    no change
> 
>   *  Root Port AER needs the "aer" interrupt; without it the behaviour is
>      unchanged from before the series.
>   ** Only patch 1 applied on top of the base, so the same debugfs error
>      injection could be used for the comparison. pci_endpoint_test fails
>      across the board there because nothing restores the endpoint after
>      the toggle.
> 
> Best regards,
> Koichiro
> ---
> Changes in v3:
>   - Rebase onto next-20261002 with Marek's PM ops v5 applied first.
>   - Keep parent IRQs registered across suspend/resume to fix the
>     unbalanced enable Geert reported. Request them in probe with
>     IRQF_NO_AUTOEN and enable/disable them in .init()/.deinit().
>   - Use a freezable workqueue for recovery and reject suspend while
>     reinitialization is pending. Re-arm Root Port events only after
>     resume has set up the link.
>   - Factor out the PORT_LINK_DEBUG1 check (patch 2) and reuse it in
>     patch 3. (Marek)
>   - Replace .reinit() with .configure, folding v2 patch 8 into patch 9.
>     Drop the reset mutex and rely on PCI core serialization. (Marek)
>   - Free the MSI domain after host .deinit(), so the driver can stop
>     its parent IRQ first (patch 10). This closes the known gap in v2.
>   - Reset the controller if recovery finds no Root Port, so a successful
>     reset can restore interrupt delivery.
>   - Drop the raw lock around virtual IRQ dispatch. Preserve MSI-form
>     latches in irq_ack() while the IRQ is in progress or disabled,
>     avoiding lost notifications without nesting port service locks.
>   - Add patch 7's Fixes tag, clarify SoC generations, separate guard()
>     from goto-based cleanup, and simplify IRQ return statements.
>     (Marek, Sashiko)
>   - Add the V4H and V4M DTS patches 17 and 18. (Marek)
>   - Collect Reviewed-by (Marek, Krzysztof) and Tested-by (Marek) tags.
>     Drop Marek's Reviewed-by from the reworked patches 12 and 14.
> 
> Changes in v2:
>   - Rebased onto next-20260925, including Marek's R-Car X5H support.
>   - Keep .link_up() and require both the APP link-up events and the
>     PORT_DEBUG1 live link check. Clear the APP latches before enabling
>     LTSSM and reuse the DWC core's polling on the RC side. (Marek)
>   - Let the R-Car driver own intreq_pcim_sub in all configurations (new
>     patch 10) and check the reset request from its own handler instead
>     of hooking into the DWC chained handler. Replace the pre_msi_irq host
>     op from v1 with an export of dw_handle_msi_irq() (patch 4). (Marek)
>   - Keep the reset_control_status() check before asserting the power
>     reset. (Marek)
>   - Separate controller reinitialization from .init()/.deinit() and split
>     out preparatory changes.
>   - Use bool flags instead of a state bitmask. (Marek)
>   - Keep Root Port AER notifications masked until .post_init, after
>     enumeration. In v1, they could be enabled during port-service probing.
>   - Keep APP interrupt sources masked if the Root Port reset callback
>     fails, regardless of the reset trigger.
>   - Hold a reference on the Root Port in the link-down recovery work.
>   - Clear only the MSI-form Root Port latches through PCIEINTSTS0CLR;
>     the INTx bits are reserved there.
>   - Serialize dispatches to the virtual Root Port IRQ from its two
>     parent interrupts.
>   - Collected Marek's Reviewed-by on patches 1, 3 and 15.
> 
> v2: https://lore.kernel.org/r/20260928165230.3397664-1-den@xxxxxxxxxxxxx/
> v1: https://lore.kernel.org/r/20260918032038.2216471-1-den@xxxxxxxxxxxxx/
> 
> 
> Koichiro Den (18):
>   PCI: dwc: Add Renesas to the RAS DES VSEC list
>   PCI: dwc: Factor out the PORT_LINK_DEBUG1 link-up check
>   PCI: rcar-gen4: Check live link status in link_up()
>   dt-bindings: PCI: rcar-gen4: Add optional "aer" interrupt
>   PCI: dwc: Export dw_handle_msi_irq()
>   PCI: rcar-gen4: Move deinitialization helpers before SoC
>     initialization
>   PCI: rcar-gen4: Assert resets when Gen5 SoC PHY initialization fails
>   PCI: rcar-gen4: Separate hardware setup from resource acquisition
>   PCI: rcar-gen4: Add Root Port reset support
>   PCI: dwc: Free the MSI domain after the host .deinit() callback
>   PCI: rcar-gen4: Take over the iMSI-RX interrupt
>   PCI: rcar-gen4: Recover the Root Port on link down
>   PCI: dwc: Let glue drivers hide the Root Port MSI capabilities
>   PCI: rcar-gen4: Route Root Port AER to a virtual Root Port IRQ
>   PCI: rcar-gen4: Route Root Port PME and bandwidth notifications
>   arm64: dts: renesas: r8a779f0: Describe the PCIe AER interrupts
>   arm64: dts: renesas: r8a779g0: Describe the PCIe AER interrupts
>   arm64: dts: renesas: r8a779h0: Describe the PCIe AER interrupt
> 
>  .../bindings/pci/rcar-gen4-pci-host.yaml      |  10 +-
>  arch/arm64/boot/dts/renesas/r8a779f0.dtsi     |  10 +-
>  arch/arm64/boot/dts/renesas/r8a779g0.dtsi     |  10 +-
>  arch/arm64/boot/dts/renesas/r8a779h0.dtsi     |   5 +-
>  .../pci/controller/dwc/pcie-designware-host.c |  38 +-
>  drivers/pci/controller/dwc/pcie-designware.c  |  15 +-
>  drivers/pci/controller/dwc/pcie-designware.h  |   2 +
>  drivers/pci/controller/dwc/pcie-rcar-gen4.c   | 723 ++++++++++++++++--
>  include/linux/pcie-dwc.h                      |   2 +
>  9 files changed, 724 insertions(+), 91 deletions(-)
> 
> 
> base-commit: f0406245cb9855e6318335a8a223551354291a46
> prerequisite-patch-id: 5b4c9da1333342baa90cf6a86dc879e8382d3f65
> -- 
> 2.51.0
> 
>