[PATCH 0/2] iommu/arm-smmu-v3: SMMU hitting kdump mid-air

[Breno Leitao]

I've found that kdump does not work on some ARM host, such as AWS
Graviton metal, and the SMMU is what seems to kills it.

The capture kernel dies about a second after it resets the SMMU, part
way through PCI enumeration, and takes the machine with it.

There is no panic and no oops: the console stops mid-line and the box
reboots, leaving no vmcore, no log and nothing on disk. Crashes on this
hardware mostly produce no dump at all.

This is my theory, from an "outsider view":

1) A crash kexec does not run .device_shutdown(), so the capture
   kernel boots on a machine whose devices are still running.

2) They keep translating through the stream table the crashed kernel
   programmed, which sits in memory the capture kernel never touches.

3) arm_smmu_device_reset() installs a fresh table with no entry for
   those StreamIDs, so their in-flight DMA raises C_BAD_STREAMID. (seen
   when I cleared CR0_EVTQEN)

4) When the platform reports hardware errors to firmware first,
   firmware answers by resetting the machine.

If my  kdump initrd has no driver for the two ENA NICs, so nothing
resets them and they keep DMAing for as long as the capture kernel runs.

Things that will not crash/interrupt kdump:

   a) Get the ENA driver NIC into the kdump. This is costly.
   b) Removing ENA driver (or PCI device) before the crash

Debugging further, I  found that faults are invisible by default,
because the capture kernel clears CR0_EVTQEN. With the event queue left
enabled, one device accounts for all of them:

  [   36.191293] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting...
  [   36.418009] event: C_BAD_STREAMID client: (unassigned sid) sid: 0x32d00 ssid: 0x0
  [   36.706345] event: C_BAD_STREAMID client: (unassigned sid) sid: 0x32d00 ssid: 0x0
  [   37.313703] pci 0003:02:02.2: Adding

with the platform resetting mid-line. sid 0x32d00 is 0003:2d:00.0, one of
ENA NICs.

My lovely robot and I came with two patches that solved the problem,
from a practical perspective and I want to share what has been tested.

Patch 1 carries the previous kernel's L1 descriptors into the new KDUMP table, so
those streams keep translating.

Patch 2 stops the core discarding that again: iommu_dma_init() already
enables deferred attach for a capture kernel, but the core only honours
it for drivers implementing .is_attach_deferred, which until now meant
amd and intel. Without it iommu_setup_default_domain() attaches
a default domain at probe and the fault returns as F_TRANSLATION.

The x86 IOMMU drivers solve the same/similar problem the same way; see
commit 38e5f33ee3596 ("iommu/amd: Reuse device table for kdump").

Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
---
Breno Leitao (2):
      iommu/arm-smmu-v3: inherit the previous kernel's stream table in kdump
      iommu/arm-smmu-v3: defer attach in kdump so inherited entries survive probe

 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 104 ++++++++++++++++++++++++----
 1 file changed, 91 insertions(+), 13 deletions(-)
---
base-commit: f2bfbc3554ca6919484030729424b9dee2942d24
change-id: 20260911-smmu_fix_aws-95f486d8ee5d

Best regards,
--  
Breno Leitao <leitao@xxxxxxxxxx>