Re: [PATCH v6 2/2] iommu/arm-smmu-v3: Default queue depths to one page in a kdump kernel

[Yuanhe Shu]

Tested-by: Yuanhe Shu <xiangzao@xxxxxxxxxxxxxxxxx>

We hit the same problem on our arm64 platform: the kdump capture kernel
runs out of memory because the SMMUv3 queues are still allocated at full
hardware size. We had started looking into it ourselves, then came across
this series, so we tested it on the affected hardware rather than prepare
a duplicate fix.

It resolves the problem for us - details below.

Test platform: an arm64 server, 128 cores, 2 NUMA nodes, 512 GiB RAM,
exposing 6 SMMUv3 instances. 64K-page kernel (PAGE_SHIFT=16) based on
Linux 7.0.14 with the series applied on top; crashkernel reservation is
512 MiB.

On this platform kdump FAILS without the patch, and the series fixes it.

Before - capture kernel WITHOUT the series:
  All 6 SMMUv3 instances still allocate the hardware-maximum queues:
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for cmdq
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for evtq
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 524288 entries for priq
    (... same for all 6 instances)
  That is 8 + 16 + 8 = 32 MiB per instance, 192 MiB total - 37.5% of the
  512 MiB reservation. The capture kernel OOMs before makedumpfile runs:
    swapper/0 invoked oom-killer: gfp_mask=0x2040cc0(GFP_KERNEL|...), order=0
    Out of memory and no killable processes...
    Rebooting in 10 seconds..
  => no vmcore is saved.

After - capture kernel WITH the series:
  All three queues are floored to one page per instance:
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 4096 entries for cmdq
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 2048 entries for evtq
    arm-smmu-v3 arm-smmu-v3.0.auto: allocated 4096 entries for priq
    (... same for all 6 instances)
  => ~1.1 MiB total (~191 MiB freed). All 6 instances re-probe cleanly
  (no -ENXIO, no allocation failure) and a 608 MB vmcore is saved.

Same machine, same 512 MiB reservation; the only difference is the v6
series, so this is a controlled before/after.

Thanks,
Yuanhe