Re: commit 7ffb791423c7 breaks steam game

[Bert Karwatzki]

Am Dienstag, dem 25.03.2025 um 13:23 +0100 schrieb Christian König:
> Am 25.03.25 um 11:14 schrieb Bert Karwatzki:
> > My /proc/iomem  contans two memory areas of 8G size which are
> > belonging to PCI 0000:03:00.0, one of the is the BAR reported by dmesg
> > [ 0.312692] [ T1] pci 0000:03:00.0: BAR 0 [mem 0xfc00000000-0xfdffffffff 64bit pref]
>
> > the other one is "afe00000000-affffffffff : 0000:03:00.0" (in the case without nokaslr) which shifts
> > to "3ffe00000000-3fffffffffff : 0000:03:00.0" in the case with nokaslr.
>
> You need to figure out where that stuff is coming from.
>
> See below for another extremely odd thing.
>
> >
> > Here's /proc/iomem in the case without nokaslr:
> > 00000000-00000fff : Reserved
> > 00001000-0009ffff : System RAM
> > 000a0000-000fffff : Reserved
> >   000a0000-000dffff : PCI Bus 0000:00
> >   000f0000-000fffff : System ROM
> > 00100000-09bfefff : System RAM
> > 09bff000-0a000fff : Reserved
> > 0a001000-0a1fffff : System RAM
> > 0a200000-0a20efff : ACPI Non-volatile Storage
> > 0a20f000-e62edfff : System RAM
> > e62ee000-e63e1fff : Reserved
> > e63e2000-e87cafff : System RAM
> > e87cb000-e87cbfff : Reserved
> > e87cc000-e9e1ffff : System RAM
> > e9e20000-eb33efff : Reserved
> >   eb31e000-eb321fff : MSFT0101:00
> >   eb322000-eb325fff : MSFT0101:00
> > eb33f000-eb39efff : ACPI Tables
> > eb39f000-eb556fff : ACPI Non-volatile Storage
> > eb557000-ed1fefff : Reserved
> > ed1ff000-edffffff : System RAM
> > ee000000-efffffff : Reserved
> > f0000000-fcffffff : PCI Bus 0000:00
> >   f0000000-f7ffffff : PCI ECAM 0000 [bus 00-7f]
> >     f0000000-f7ffffff : pnp 00:00
> >   fc500000-fc9fffff : PCI Bus 0000:08
> >     fc500000-fc5fffff : 0000:08:00.7
> >       fc500000-fc5fffff : pcie_mp2_amd
> >     fc600000-fc6fffff : 0000:08:00.4
> >       fc600000-fc6fffff : xhci-hcd
> >     fc700000-fc7fffff : 0000:08:00.3
> >       fc700000-fc7fffff : xhci-hcd
> >     fc800000-fc8fffff : 0000:08:00.2
> >       fc800000-fc8fffff : ccp
> >     fc900000-fc97ffff : 0000:08:00.0
> >     fc980000-fc9bffff : 0000:08:00.5
> >       fc980000-fc9bffff : AMD ACP3x audio
> >         fc980000-fc990200 : acp_pdm_iomem
> >     fc9c0000-fc9c7fff : 0000:08:00.6
> >       fc9c0000-fc9c7fff : ICH HD audio
> >     fc9c8000-fc9cbfff : 0000:08:00.1
> >       fc9c8000-fc9cbfff : ICH HD audio
> >     fc9cc000-fc9cdfff : 0000:08:00.7
> >     fc9ce000-fc9cffff : 0000:08:00.2
> >       fc9ce000-fc9cffff : ccp
> >   fca00000-fccfffff : PCI Bus 0000:01
> >     fca00000-fcbfffff : PCI Bus 0000:02
> >       fca00000-fcbfffff : PCI Bus 0000:03
> >         fca00000-fcafffff : 0000:03:00.0
> >         fcb00000-fcb1ffff : 0000:03:00.0
> >         fcb20000-fcb23fff : 0000:03:00.1
> >           fcb20000-fcb23fff : ICH HD audio
> >     fcc00000-fcc03fff : 0000:01:00.0
> >   fcd00000-fcdfffff : PCI Bus 0000:07
> >     fcd00000-fcd03fff : 0000:07:00.0
> >       fcd00000-fcd03fff : nvme
> >   fce00000-fcefffff : PCI Bus 0000:06
> >     fce00000-fce03fff : 0000:06:00.0
> >       fce00000-fce03fff : nvme
> >   fcf00000-fcffffff : PCI Bus 0000:05
> >     fcf00000-fcf03fff : 0000:05:00.0
> >     fcf04000-fcf04fff : 0000:05:00.0
> >       fcf04000-fcf04fff : r8169
> > fd300000-fd37ffff : amd_iommu
> > fec00000-fec003ff : IOAPIC 0
> > fec01000-fec013ff : IOAPIC 1
> > fec10000-fec10fff : Reserved
> >   fec10000-fec10fff : pnp 00:04
> > fed00000-fed00fff : Reserved
> >   fed00000-fed003ff : HPET 0
> >     fed00000-fed003ff : PNP0103:00
> > fed40000-fed44fff : Reserved
> > fed80000-fed8ffff : Reserved
> >   fed81200-fed812ff : AMDI0030:00
> >   fed81500-fed818ff : AMDI0030:00
> >     fed81500-fed818ff : AMDI0030:00 AMDI0030:00
> > fedc0000-fedc0fff : pnp 00:04
> > fedc4000-fedc9fff : Reserved
> >   fedc5000-fedc5fff : AMDI0010:03
> >     fedc5000-fedc5fff : AMDI0010:03 AMDI0010:03
> > fedcc000-fedcefff : Reserved
> > fedd5000-fedd5fff : Reserved
> > fee00000-fee00fff : pnp 00:04
> > ff000000-ffffffff : pnp 00:04
> > 100000000-fee2fffff : System RAM
> >   825600000-8261fa1b1 : Kernel code
> >   826200000-82663dfff : Kernel rodata
> >   826800000-82692ef3f : Kernel data
> >   826eaf000-826ffffff : Kernel bss
>
> > fee300000-100fffffff : Reserved
>
> First it says that this range is reserved.
>
> > 1010000000-ffffffffff : PCI Bus 0000:00
>
> And this range here used for the PCI Bus.
>
> >   fc00000000-fe0fffffff : PCI Bus 0000:01
> >     fc00000000-fe0fffffff : PCI Bus 0000:02
> >       fc00000000-fe0fffffff : PCI Bus 0000:03
> >         fc00000000-fdffffffff : 0000:03:00.0 This is the usual BAR reported by dmesg (and lspci -vv)
> >         fe00000000-fe0fffffff : 0000:03:00.0
>
> And then it says that the PCIe devices are inside the reserved range.
>
> >   fe20000000-fe301fffff : PCI Bus 0000:08
> >     fe20000000-fe2fffffff : 0000:08:00.0
> >     fe30000000-fe301fffff : 0000:08:00.0
> >   fe30300000-fe304fffff : PCI Bus 0000:04
> >     fe30300000-fe303fffff : 0000:04:00.0
> >       fe30300000-fe303fffff : 0000:04:00.0
> >     fe30400000-fe30403fff : 0000:04:00.0
> >     fe30404000-fe30404fff : 0000:04:00.0
>
> > afe00000000-affffffffff : 0000:03:00.0 This is the memory which shifts with nokaslr
>
> To be honest that looks like a broken ACPI table to me, but it doesn't explain why this range here shifts when activating/deactivating nokaslr.
>
> Do we have some option to enable printing when ranges are added to the iomem reservation? If yes than we should probably do that and take a look at the dmesg again.
>
> Regards,
> Christian.
>
> >
> > Bert Karwatzki

As Balbir Singh found out this memory comes from amdkfd
(kgd2kfd_init_zone_device()) with CONFIG_HSA_AMD_SVM=y. The memory gets placed
by devm_request_free_mem_region() which places the memory at the end of the
physical address space (DIRECT_MAP_PHYSMEM_END). DIRECT_MAP_PHYSMEM_END changes
when using nokaslr and so the memory shifts.
 One can work around this by removing the GFR_DESCENDING flag from
devm_request_free_mem_region() so the memory gets placed right after the other
resources:

tail -n 15 /proc/iomem
1010000000-ffffffffff : PCI Bus 0000:00
  fc00000000-fe0fffffff : PCI Bus 0000:01
    fc00000000-fe0fffffff : PCI Bus 0000:02
      fc00000000-fe0fffffff : PCI Bus 0000:03
        fc00000000-fdffffffff : 0000:03:00.0
        fe00000000-fe0fffffff : 0000:03:00.0
  fe20000000-fe301fffff : PCI Bus 0000:08
    fe20000000-fe2fffffff : 0000:08:00.0
    fe30000000-fe301fffff : 0000:08:00.0
  fe30300000-fe304fffff : PCI Bus 0000:04
    fe30300000-fe303fffff : 0000:04:00.0
      fe30300000-fe303fffff : 0000:04:00.0
    fe30400000-fe30403fff : 0000:04:00.0
    fe30404000-fe30404fff : 0000:04:00.0
10000000000-101ffffffff : 0000:03:00.0

Bert Karwatzki