Re: commit 7ffb791423c7 breaks steam game

[Christian König]

Am 24.03.25 um 12:23 schrieb Bert Karwatzki:
> Am Sonntag, dem 23.03.2025 um 17:51 +1100 schrieb Balbir Singh:
>> On 3/22/25 23:23, Bert Karwatzki wrote:
>>> ...
>>> So why is use_dma32 enabled with nokaslr? Some more printk()s give this result:
>>>
>>> The GPUs:
>>> built-in:
>>> 08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c5)
>>> discrete:
>>> 03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c3)
>>>
>>> With nokaslr:
>>> [    1.266517] [    T328] dma_addressing_limited: mask = 0xfffffffffff bus_dma_limit = 0x0 required_mask = 0xfffffffff
>>> [    1.266519] [    T328] dma_addressing_limited: ops = 0000000000000000 use_dma_iommu(dev) = 0
>>> [    1.266520] [    T328] dma_direct_all_ram_mapped: returning true
>>> [    1.266521] [    T328] dma_addressing_limited: returning ret = 0
>>> [    1.266521] [    T328] amdgpu 0000:03:00.0: amdgpu: amdgpu_ttm_init: calling ttm_device_init() with use_dma32 = 0
>>> [    1.266525] [    T328] entering ttm_device_init, use_dma32 = 0
>>> [    1.267115] [    T328] entering ttm_pool_init, use_dma32 = 0
>>>
>>> [    3.965669] [    T328] dma_addressing_limited: mask = 0xfffffffffff bus_dma_limit = 0x0 required_mask = 0x3fffffffffff
>>> [    3.965671] [    T328] dma_addressing_limited: returning true
>>> [    3.965672] [    T328] amdgpu 0000:08:00.0: amdgpu: amdgpu_ttm_init: calling ttm_device_init() with use_dma32 = 1
>>> [    3.965674] [    T328] entering ttm_device_init, use_dma32 = 1
>>> [    3.965747] [    T328] entering ttm_pool_init, use_dma32 = 1
>>>
>>> Without nokaslr:
>>> [    1.300907] [    T351] dma_addressing_limited: mask = 0xfffffffffff bus_dma_limit = 0x0 required_mask = 0xfffffffff
>>> [    1.300909] [    T351] dma_addressing_limited: ops = 0000000000000000 use_dma_iommu(dev) = 0
>>> [    1.300910] [    T351] dma_direct_all_ram_mapped: returning true
>>> [    1.300910] [    T351] dma_addressing_limited: returning ret = 0
>>> [    1.300911] [    T351] amdgpu 0000:03:00.0: amdgpu: amdgpu_ttm_init: calling ttm_device_init() with use_dma32 = 0
>>> [    1.300915] [    T351] entering ttm_device_init, use_dma32 = 0
>>> [    1.301210] [    T351] entering ttm_pool_init, use_dma32 = 0
>>>
>>> [    4.000602] [    T351] dma_addressing_limited: mask = 0xfffffffffff bus_dma_limit = 0x0 required_mask = 0xfffffffffff
>>> [    4.000603] [    T351] dma_addressing_limited: ops = 0000000000000000 use_dma_iommu(dev) = 0
>>> [    4.000604] [    T351] dma_direct_all_ram_mapped: returning true
>>> [    4.000605] [    T351] dma_addressing_limited: returning ret = 0
>>> [    4.000606] [    T351] amdgpu 0000:08:00.0: amdgpu: amdgpu_ttm_init: calling ttm_device_init() with use_dma32 = 0
>>> [    4.000610] [    T351] entering ttm_device_init, use_dma32 = 0
>>> [    4.000687] [    T351] entering ttm_pool_init, use_dma32 = 0
>>>
>>> So with nokaslr the reuqired mask for the built-in GPU changes from 0xfffffffffff
>>> to 0x3fffffffffff which causes dma_addressing_limited to return true which causes
>>> the ttm_device init to be called with use_dma32 = true.
>> Thanks, this is really the root cause, from what I understand.

Yeah, completely agree.

>>
>>>  It also show that for the discreate GPU nothing changes so the bug does not occur
>>> there.
>>>
>>> I also was able to work around the bug by calling ttm_device_init() with use_dma32=false
>>> from amdgpu_ttm_init()  (drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c) but I'm not sure if this
>>> has unwanted side effects.
>>>
>>> int amdgpu_ttm_init(struct amdgpu_device *adev)
>>> {
>>> 	uint64_t gtt_size;
>>> 	int r;
>>>
>>> 	mutex_init(&adev->mman.gtt_window_lock);
>>>
>>> 	dma_set_max_seg_size(adev->dev, UINT_MAX);
>>> 	/* No others user of address space so set it to 0 */
>>> 	dev_info(adev->dev, "%s: calling ttm_device_init() with use_dma32 = 0 ignoring %d\n", __func__, dma_addressing_limited(adev->dev));
>>> 	r = ttm_device_init(&adev->mman.bdev, &amdgpu_bo_driver, adev->dev,
>>> 			       adev_to_drm(adev)->anon_inode->i_mapping,
>>> 			       adev_to_drm(adev)->vma_offset_manager,
>>> 			       adev->need_swiotlb,
>>> 			       false /* use_dma32 */);
>>> 	if (r) {
>>> 		DRM_ERROR("failed initializing buffer object driver(%d).\n", r);
>>> 		return r;
>>> 	}
>>>
>> I think this brings us really close, instead of forcing use_dma32 to false, I wonder if we need something like
>>
>> uin64_t dma_bits = fls64(dma_get_mask(adev->dev));
>>
>> to ttm_device_init, pass the last argument (use_dma32) as dma_bits < 32?

The handling is completely correct as far as i can see.

>>
>>
>> Thanks,
>> Balbir Singh
>>
> Do these address bits have to shift when using nokaslr or PCI_P2PDMA, I think
> this shift cause the increase of the required_dma_mask to 0x3fffffffffff?
>
> @@ -104,4 +104,4 @@
>        fe30300000-fe303fffff : 0000:04:00.0
>      fe30400000-fe30403fff : 0000:04:00.0
>      fe30404000-fe30404fff : 0000:04:00.0
> -afe00000000-affffffffff : 0000:03:00.0
> +3ffe00000000-3fffffffffff : 0000:03:00.0
>
> And what memory is this? It's 8G in size so it could be the RAM of the discrete
> GPU (which is at PCI 0000:03:00.0), but that is already here (part of
> /proc/iomem):
>
> 1010000000-ffffffffff : PCI Bus 0000:00
>   fc00000000-fe0fffffff : PCI Bus 0000:01
>     fc00000000-fe0fffffff : PCI Bus 0000:02
>       fc00000000-fe0fffffff : PCI Bus 0000:03
>         fc00000000-fdffffffff : 0000:03:00.0  GPU RAM
>         fe00000000-fe0fffffff : 0000:03:00.0
>
> lspci -v reports 8G of memory at 0xfc00000000 so I assmumed that is the GPU RAM.
> 03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23
> [Radeon RX 6600/6600 XT/6600M] (rev c3)
> 	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1313
> 	Flags: bus master, fast devsel, latency 0, IRQ 107, IOMMU group 14
> 	Memory at fc00000000 (64-bit, prefetchable) [size=8G]
> 	Memory at fe00000000 (64-bit, prefetchable) [size=256M]
> 	Memory at fca00000 (32-bit, non-prefetchable) [size=1M]
> 	Expansion ROM at fcb00000 [disabled] [size=128K]

Well when you set nokaslr then that moves the BAR address of the dGPU above the limit the integrated GPU can access on the bus (usually 40 bits).

Because of this the integrated GPU starts to fallback to system memory below the 4GB limit to make sure that the stuff is always accessible by everyone.

Since the memory below 4GB is very very limited we are now starting to constantly swap things in and out of that area. Basically completely killing the performance of your Steam game.

As far as I can see till that point the handling is completely intentional and working as expected.

The only thing which eludes me is why setting nokaslr changes the BAR of the dGPU? Can I get the full dmesg with and with nokasl?

Thanks in advantage,
Christian.

>
> Bert Karwatzki
>