Re: [BUG 5.14] arm64/mm: dma memory mapping fails (in some cases)
From: Alex Bee
Date: Tue Aug 24 2021 - 16:14:07 EST
Hi Mike,
thanks for your reply
Am 24.08.21 um 20:28 schrieb Mike Rapoport:
On Tue, Aug 24, 2021 at 06:37:41PM +0100, Catalin Marinas wrote:
Hi Alex,
Thanks for the report.
On Tue, Aug 24, 2021 at 03:40:47PM +0200, Alex Bee wrote:
it seems there is a regression in arm64 memory mapping in 5.14, since it
fails on Rockchip RK3328 when the pl330 dmac tries to map with:
[ 8.921909] ------------[ cut here ]------------
[ 8.921940] WARNING: CPU: 2 PID: 373 at kernel/dma/mapping.c:235 dma_map_resource+0x68/0xc0
[ 8.921973] Modules linked in: spi_rockchip(+) fuse
[ 8.921996] CPU: 2 PID: 373 Comm: systemd-udevd Not tainted 5.14.0-rc7 #1
[ 8.922004] Hardware name: Pine64 Rock64 (DT)
[ 8.922011] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
[ 8.922018] pc : dma_map_resource+0x68/0xc0
[ 8.922026] lr : pl330_prep_slave_fifo+0x78/0xd0
[ 8.922040] sp : ffff800012102ae0
[ 8.922043] x29: ffff800012102ae0 x28: ffff000005c94800 x27: 0000000000000000
[ 8.922056] x26: ffff000000566bd0 x25: 0000000000000001 x24: 0000000000000001
[ 8.922067] x23: 0000000000000002 x22: ffff000000628c00 x21: 0000000000000001
[ 8.922078] x20: ffff000000566bd0 x19: 0000000000000001 x18: 0000000000000000
[ 8.922089] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
[ 8.922100] x14: 0000000000000277 x13: 0000000000000001 x12: 0000000000000000
[ 8.922111] x11: 0000000000000001 x10: 00000000000008e0 x9 : ffff800012102a80
[ 8.922123] x8 : ffff000000d14b80 x7 : ffff0000fe7b12f0 x6 : ffff0000fe7b1100
[ 8.922134] x5 : fffffc000000000f x4 : 0000000000000000 x3 : 0000000000000001
[ 8.922145] x2 : 0000000000000001 x1 : 00000000ff190800 x0 : ffff000000628c00
[ 8.922158] Call trace:
[ 8.922163] dma_map_resource+0x68/0xc0
[ 8.922173] pl330_prep_slave_sg+0x58/0x220
[ 8.922181] rockchip_spi_prepare_dma+0xd8/0x2c0 [spi_rockchip]
[ 8.922208] rockchip_spi_transfer_one+0x294/0x3d8 [spi_rockchip]
[...]
Note: This does not relate to the spi driver - when disabling this device in
the device tree it fails for any other (i2s, for instance) which uses dma.
Commenting out the failing check at [1], however, helps and the mapping
works again.
Do you know which address dma_map_resource() is trying to map (maybe
add some printk())? It's not supposed to map RAM, hence the warning.
Random guess, the address is 0xff190800 (based on the x1 above but the
regs might as well be mangled).
0xff190800 will cause this warning for sure. It has a memory map, but it is
not RAM so old version of pfn_valid() would return 0 and the new one
returns 1.
I tried to follow the recent changes for arm64 mm which could relate to the
check failing at [1] and reverting
commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID")
helps and makes it work again, but I'm 100% uncertain if that commit is
really the culprit.
Note, that the firmware (legacy u-boot) injects memory configuration in the
device tree as follows:
/memreserve/ 0x00000000fcefc000 0x000000000000d000;
/ {
..
compatible = "pine64,rock64\0rockchip,rk3328";
..
memory {
reg = <0x00 0x200000 0x00 0xfee00000 0x00 0x00 0x00 0x00>;
device_type = "memory";
};
..
}
Either pfn_valid() gets confused in 5.14 or something is wrong with the
DT. I have a suspicion it's the former since reverting the above commit
makes it disappear.
I think pfn_valid() actually behaves as expected but the caller is wrong
because pfn_valid != RAM (this applies btw to !arm64 as well).
/* Don't allow RAM to be mapped */
if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr))))
return DMA_MAPPING_ERROR;
Alex, can you please try this patch:
diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
index 2b06a809d0b9..4715e9641a29 100644
--- a/kernel/dma/mapping.c
+++ b/kernel/dma/mapping.c
@@ -232,7 +232,7 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr,
return DMA_MAPPING_ERROR;
/* Don't allow RAM to be mapped */
- if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr))))
+ if (WARN_ON_ONCE(!memblock_is_memory(phys_addr)))
return DMA_MAPPING_ERROR;
if (dma_map_direct(dev, ops))
Nope, doesn't help:
[ 8.353879] dma_map_resource Failed to map address 0xff190800
[ 8.353886] dma_map_resource pfn_valid(PHYS_PFN(0xff190800)): 1
[ 8.353892] dma_map_resource memblock_is_memory(0xff190800): 0
If understand the comment for that check correct, that we _don't_ want
RAM to be mapped - shoudn't that be:
+ if (WARN_ON_ONCE(memblock_is_memory(phys_addr)))
?
Alex
So: there is a "hole" in the mappable memory and reading the commit message
of
commit a7d9f306ba70 ("arm64: drop pfn_valid_within() and simplify
pfn_valid()")
suggests, there was a change for that case recently.
I think the change from the arm64 pfn_valid() to the generic one is
avoiding the call to memblock_is_memory(). I wonder whether pfn_valid()
returns true just because we have a struct page available but the memory
may have been reserved.
Cc'ing Mike R.
I also noticed there is a diff in the kernel log regarding memory init up
until 5.13.12 it says
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] On node 0 totalpages: 1043968
[ 0.000000] DMA zone: 16312 pages used for memmap
[ 0.000000] DMA zone: 0 pages reserved
[ 0.000000] DMA zone: 1043968 pages, LIFO batch:63
In contrary in 5.14-rc7 it says:
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] On node 0, zone DMA: 512 pages in unavailable ranges
[ 0.000000] On node 0, zone DMA: 4096 pages in unavailable ranges
(note the "unavailable ranges")
I'm uncertain again here, if that diff is expected behavior because of those
recent mm changes for arm64.
After reverting
commit 16c9afc77660 ("arm64/mm: drop HAVE_ARCH_PFN_VALID")
the log changes to
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] DMA32 empty
[ 0.000000] Normal empty
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000200000-0x00000000feffffff]
[ 0.000000] Initmem setup node 0 [mem
0x0000000000200000-0x00000000feffffff]
(no DMA zones here)
As you might have noticed I have _zero_ clue about memory mapping and dma
subsystem - so let me know if there is any more information needed for that
and thanks for your help.
Adding Robin as well, he has a better clue than us on DMA ;).
Alex
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/dma/mapping.c?id=e22ce8eb631bdc47a4a4ea7ecf4e4ba499db4f93#n235
--
Catalin