Re: Regression: memory corruption on Atmel SAMA5D31

[Tudor.Ambarus]

On 7/28/22 10:45, Tudor.Ambarus@xxxxxxxxxxxxx wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
> 
> On 7/13/22 19:01, Tudor.Ambarus@xxxxxxxxxxxxx wrote:
>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the content is safe
>>
>> Hi, Peter,
>>
>> Thanks for the patience. I was still out of office last week,
>> but now I have some news.
>>
>> On 6/27/22 19:53, Tudor.Ambarus@xxxxxxxxxxxxx wrote:
>>> I think these are the last less invasive
>>> changes that I try, I'll have to rewrite the logic anyway.
>>
>> I've chopped the driver to use virt-dma (check [1]). It's not clean, but
>> it works and one can see how the logic is changed. Unfortunately the mem
>> corruption is still present on high loads. Maybe it's a coherency problem.
>> I need more time on it. Will get back to you.
>>
>> Cheers,
>> ta
>>
>> [1] To github.com:ambarus/linux-0day.git
>>    a7351e6f4c12..1557e0df0fd0  at-hdmac-virt-dma -> at-hdmac-virt-dma
> 
> Hi, Peter,
> 
> Does this [1] one line patch solve the mem corruption on your side?
> Even if yes, there are still bugs in at-hdmac that can be squashed by
> using virt-dma. I'd like to follow up with patches that integrate
> virt-dma logic in at-hdmac.
> 
> Cheers,
> ta
> 
> [1] https://lore.kernel.org/linux-mtd/20220728074014.145406-1-tudor.ambarus@xxxxxxxxxxxxx/T/#u

Hi, Peter,

Looks like I've already caught an oops in at-hdmac driver when not using virt-dma,
see below. Would you please test with all the patches from [2] instead of just
using the patch from [1]? I've run stress tests over night by using [2] and
everything went fine on my side.

Cheers,
ta

[2] To github.com:ambarus/linux-0day.git
 * [new branch]                at-hdmac-virt-dma-2nd-iteration -> at-hdmac-virt-dma-2nd-iteration

root@sama5d3-xplained:~# while :; do cat testfile | sha256sum; done
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
dad5c65bf4c2a009ad6bad0d279096841de91631636057dd5dc47d0a45f9ec84  -
[ 4115.100000] ------------[ cut here ]------------
[ 4115.100000] kernel BUG at drivers/dma/dmaengine.h:54!
[ 4115.100000] Internal error: Oops - BUG: 0 [#1] ARM
[ 4115.100000] CPU: 0 PID: 480 Comm: cat Not tainted 5.18.0-rc7+ #40
[ 4115.100000] Hardware name: Atmel SAMA5
[ 4115.100000] PC is at atc_chain_complete+0x150/0x168
[ 4115.100000] LR is at atc_advance_work+0x78/0x184
[ 4115.100000] pc : [<c03d1fe4>]    lr : [<c03d21d4>]    psr: 60030093
[ 4115.100000] sp : c4e597e8  ip : c0d72800  fp : 00000001
[ 4115.100000] r10: c4e59848  r9 : c17c4300  r8 : 60030013
[ 4115.100000] r7 : c0d72c58  r6 : c17c4300  r5 : c0d72c88  r4 : c0d72be8
[ 4115.100000] r3 : 00000000  r2 : 00000000  r1 : c0d72c3c  r0 : c0d72be8
[ 4115.100000] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment none
[ 4115.100000] Control: 10c53c7d  Table: 20ae0059  DAC: 00000051
[ 4115.100000] Register r0 information: slab kmalloc-2k start c0d72800 pointer offset 1000 size 2048
[ 4115.100000] Register r1 information: slab kmalloc-2k start c0d72800 pointer offset 1084 size 2048
[ 4115.100000] Register r2 information: NULL pointer
[ 4115.100000] Register r3 information: NULL pointer
[ 4115.100000] Register r4 information: slab kmalloc-2k start c0d72800 pointer offset 1000 size 2048
[ 4115.100000] Register r5 information: slab kmalloc-2k start c0d72800 pointer offset 1160 size 2048
[ 4115.100000] Register r6 information: slab task_struct start c17c4300 pointer offset 0
[ 4115.100000] Register r7 information: slab kmalloc-2k start c0d72800 pointer offset 1112 size 2048
[ 4115.100000] Register r8 information: non-paged memory
[ 4115.100000] Register r9 information: slab task_struct start c17c4300 pointer offset 0
[ 4115.100000] Register r10 information: 2-page vmalloc region starting at 0xc4e58000 allocated at kernel_clone+0xb4/0x358
[ 4115.100000] Register r11 information: non-paged memory
[ 4115.100000] Register r12 information: slab kmalloc-2k start c0d72800 pointer offset 0 size 2048
[ 4115.100000] Process cat (pid: 480, stack limit = 0x511e7a27)
[ 4115.100000] Stack: (0xc4e597e8 to 0xc4e5a000)
[ 4115.100000] 97e0:                   c4e59848 00000001 00000010 4f21ffb9 c0d72be8 c0d72c88
[ 4115.100000] 9800: c17c4300 00000002 00000800 c03d21d4 00000800 c17c4300 c4e59848 4f21ffb9
[ 4115.100000] 9820: 00000010 c0e7d100 00000000 20f28000 00000002 00000800 c17c4300 c0467490
[ 4115.100000] 9840: 00000003 00000000 00000001 c4e5984c c4e5984c 4f21ffb9 c0e6e040 c0f28000
[ 4115.100000] 9860: 00000800 c0e7d100 c090e8bc c090e99c c0b41d18 c0467688 00000002 00000000
[ 4115.100000] 9880: c4e59888 00000000 00000000 c4e59990 c08f4bd4 00000014 00000001 c04677fc
[ 4115.100000] 98a0: c4e59990 c045a918 00000000 00000000 00000000 00000004 c0833744 00000001
[ 4115.100000] 98c0: 00000000 00000000 00000001 c4e59990 c4e59990 c4e59990 00000001 c0832df4
[ 4115.100000] 98e0: c0b41d18 00000000 c4e59984 c0e6e050 c17c4300 c090e904 c4e59990 00000001
[ 4115.100000] 9900: 00000000 c4e59990 00000001 00000000 00000000 c4e59990 00000001 00000000
[ 4115.100000] 9920: c4e59990 00000001 00000000 00000000 c4e59990 00000001 00000000 c4e59990
[ 4115.100000] 9940: 00000000 00000000 00000000 4f21ffb9 00000200 00000000 c0e6e050 c17c4300
[ 4115.100000] 9960: c0f28000 c0464d54 00000800 c0831be0 c4e59ac4 c04595b4 c17c4300 00000000
[ 4115.100000] 9980: c4e59ac4 00000000 c4e59990 00000001 00000002 00000800 c0f28000 4f21ff00
[ 4115.100000] 99a0: 00000000 4f21ffb9 00000000 c0e6e050 00000000 c0f28000 00000000 c0467b38
[ 4115.100000] 99c0: 00000000 c0466dc4 00000000 ffffffff c0e6e050 c0e6e050 00000480 00000000
[ 4115.100000] 99e0: 00000000 c0467b54 00000000 00000000 c4e59ac4 c04569b4 00001030 00000000
[ 4115.100000] 9a00: 00000000 00000000 c4e59a4c c0f28000 00000800 00000001 0001e872 c0e6e12c
[ 4115.100000] 9a20: 00000000 c1704bb0 c0f28000 00000480 00000000 0001e872 00000000 00000000
[ 4115.100000] 9a40: 00000000 00000040 00000000 c0b41d18 00000134 00000000 00000002 00000000
[ 4115.100000] 9a60: 00000000 c0e6e050 c0f67000 0ec38450 00000000 00000134 00000000 00000000
[ 4115.100000] 9a80: c4e59ac4 c0448d8c c4e59ac4 c171f0c0 c0913590 c17c4300 c4e59ac4 c4e59b38
[ 4115.100000] 9aa0: c1704000 0ec38450 00000000 00001030 c0f67000 c0448eac c4e59ac4 c01b6d98
[ 4115.100000] 9ac0: 000000bc 00000000 00001030 00000000 00000000 00000000 00000000 c1704000
[ 4115.100000] 9ae0: 00000000 4f21ffb9 c1410800 c08dcbf8 c0fee000 00001030 00000004 00000761
[ 4115.100000] 9b00: 00018450 c1704000 c0913590 c0473c24 00001030 c4e59b38 c1704000 c0470f9c
[ 4115.100000] 9b20: 0ec38450 00000000 00000000 c17c4300 c0913528 c0b42660 c17c4300 4f21ffb9
[ 4115.100000] 9b40: 4f21ffb9 00000000 c0fee000 c1410800 000006c5 c17c4300 00000000 c1704000
[ 4115.100000] 9b60: 00000761 c047177c 00001030 c02c363c c1630000 00001ec0 00000000 4f21ffb9
[ 4115.100000] 9b80: 00000761 4f21ffb9 c171f0c0 c1410800 00001030 00000000 00017450 000006c5
[ 4115.100000] 9ba0: c0fee000 c1704000 00000000 c047050c 00017450 00001030 00000000 4f21ffb9
[ 4115.100000] 9bc0: 4f21ffb9 c1630000 c1630000 00001030 000006c5 00017450 c17c4300 c4e59d10
[ 4115.100000] 9be0: 000006c5 c02c3050 00001030 00000000 c1630000 c0b3e460 c4e59c98 c1704000
[ 4115.100000] 9c00: c1630000 c02c6ddc 00001030 00000001 00001013 c02c84b0 c1630000 00017450
[ 4115.100000] 9c20: 00001030 00000001 c0be9e00 c0b0ec00 c17c4300 c0100bb8 00000001 4f21ffb9
[ 4115.100000] 9c40: c4e59ca8 c0100bb8 c27c8c20 c1704c50 00000360 29307c32 92b74a1c 4f21ffb9
[ 4115.100000] 9c60: 24da6e32 c1630000 00000000 c4e59d10 c16300e4 00000000 c1704000 00000000
[ 4115.100000] 9c80: c17c4300 c02c8874 20030013 00000801 00000003 c0be9e00 000042eb 2000460c
[ 4115.100000] 9ca0: 00000000 000006c5 00017450 00001030 c17c4300 4f21ffb9 c1704000 c3fca920
[ 4115.100000] 9cc0: 00004f00 c11f7410 c0b3dc20 0000460c c17c4300 000042eb c1704000 c02b88c4
[ 4115.100000] 9ce0: 00000000 c0b57ef4 00000000 c0b3c32c 8af8af8b 00140cca 00352ac1 04f00000
[ 4115.100000] 9d00: c27c9000 c1630000 c17c4300 c3fca920 000042eb 2000460c c0b57ea4 4f21ffb9
[ 4115.100000] 9d20: c4e59d70 c3fca920 c17c4300 c1630000 c19ff600 0000460c 0000460b c11f7410
[ 4115.100000] 9d40: c11f7504 c02b9c90 c0b57ea4 c11f74f8 c3fca920 00000001 00000cc0 c01715d4
[ 4115.100000] 9d60: c4e59dc0 00000001 c17c4300 c3fca93c c11f74fc 4f21ffb9 000c0000 c3fca920
[ 4115.100000] 9d80: c17c4300 c4e59f28 c19ff600 c3fca920 00000000 00004620 c11f7504 c0170364
[ 4115.100000] 9da0: 00000000 00000cc0 c3fca920 0000460c c3fca920 c0171780 c4e59dc0 4f21ffb9
[ 4115.100000] 9dc0: 0034d869 c11f74f8 c4e59e8c c4e59f28 0000460c c3fca920 00000000 c017242c
[ 4115.100000] 9de0: c198aabc c19ff600 c19ff650 c17c4300 00100008 c4e59f10 c0b02800 61c88647
[ 4115.100000] 9e00: c4e59f08 0000b000 c19ff600 c11f74f8 c19ff650 0000460c 00000000 00000000
[ 4115.100000] 9e20: c0b0b9cc 4f21ffb9 00007000 c11f7410 c4e59e90 c4e59f28 0000c000 c4e59f10
[ 4115.100000] 9e40: 00000000 00001000 00000000 c0175ed8 00001000 00000000 04620000 00000000
[ 4115.100000] 9e60: 10b072e6 c4e59f10 c19ff600 00000000 04f00000 00000000 00000000 c11f7410
[ 4115.100000] 9e80: c11f74f8 c17c4300 00000000 00020000 c3fca900 c19ff540 c4e59f08 c01c637c
[ 4115.100000] 9ea0: 0000005e c0100bb8 c4e59f2c 00000000 ffffffe4 00000000 00020000 c19ff600
[ 4115.100000] 9ec0: 00000000 00004004 c4e59f78 4f21ffb9 00020000 00020000 c19ff600 04600000
[ 4115.100000] 9ee0: 00000000 c4e59f78 c17c4300 00020000 0000005e c01bdd74 00020000 4f21ffb9
[ 4115.100000] 9f00: b6ebe000 00020000 b6ebe000 00020000 00000000 0000c000 00014000 c4e59f08
[ 4115.100000] 9f20: 00000001 00000000 c19ff600 00000000 0460c000 00000000 00000000 00000000
[ 4115.100000] 9f40: 00000000 00004004 00000000 00000000 004660b4 4f21ffb9 b6ff5940 c19ff600
[ 4115.100000] 9f60: c19ff600 04600000 00000000 c17c4300 b6ebe000 c01be668 04600000 00000000
[ 4115.100000] 9f80: 00000003 4f21ffb9 004660b4 00020000 b6ff5940 00000003 c01002c4 c17c4300
[ 4115.100000] 9fa0: 00000003 c0100060 004660b4 00020000 00000003 b6ebe000 00020000 00000000
[ 4115.100000] 9fc0: 004660b4 00020000 b6ff5940 00000003 00000003 00020000 00020000 0000005e
[ 4115.100000] 9fe0: 00000003 be907b58 b6f6d187 b6ef9c66 60070030 00000003 00000000 00000000
[ 4115.100000]  atc_chain_complete from atc_advance_work+0x78/0x184
[ 4115.100000]  atc_advance_work from atmel_nand_dma_transfer+0x114/0x23c
[ 4115.100000]  atmel_nand_dma_transfer from atmel_nand_data_in+0xd0/0x108
[ 4115.100000]  atmel_nand_data_in from atmel_hsmc_exec_rw+0x34/0x3c
[ 4115.100000]  atmel_hsmc_exec_rw from nand_op_parser_exec_op+0x3b0/0x5c4
[ 4115.100000]  nand_op_parser_exec_op from nand_read_data_op+0x1a4/0x27c
[ 4115.100000]  nand_read_data_op from atmel_nand_pmecc_read_pg.constprop.8+0x68/0xc4
[ 4115.100000]  atmel_nand_pmecc_read_pg.constprop.8 from atmel_hsmc_nand_pmecc_read_page+0x1c/0x24
[ 4115.100000]  atmel_hsmc_nand_pmecc_read_page from nand_read_oob+0x1bc/0x7c0
[ 4115.100000]  nand_read_oob from mtd_read_oob+0x88/0x14c
[ 4115.100000]  mtd_read_oob from mtd_read+0x5c/0x80
[ 4115.100000]  mtd_read from ubi_io_read+0xe4/0x36c
[ 4115.100000]  ubi_io_read from ubi_eba_read_leb+0xd0/0x4e8
[ 4115.100000]  ubi_eba_read_leb from ubi_leb_read+0x94/0x100
[ 4115.100000]  ubi_leb_read from ubifs_leb_read+0x2c/0x78
[ 4115.100000]  ubifs_leb_read from fallible_read_node+0x84/0x27c
[ 4115.100000]  fallible_read_node from ubifs_tnc_locate+0xfc/0x1cc
[ 4115.100000]  ubifs_tnc_locate from do_readpage+0x19c/0x494
[ 4115.100000]  do_readpage from ubifs_readpage+0x48/0x4a4
[ 4115.100000]  ubifs_readpage from filemap_read_folio+0x44/0x1fc
[ 4115.100000]  filemap_read_folio from filemap_get_pages+0x4cc/0x790
[ 4115.100000]  filemap_get_pages from filemap_read+0xcc/0x3bc
[ 4115.100000]  filemap_read from vfs_read+0x25c/0x2e4
[ 4115.100000]  vfs_read from ksys_read+0xa0/0xd0
[ 4115.100000]  ksys_read from ret_fast_syscall+0x0/0x54
[ 4115.100000] Exception stack(0xc4e59fa8 to 0xc4e59ff0)
[ 4115.100000] 9fa0:                   004660b4 00020000 00000003 b6ebe000 00020000 00000000
[ 4115.100000] 9fc0: 004660b4 00020000 b6ff5940 00000003 00000003 00020000 00020000 0000005e
[ 4115.100000] 9fe0: 00000003 be907b58 b6f6d187 b6ef9c66
[ 4115.100000] Code: ebf78ba1 e3a03000 e5c43068 eaffffc3 (e7f001f2) 
[ 4115.100000] ---[ end trace 0000000000000000 ]---