Kernel Panic in skb_release_data using genet
From: Maxime Ripard
Date: Mon May 24 2021 - 09:02:00 EST
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
[ 6.109454] mmc0: SDHCI controller on fe300000.sdhci [fe300000.sdhci] using PIO
[ 6.124819] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 6.133391] ==================================================================
[ 6.140736] BUG: KASAN: user-memory-access in skb_release_data+0x14c/0x1fc
[ 6.147748] Read of size 4 at addr 1c8befdc by task swapper/0/0
[ 6.153776]
[ 6.155300] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-rc1-v7l #165
[ 6.162214] Hardware name: BCM2711
[ 6.165679] Backtrace:
[ 6.168183] [<c110f5a8>] (dump_backtrace) from [<c110f930>] (show_stack+0x20/0x24)
[ 6.175931] r7:c1e00000 r6:00000193 r5:00000000 r4:c837f8e0
[ 6.181683] [<c110f910>] (show_stack) from [<c11156c0>] (dump_stack+0xb8/0xdc)
[ 6.189051] [<c1115608>] (dump_stack) from [<c0514b30>] (kasan_report+0x11c/0x1c0)
[ 6.196789] r9:cc97ff02 r8:cc57e400 r7:c0ea3628 r6:00000000 r5:00000000 r4:1c8befdc
[ 6.204655] [<c0514a14>] (kasan_report) from [<c05154d4>] (__asan_load4+0x74/0x90)
[ 6.212393] r7:cc97ff00 r6:00000000 r5:cc97ff28 r4:1c8befd4
[ 6.218144] [<c0515460>] (__asan_load4) from [<c0ea3628>] (skb_release_data+0x14c/0x1fc)
[ 6.226395] [<c0ea34dc>] (skb_release_data) from [<c0ea9d2c>] (consume_skb+0x60/0x134)
[ 6.234479] r10:0000a8d8 r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e4ac
[ 6.242438] r4:cc57e400 r3:cc97f680
[ 6.246074] [<c0ea9ccc>] (consume_skb) from [<c0ec0d74>] (__dev_kfree_skb_any+0x60/0x64)
[ 6.254337] r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e400 r4:c1e00000
[ 6.262203] [<c0ec0d14>] (__dev_kfree_skb_any) from [<c0c814d4>] (bcmgenet_rx_poll+0x578/0x770)
[ 6.271081] r7:cc560580 r6:a8d81759 r5:cc57e400 r4:cc563ed8
[ 6.276831] [<c0c80f5c>] (bcmgenet_rx_poll) from [<c0ed3f0c>] (__napi_poll+0x60/0x2b8)
[ 6.284925] r10:c1e03d20 r9:c1e05d00 r8:cc563ee0 r7:c1e03d10 r6:00000040 r5:00000001
[ 6.292881] r4:cc563ed8
[ 6.295460] [<c0ed3eac>] (__napi_poll) from [<c0ed4a14>] (net_rx_action+0x580/0x620)
[ 6.303377] r10:c1e03d20 r9:c1e05d00 r8:0000012c r7:cc563edc r6:cc560000 r5:cc563ed8
[ 6.311333] r4:c1e03d80
[ 6.313911] [<c0ed4494>] (net_rx_action) from [<c02012e8>] (__do_softirq+0x1f0/0x69c)
[ 6.321916] r10:c1e00000 r9:00000008 r8:16b2f000 r7:00000003 r6:00000004 r5:c18b9360
[ 6.329872] r4:c1e0508c
[ 6.332449] [<c02010f8>] (__do_softirq) from [<c02367a4>] (irq_exit+0x188/0x1b0)
[ 6.340012] r10:16b2f000 r9:c1e03ec0 r8:16b2f000 r7:c1e03e28 r6:ffffc000 r5:c1cc0940
[ 6.347969] r4:c1e06ea4
[ 6.350546] [<c023661c>] (irq_exit) from [<c02c75fc>] (__handle_domain_irq+0xc4/0x128)
[ 6.353302] bcmgenet fd580000.ethernet eth0: Link is Down
[ 6.358635] r9:c1e03ec0 r8:00000001 r7:00000000 r6:c1e00000 r5:00000000 r4:c1cbfe80
[ 6.371956] [<c02c7538>] (__handle_domain_irq) from [<c09ef2b4>] (gic_handle_irq+0x9c/0xb4)
[ 6.380496] r10:f080200c r9:f0802000 r8:c1e03ec0 r7:c1e07878 r6:c1cbfe8c r5:000000bd
[ 6.388452] r4:000000bd
[ 6.391030] [<c09ef218>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x80)
[ 6.398666] Exception stack(0xc1e03ec0 to 0xc1e03f08)
[ 6.403821] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
[ 6.412145] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
[ 6.420459] 3f00: 60000013 ffffffff
[ 6.424026] r10:30c5387d r9:c1e00000 r8:c84ff712 r7:c1e03ef4 r6:ffffffff r5:60000013
[ 6.431983] r4:c020a208
[ 6.434561] [<c020a1b8>] (arch_cpu_idle) from [<c112af34>] (default_idle_call+0x48/0x188)
[ 6.442906] [<c112aeec>] (default_idle_call) from [<c0287578>] (do_idle+0x11c/0x180)
[ 6.450816] r9:c121e120 r8:c84ff712 r7:c1e06e6c r6:00000000 r5:c1e06e1c r4:c1e00000
[ 6.458681] [<c028745c>] (do_idle) from [<c0287a00>] (cpu_startup_entry+0x28/0x2c)
[ 6.466416] r9:410fd083 r8:c187df68 r7:c1e00000 r6:ca9d6000 r5:c85201e0 r4:000000e1
[ 6.474283] [<c02879d8>] (cpu_startup_entry) from [<c111d7b8>] (rest_init+0x148/0x150)
[ 6.482358] [<c111d670>] (rest_init) from [<c1801534>] (arch_call_rest_init+0x18/0x1c)
[ 6.490450] r7:c1e06dc0 r6:c1e00000 r5:c1e00000 r4:c851d5c0
[ 6.496202] [<c180151c>] (arch_call_rest_init) from [<c1801990>] (start_kernel+0x3e0/0x424)
[ 6.504723] [<c18015b0>] (start_kernel) from [<00000000>] (0x0)
[ 6.510776] r8:2eff9400 r7:00000c42 r6:30c0387d r5:00000000 r4:c1800334
[ 6.517584] ==================================================================
[ 6.524921] Disabling lock debugging due to kernel taint
[ 6.530467] 8<--- cut here ---
[ 6.533628] Unable to handle kernel paging request at virtual address 1c8befdc
[ 6.541025] pgd = (ptrval)
[ 6.543837] [1c8befdc] *pgd=80000000004003, *pmd=00000000
[ 6.549431] Internal error: Oops: 206 [#1] SMP ARM
[ 6.554311] Modules linked in:
[ 6.557433] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G B 5.13.0-rc1-v7l #165
[ 6.565755] Hardware name: BCM2711
[ 6.569217] PC is at skb_release_data+0x14c/0x1fc
[ 6.574015] LR is at end_report+0x6c/0xf0
[ 6.578109] pc : [<c0ea3628>] lr : [<c05148ac>] psr: 60000113
[ 6.584484] sp : c1e03ac8 ip : c1e03a60 fp : c1e03af4
[ 6.589801] r10: cc57e462 r9 : cc97ff02 r8 : cc57e400
[ 6.595116] r7 : cc97ff00 r6 : 00000000 r5 : cc97ff28 r4 : 1c8befd4
[ 6.601755] r3 : 00000000 r2 : c1e0ccc0 r1 : c0514884 r0 : 00000001
[ 6.608393] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 6.615657] Control: 30c5383d Table: 00003000 DAC: fffffffd
[ 6.621497] Register r0 information: non-paged memory
[ 6.626651] Register r1 information: non-slab/vmalloc memory
[ 6.632418] Register r2 information: non-slab/vmalloc memory
[ 6.638185] Register r3 information: NULL pointer
[ 6.642981] Register r4 information: non-paged memory
[ 6.648129] Register r5 information: non-slab/vmalloc memory
[ 6.653895] Register r6 information: NULL pointer
[ 6.658690] Register r7 information: non-slab/vmalloc memory
[ 6.664455] Register r8 information: slab skbuff_head_cache start cc57e400 pointer offset 0 size 48
[ 6.673715] Register r9 information: non-slab/vmalloc memory
[ 6.679481] Register r10 information: slab skbuff_head_cache start cc57e400 pointer offset 98 size 48
[ 6.688914] Register r11 information: non-slab/vmalloc memory
[ 6.694768] Register r12 information: non-slab/vmalloc memory
[ 6.700621] Process swapper/0 (pid: 0, stack limit = 0x(ptrval))
[ 6.706730] Stack: (0xc1e03ac8 to 0xc1e04000)
[ 6.711177] 3ac0: cc97f680 cc57e400 cc57e4ac 00000001 cc560580 00000000
[ 6.719502] 3ae0: cc560000 0000a8d8 c1e03b1c c1e03af8 c0ea9d2c c0ea34e8 c1e00000 cc57e400
[ 6.727825] 3b00: 00000001 cc560580 00000000 cc560000 c1e03b3c c1e03b20 c0ec0d74 c0ea9cd8
[ 6.736149] 3b20: cc563ed8 cc57e400 a8d81759 cc560580 c1e03c54 c1e03b40 c0c814d4 c0ec0d20
[ 6.744473] 3b40: c0210414 c05154fc c02103d8 ffffc000 c1e03b84 c1e03be0 c1e03c20 b73c0778
[ 6.752795] 3b60: 00000040 cc5640e8 cc560588 cc560088 cc561944 cc563fe0 cc561580 cc563fd8
[ 6.761118] 3b80: ca5c3c00 cc563fc8 cc563fd4 cc564078 0000000c 00000000 c1e03be4 00000000
[ 6.769441] 3ba0: 00000000 cc563580 c0210430 00000000 00000000 cc920374 c1e03be4 c1e03c88
[ 6.777764] 3bc0: 41b58ab3 c1730000 c0c80f5c cc920374 cc920340 00000001 c1e03d04 c1e03be8
[ 6.786084] 3be0: 00000000 00000000 00000000 00000000 00000000 00000000 c1e03c24 c1e03c08
[ 6.794406] 3c00: 41b58ab3 c16ca308 c02a73bc d87efd80 cab4d000 d87f0318 c1e03c4c 0147adf0
[ 6.802729] 3c20: c175a5d0 b5ed3f2f c02012e8 cc563ed8 00000001 00000040 c1e03d10 cc563ee0
[ 6.811052] 3c40: c1e05d00 c1e03d20 c1e03c94 c1e03c58 c0ed3f0c c0c80f68 c03a9ed0 c05154fc
[ 6.819376] 3c60: c03aa120 cc563ed8 60000113 c1e03d80 cc563ed8 cc560000 cc563edc 0000012c
[ 6.827700] 3c80: c1e05d00 c1e03d20 c1e03db4 c1e03c98 c0ed4a14 c0ed3eb8 d87f0740 b73c079c
[ 6.836023] 3ca0: c02104fc c1e00000 c1e00010 c1e00000 c1e03d40 16b2f000 c1cc1740 c1e05d00
[ 6.844347] 3cc0: ffff8d38 c1e03d20 c0210430 0000004c c1e03d04 c1e03ce0 c0c7e6e4 c051546c
[ 6.852670] 3ce0: 41b58ab3 c17447f0 c0ed4494 cb17fc00 c1e03da0 0000004c c1e03d54 c1e03d08
[ 6.860994] 3d00: c1e03e4c c1e03e28 c03a9eb0 c02367a4 c02cdd00 c051546c d87efdc0 cb17fc44
[ 6.869316] 3d20: c1e03d20 c1e03d20 d87efdf0 cb17fc00 c1e03d54 c1e03d40 c112b730 c051546c
[ 6.877640] 3d40: c1e03d40 c1e03d40 c02367a4 c0201240 c02c8548 c1e00000 c1cbec50 c02367a4
[ 6.885965] 3d60: c0201240 c1e00004 16b2f000 c1e00000 c1e03db4 c1e03d80 c03a9ed0 c05154fc
[ 6.894287] 3d80: 41b58ab3 b5ed3f2f c1e03db4 c1e0508c c18b9360 00000004 00000003 16b2f000
[ 6.902610] 3da0: 00000008 c1e00000 c1e03e24 c1e03db8 c02012e8 c0ed44a0 c1e03de4 c1e03dc8
[ 6.910932] 3dc0: 00000001 00200002 c1213840 c1e05d00 ffff8d37 c18b92d4 0000000a c1cc0940
[ 6.919256] 3de0: c09eed7c c18b9350 c1e05080 c1e03db8 00000101 c1e06e1c c1e03e24 c1e06ea4
[ 6.927580] 3e00: c1cc0940 ffffc000 c1e03e28 16b2f000 c1e03ec0 16b2f000 c1e03e4c c1e03e28
[ 6.935901] 3e20: c02367a4 c0201104 c1cbfe80 00000000 c1e00000 00000000 00000001 c1e03ec0
[ 6.944224] 3e40: c1e03e84 c1e03e50 c02c75fc c0236628 c112af34 ca91f000 c1e03ebc 000000bd
[ 6.952547] 3e60: 000000bd c1cbfe8c c1e07878 c1e03ec0 f0802000 f080200c c1e03ebc c1e03e88
[ 6.960872] 3e80: c09ef2b4 c02c7544 c03aa120 c021043c c020a204 c020a208 60000013 ffffffff
[ 6.969195] 3ea0: c1e03ef4 c84ff712 c1e00000 30c5387d c1e03f1c c1e03ec0 c0200abc c09ef224
[ 6.977516] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
[ 6.985840] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
[ 6.994163] 3f00: 60000013 ffffffff c020a1f4 00000000 c1e03f44 c1e03f20 c112af34 c020a1c4
[ 7.002486] 3f20: c1e00000 c1e06e1c 00000000 c1e06e6c c84ff712 c121e120 c1e03f6c c1e03f48
[ 7.010810] 3f40: c0287578 c112aef8 000000e1 c85201e0 ca9d6000 c1e00000 c187df68 410fd083
[ 7.019134] 3f60: c1e03f7c c1e03f70 c0287a00 c0287468 c1e03f9c c1e03f80 c111d7b8 c02879e4
[ 7.027458] 3f80: c851d5c0 c1e00000 c1e00000 c1e06dc0 c1e03fac c1e03fa0 c1801534 c111d67c
[ 7.035781] 3fa0: c1e03ff4 c1e03fb0 c1801990 c1801528 ffffffff ffffffff 00000000 c18006b8
[ 7.044103] 3fc0: 00000000 c187df68 b5e8322f 00000000 410fd083 c1800334 00000000 30c0387d
[ 7.052425] 3fe0: 00000c42 2eff9400 00000000 c1e03ff8 00000000 c18015bc 00000000 00000000
[ 7.060734] Backtrace:
[ 7.063237] [<c0ea34dc>] (skb_release_data) from [<c0ea9d2c>] (consume_skb+0x60/0x134)
[ 7.071327] r10:0000a8d8 r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e4ac
[ 7.079286] r4:cc57e400 r3:cc97f680
[ 7.082923] [<c0ea9ccc>] (consume_skb) from [<c0ec0d74>] (__dev_kfree_skb_any+0x60/0x64)
[ 7.091187] r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e400 r4:c1e00000
[ 7.099054] [<c0ec0d14>] (__dev_kfree_skb_any) from [<c0c814d4>] (bcmgenet_rx_poll+0x578/0x770)
[ 7.107934] r7:cc560580 r6:a8d81759 r5:cc57e400 r4:cc563ed8
[ 7.113686] [<c0c80f5c>] (bcmgenet_rx_poll) from [<c0ed3f0c>] (__napi_poll+0x60/0x2b8)
[ 7.121778] r10:c1e03d20 r9:c1e05d00 r8:cc563ee0 r7:c1e03d10 r6:00000040 r5:00000001
[ 7.129735] r4:cc563ed8
[ 7.132313] [<c0ed3eac>] (__napi_poll) from [<c0ed4a14>] (net_rx_action+0x580/0x620)
[ 7.140233] r10:c1e03d20 r9:c1e05d00 r8:0000012c r7:cc563edc r6:cc560000 r5:cc563ed8
[ 7.148189] r4:c1e03d80
[ 7.150767] [<c0ed4494>] (net_rx_action) from [<c02012e8>] (__do_softirq+0x1f0/0x69c)
[ 7.158772] r10:c1e00000 r9:00000008 r8:16b2f000 r7:00000003 r6:00000004 r5:c18b9360
[ 7.166728] r4:c1e0508c
[ 7.169306] [<c02010f8>] (__do_softirq) from [<c02367a4>] (irq_exit+0x188/0x1b0)
[ 7.176870] r10:16b2f000 r9:c1e03ec0 r8:16b2f000 r7:c1e03e28 r6:ffffc000 r5:c1cc0940
[ 7.184827] r4:c1e06ea4
[ 7.187405] [<c023661c>] (irq_exit) from [<c02c75fc>] (__handle_domain_irq+0xc4/0x128)
[ 7.195497] r9:c1e03ec0 r8:00000001 r7:00000000 r6:c1e00000 r5:00000000 r4:c1cbfe80
[ 7.203363] [<c02c7538>] (__handle_domain_irq) from [<c09ef2b4>] (gic_handle_irq+0x9c/0xb4)
[ 7.211902] r10:f080200c r9:f0802000 r8:c1e03ec0 r7:c1e07878 r6:c1cbfe8c r5:000000bd
[ 7.219858] r4:000000bd
[ 7.222436] [<c09ef218>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x80)
[ 7.230074] Exception stack(0xc1e03ec0 to 0xc1e03f08)
[ 7.235228] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
[ 7.243552] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
[ 7.251865] 3f00: 60000013 ffffffff
[ 7.255432] r10:30c5387d r9:c1e00000 r8:c84ff712 r7:c1e03ef4 r6:ffffffff r5:60000013
[ 7.263389] r4:c020a208
[ 7.265967] [<c020a1b8>] (arch_cpu_idle) from [<c112af34>] (default_idle_call+0x48/0x188)
[ 7.274309] [<c112aeec>] (default_idle_call) from [<c0287578>] (do_idle+0x11c/0x180)
[ 7.282219] r9:c121e120 r8:c84ff712 r7:c1e06e6c r6:00000000 r5:c1e06e1c r4:c1e00000
[ 7.290085] [<c028745c>] (do_idle) from [<c0287a00>] (cpu_startup_entry+0x28/0x2c)
[ 7.297821] r9:410fd083 r8:c187df68 r7:c1e00000 r6:ca9d6000 r5:c85201e0 r4:000000e1
[ 7.305687] [<c02879d8>] (cpu_startup_entry) from [<c111d7b8>] (rest_init+0x148/0x150)
[ 7.313764] [<c111d670>] (rest_init) from [<c1801534>] (arch_call_rest_init+0x18/0x1c)
[ 7.321855] r7:c1e06dc0 r6:c1e00000 r5:c1e00000 r4:c851d5c0
[ 7.327606] [<c180151c>] (arch_call_rest_init) from [<c1801990>] (start_kernel+0x3e0/0x424)
[ 7.336126] [<c18015b0>] (start_kernel) from [<00000000>] (0x0)
[ 7.342179] r8:2eff9400 r7:00000c42 r6:30c0387d r5:00000000 r4:c1800334
[ 7.349000] Code: ebd9c790 e5954000 e2840008 ebd9c78d (e5943008)
[ 7.355247] ---[ end trace 38b3df6838c109c3 ]---
Let me know if you need any other information, thanks!
Maxime
Attachment:
signature.asc
Description: PGP signature