Re: Kernel Panic in skb_release_data using genet

[Florian Fainelli]

Hi Maxime,

On 5/24/2021 6:01 AM, Maxime Ripard wrote:
> Hi Doug, Florian,
> 
> I've been running a RaspberryPi4 with a mainline kernel for a while,
> booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
> I'm getting a kernel panic around the time init is started.
> 
> I was debugging a kernel based on drm-misc-next-2021-05-17 today with
> KASAN enabled and got this, which looks related:

Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?

How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.

> 
> [    6.109454] mmc0: SDHCI controller on fe300000.sdhci [fe300000.sdhci] using PIO
> [    6.124819] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
> [    6.133391] ==================================================================
> [    6.140736] BUG: KASAN: user-memory-access in skb_release_data+0x14c/0x1fc
> [    6.147748] Read of size 4 at addr 1c8befdc by task swapper/0/0
> [    6.153776] 
> [    6.155300] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-rc1-v7l #165
> [    6.162214] Hardware name: BCM2711
> [    6.165679] Backtrace: 
> [    6.168183] [<c110f5a8>] (dump_backtrace) from [<c110f930>] (show_stack+0x20/0x24)
> [    6.175931]  r7:c1e00000 r6:00000193 r5:00000000 r4:c837f8e0
> [    6.181683] [<c110f910>] (show_stack) from [<c11156c0>] (dump_stack+0xb8/0xdc)
> [    6.189051] [<c1115608>] (dump_stack) from [<c0514b30>] (kasan_report+0x11c/0x1c0)
> [    6.196789]  r9:cc97ff02 r8:cc57e400 r7:c0ea3628 r6:00000000 r5:00000000 r4:1c8befdc
> [    6.204655] [<c0514a14>] (kasan_report) from [<c05154d4>] (__asan_load4+0x74/0x90)
> [    6.212393]  r7:cc97ff00 r6:00000000 r5:cc97ff28 r4:1c8befd4
> [    6.218144] [<c0515460>] (__asan_load4) from [<c0ea3628>] (skb_release_data+0x14c/0x1fc)
> [    6.226395] [<c0ea34dc>] (skb_release_data) from [<c0ea9d2c>] (consume_skb+0x60/0x134)
> [    6.234479]  r10:0000a8d8 r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e4ac
> [    6.242438]  r4:cc57e400 r3:cc97f680
> [    6.246074] [<c0ea9ccc>] (consume_skb) from [<c0ec0d74>] (__dev_kfree_skb_any+0x60/0x64)
> [    6.254337]  r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e400 r4:c1e00000
> [    6.262203] [<c0ec0d14>] (__dev_kfree_skb_any) from [<c0c814d4>] (bcmgenet_rx_poll+0x578/0x770)
> [    6.271081]  r7:cc560580 r6:a8d81759 r5:cc57e400 r4:cc563ed8
> [    6.276831] [<c0c80f5c>] (bcmgenet_rx_poll) from [<c0ed3f0c>] (__napi_poll+0x60/0x2b8)
> [    6.284925]  r10:c1e03d20 r9:c1e05d00 r8:cc563ee0 r7:c1e03d10 r6:00000040 r5:00000001
> [    6.292881]  r4:cc563ed8
> [    6.295460] [<c0ed3eac>] (__napi_poll) from [<c0ed4a14>] (net_rx_action+0x580/0x620)
> [    6.303377]  r10:c1e03d20 r9:c1e05d00 r8:0000012c r7:cc563edc r6:cc560000 r5:cc563ed8
> [    6.311333]  r4:c1e03d80
> [    6.313911] [<c0ed4494>] (net_rx_action) from [<c02012e8>] (__do_softirq+0x1f0/0x69c)
> [    6.321916]  r10:c1e00000 r9:00000008 r8:16b2f000 r7:00000003 r6:00000004 r5:c18b9360
> [    6.329872]  r4:c1e0508c
> [    6.332449] [<c02010f8>] (__do_softirq) from [<c02367a4>] (irq_exit+0x188/0x1b0)
> [    6.340012]  r10:16b2f000 r9:c1e03ec0 r8:16b2f000 r7:c1e03e28 r6:ffffc000 r5:c1cc0940
> [    6.347969]  r4:c1e06ea4
> [    6.350546] [<c023661c>] (irq_exit) from [<c02c75fc>] (__handle_domain_irq+0xc4/0x128)
> [    6.353302] bcmgenet fd580000.ethernet eth0: Link is Down
> [    6.358635]  r9:c1e03ec0 r8:00000001 r7:00000000 r6:c1e00000 r5:00000000 r4:c1cbfe80
> [    6.371956] [<c02c7538>] (__handle_domain_irq) from [<c09ef2b4>] (gic_handle_irq+0x9c/0xb4)
> [    6.380496]  r10:f080200c r9:f0802000 r8:c1e03ec0 r7:c1e07878 r6:c1cbfe8c r5:000000bd
> [    6.388452]  r4:000000bd
> [    6.391030] [<c09ef218>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x80)
> [    6.398666] Exception stack(0xc1e03ec0 to 0xc1e03f08)
> [    6.403821] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
> [    6.412145] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
> [    6.420459] 3f00: 60000013 ffffffff
> [    6.424026]  r10:30c5387d r9:c1e00000 r8:c84ff712 r7:c1e03ef4 r6:ffffffff r5:60000013
> [    6.431983]  r4:c020a208
> [    6.434561] [<c020a1b8>] (arch_cpu_idle) from [<c112af34>] (default_idle_call+0x48/0x188)
> [    6.442906] [<c112aeec>] (default_idle_call) from [<c0287578>] (do_idle+0x11c/0x180)
> [    6.450816]  r9:c121e120 r8:c84ff712 r7:c1e06e6c r6:00000000 r5:c1e06e1c r4:c1e00000
> [    6.458681] [<c028745c>] (do_idle) from [<c0287a00>] (cpu_startup_entry+0x28/0x2c)
> [    6.466416]  r9:410fd083 r8:c187df68 r7:c1e00000 r6:ca9d6000 r5:c85201e0 r4:000000e1
> [    6.474283] [<c02879d8>] (cpu_startup_entry) from [<c111d7b8>] (rest_init+0x148/0x150)
> [    6.482358] [<c111d670>] (rest_init) from [<c1801534>] (arch_call_rest_init+0x18/0x1c)
> [    6.490450]  r7:c1e06dc0 r6:c1e00000 r5:c1e00000 r4:c851d5c0
> [    6.496202] [<c180151c>] (arch_call_rest_init) from [<c1801990>] (start_kernel+0x3e0/0x424)
> [    6.504723] [<c18015b0>] (start_kernel) from [<00000000>] (0x0)
> [    6.510776]  r8:2eff9400 r7:00000c42 r6:30c0387d r5:00000000 r4:c1800334
> [    6.517584] ==================================================================
> [    6.524921] Disabling lock debugging due to kernel taint
> [    6.530467] 8<--- cut here ---
> [    6.533628] Unable to handle kernel paging request at virtual address 1c8befdc
> [    6.541025] pgd = (ptrval)
> [    6.543837] [1c8befdc] *pgd=80000000004003, *pmd=00000000
> [    6.549431] Internal error: Oops: 206 [#1] SMP ARM
> [    6.554311] Modules linked in:
> [    6.557433] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G    B             5.13.0-rc1-v7l #165
> [    6.565755] Hardware name: BCM2711
> [    6.569217] PC is at skb_release_data+0x14c/0x1fc
> [    6.574015] LR is at end_report+0x6c/0xf0
> [    6.578109] pc : [<c0ea3628>]    lr : [<c05148ac>]    psr: 60000113
> [    6.584484] sp : c1e03ac8  ip : c1e03a60  fp : c1e03af4
> [    6.589801] r10: cc57e462  r9 : cc97ff02  r8 : cc57e400
> [    6.595116] r7 : cc97ff00  r6 : 00000000  r5 : cc97ff28  r4 : 1c8befd4
> [    6.601755] r3 : 00000000  r2 : c1e0ccc0  r1 : c0514884  r0 : 00000001
> [    6.608393] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [    6.615657] Control: 30c5383d  Table: 00003000  DAC: fffffffd
> [    6.621497] Register r0 information: non-paged memory
> [    6.626651] Register r1 information: non-slab/vmalloc memory
> [    6.632418] Register r2 information: non-slab/vmalloc memory
> [    6.638185] Register r3 information: NULL pointer
> [    6.642981] Register r4 information: non-paged memory
> [    6.648129] Register r5 information: non-slab/vmalloc memory
> [    6.653895] Register r6 information: NULL pointer
> [    6.658690] Register r7 information: non-slab/vmalloc memory
> [    6.664455] Register r8 information: slab skbuff_head_cache start cc57e400 pointer offset 0 size 48
> [    6.673715] Register r9 information: non-slab/vmalloc memory
> [    6.679481] Register r10 information: slab skbuff_head_cache start cc57e400 pointer offset 98 size 48
> [    6.688914] Register r11 information: non-slab/vmalloc memory
> [    6.694768] Register r12 information: non-slab/vmalloc memory
> [    6.700621] Process swapper/0 (pid: 0, stack limit = 0x(ptrval))
> [    6.706730] Stack: (0xc1e03ac8 to 0xc1e04000)
> [    6.711177] 3ac0:                   cc97f680 cc57e400 cc57e4ac 00000001 cc560580 00000000
> [    6.719502] 3ae0: cc560000 0000a8d8 c1e03b1c c1e03af8 c0ea9d2c c0ea34e8 c1e00000 cc57e400
> [    6.727825] 3b00: 00000001 cc560580 00000000 cc560000 c1e03b3c c1e03b20 c0ec0d74 c0ea9cd8
> [    6.736149] 3b20: cc563ed8 cc57e400 a8d81759 cc560580 c1e03c54 c1e03b40 c0c814d4 c0ec0d20
> [    6.744473] 3b40: c0210414 c05154fc c02103d8 ffffc000 c1e03b84 c1e03be0 c1e03c20 b73c0778
> [    6.752795] 3b60: 00000040 cc5640e8 cc560588 cc560088 cc561944 cc563fe0 cc561580 cc563fd8
> [    6.761118] 3b80: ca5c3c00 cc563fc8 cc563fd4 cc564078 0000000c 00000000 c1e03be4 00000000
> [    6.769441] 3ba0: 00000000 cc563580 c0210430 00000000 00000000 cc920374 c1e03be4 c1e03c88
> [    6.777764] 3bc0: 41b58ab3 c1730000 c0c80f5c cc920374 cc920340 00000001 c1e03d04 c1e03be8
> [    6.786084] 3be0: 00000000 00000000 00000000 00000000 00000000 00000000 c1e03c24 c1e03c08
> [    6.794406] 3c00: 41b58ab3 c16ca308 c02a73bc d87efd80 cab4d000 d87f0318 c1e03c4c 0147adf0
> [    6.802729] 3c20: c175a5d0 b5ed3f2f c02012e8 cc563ed8 00000001 00000040 c1e03d10 cc563ee0
> [    6.811052] 3c40: c1e05d00 c1e03d20 c1e03c94 c1e03c58 c0ed3f0c c0c80f68 c03a9ed0 c05154fc
> [    6.819376] 3c60: c03aa120 cc563ed8 60000113 c1e03d80 cc563ed8 cc560000 cc563edc 0000012c
> [    6.827700] 3c80: c1e05d00 c1e03d20 c1e03db4 c1e03c98 c0ed4a14 c0ed3eb8 d87f0740 b73c079c
> [    6.836023] 3ca0: c02104fc c1e00000 c1e00010 c1e00000 c1e03d40 16b2f000 c1cc1740 c1e05d00
> [    6.844347] 3cc0: ffff8d38 c1e03d20 c0210430 0000004c c1e03d04 c1e03ce0 c0c7e6e4 c051546c
> [    6.852670] 3ce0: 41b58ab3 c17447f0 c0ed4494 cb17fc00 c1e03da0 0000004c c1e03d54 c1e03d08
> [    6.860994] 3d00: c1e03e4c c1e03e28 c03a9eb0 c02367a4 c02cdd00 c051546c d87efdc0 cb17fc44
> [    6.869316] 3d20: c1e03d20 c1e03d20 d87efdf0 cb17fc00 c1e03d54 c1e03d40 c112b730 c051546c
> [    6.877640] 3d40: c1e03d40 c1e03d40 c02367a4 c0201240 c02c8548 c1e00000 c1cbec50 c02367a4
> [    6.885965] 3d60: c0201240 c1e00004 16b2f000 c1e00000 c1e03db4 c1e03d80 c03a9ed0 c05154fc
> [    6.894287] 3d80: 41b58ab3 b5ed3f2f c1e03db4 c1e0508c c18b9360 00000004 00000003 16b2f000
> [    6.902610] 3da0: 00000008 c1e00000 c1e03e24 c1e03db8 c02012e8 c0ed44a0 c1e03de4 c1e03dc8
> [    6.910932] 3dc0: 00000001 00200002 c1213840 c1e05d00 ffff8d37 c18b92d4 0000000a c1cc0940
> [    6.919256] 3de0: c09eed7c c18b9350 c1e05080 c1e03db8 00000101 c1e06e1c c1e03e24 c1e06ea4
> [    6.927580] 3e00: c1cc0940 ffffc000 c1e03e28 16b2f000 c1e03ec0 16b2f000 c1e03e4c c1e03e28
> [    6.935901] 3e20: c02367a4 c0201104 c1cbfe80 00000000 c1e00000 00000000 00000001 c1e03ec0
> [    6.944224] 3e40: c1e03e84 c1e03e50 c02c75fc c0236628 c112af34 ca91f000 c1e03ebc 000000bd
> [    6.952547] 3e60: 000000bd c1cbfe8c c1e07878 c1e03ec0 f0802000 f080200c c1e03ebc c1e03e88
> [    6.960872] 3e80: c09ef2b4 c02c7544 c03aa120 c021043c c020a204 c020a208 60000013 ffffffff
> [    6.969195] 3ea0: c1e03ef4 c84ff712 c1e00000 30c5387d c1e03f1c c1e03ec0 c0200abc c09ef224
> [    6.977516] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
> [    6.985840] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
> [    6.994163] 3f00: 60000013 ffffffff c020a1f4 00000000 c1e03f44 c1e03f20 c112af34 c020a1c4
> [    7.002486] 3f20: c1e00000 c1e06e1c 00000000 c1e06e6c c84ff712 c121e120 c1e03f6c c1e03f48
> [    7.010810] 3f40: c0287578 c112aef8 000000e1 c85201e0 ca9d6000 c1e00000 c187df68 410fd083
> [    7.019134] 3f60: c1e03f7c c1e03f70 c0287a00 c0287468 c1e03f9c c1e03f80 c111d7b8 c02879e4
> [    7.027458] 3f80: c851d5c0 c1e00000 c1e00000 c1e06dc0 c1e03fac c1e03fa0 c1801534 c111d67c
> [    7.035781] 3fa0: c1e03ff4 c1e03fb0 c1801990 c1801528 ffffffff ffffffff 00000000 c18006b8
> [    7.044103] 3fc0: 00000000 c187df68 b5e8322f 00000000 410fd083 c1800334 00000000 30c0387d
> [    7.052425] 3fe0: 00000c42 2eff9400 00000000 c1e03ff8 00000000 c18015bc 00000000 00000000
> [    7.060734] Backtrace: 
> [    7.063237] [<c0ea34dc>] (skb_release_data) from [<c0ea9d2c>] (consume_skb+0x60/0x134)
> [    7.071327]  r10:0000a8d8 r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e4ac
> [    7.079286]  r4:cc57e400 r3:cc97f680
> [    7.082923] [<c0ea9ccc>] (consume_skb) from [<c0ec0d74>] (__dev_kfree_skb_any+0x60/0x64)
> [    7.091187]  r9:cc560000 r8:00000000 r7:cc560580 r6:00000001 r5:cc57e400 r4:c1e00000
> [    7.099054] [<c0ec0d14>] (__dev_kfree_skb_any) from [<c0c814d4>] (bcmgenet_rx_poll+0x578/0x770)
> [    7.107934]  r7:cc560580 r6:a8d81759 r5:cc57e400 r4:cc563ed8
> [    7.113686] [<c0c80f5c>] (bcmgenet_rx_poll) from [<c0ed3f0c>] (__napi_poll+0x60/0x2b8)
> [    7.121778]  r10:c1e03d20 r9:c1e05d00 r8:cc563ee0 r7:c1e03d10 r6:00000040 r5:00000001
> [    7.129735]  r4:cc563ed8
> [    7.132313] [<c0ed3eac>] (__napi_poll) from [<c0ed4a14>] (net_rx_action+0x580/0x620)
> [    7.140233]  r10:c1e03d20 r9:c1e05d00 r8:0000012c r7:cc563edc r6:cc560000 r5:cc563ed8
> [    7.148189]  r4:c1e03d80
> [    7.150767] [<c0ed4494>] (net_rx_action) from [<c02012e8>] (__do_softirq+0x1f0/0x69c)
> [    7.158772]  r10:c1e00000 r9:00000008 r8:16b2f000 r7:00000003 r6:00000004 r5:c18b9360
> [    7.166728]  r4:c1e0508c
> [    7.169306] [<c02010f8>] (__do_softirq) from [<c02367a4>] (irq_exit+0x188/0x1b0)
> [    7.176870]  r10:16b2f000 r9:c1e03ec0 r8:16b2f000 r7:c1e03e28 r6:ffffc000 r5:c1cc0940
> [    7.184827]  r4:c1e06ea4
> [    7.187405] [<c023661c>] (irq_exit) from [<c02c75fc>] (__handle_domain_irq+0xc4/0x128)
> [    7.195497]  r9:c1e03ec0 r8:00000001 r7:00000000 r6:c1e00000 r5:00000000 r4:c1cbfe80
> [    7.203363] [<c02c7538>] (__handle_domain_irq) from [<c09ef2b4>] (gic_handle_irq+0x9c/0xb4)
> [    7.211902]  r10:f080200c r9:f0802000 r8:c1e03ec0 r7:c1e07878 r6:c1cbfe8c r5:000000bd
> [    7.219858]  r4:000000bd
> [    7.222436] [<c09ef218>] (gic_handle_irq) from [<c0200abc>] (__irq_svc+0x5c/0x80)
> [    7.230074] Exception stack(0xc1e03ec0 to 0xc1e03f08)
> [    7.235228] 3ec0: c175a018 d87f0614 00000000 c0222bc0 c1e00000 c1e06e1c 00000000 c1e06e6c
> [    7.243552] 3ee0: c84ff712 c121e120 30c5387d c1e03f1c c175a018 c1e03f10 c020a204 c020a208
> [    7.251865] 3f00: 60000013 ffffffff
> [    7.255432]  r10:30c5387d r9:c1e00000 r8:c84ff712 r7:c1e03ef4 r6:ffffffff r5:60000013
> [    7.263389]  r4:c020a208
> [    7.265967] [<c020a1b8>] (arch_cpu_idle) from [<c112af34>] (default_idle_call+0x48/0x188)
> [    7.274309] [<c112aeec>] (default_idle_call) from [<c0287578>] (do_idle+0x11c/0x180)
> [    7.282219]  r9:c121e120 r8:c84ff712 r7:c1e06e6c r6:00000000 r5:c1e06e1c r4:c1e00000
> [    7.290085] [<c028745c>] (do_idle) from [<c0287a00>] (cpu_startup_entry+0x28/0x2c)
> [    7.297821]  r9:410fd083 r8:c187df68 r7:c1e00000 r6:ca9d6000 r5:c85201e0 r4:000000e1
> [    7.305687] [<c02879d8>] (cpu_startup_entry) from [<c111d7b8>] (rest_init+0x148/0x150)
> [    7.313764] [<c111d670>] (rest_init) from [<c1801534>] (arch_call_rest_init+0x18/0x1c)
> [    7.321855]  r7:c1e06dc0 r6:c1e00000 r5:c1e00000 r4:c851d5c0
> [    7.327606] [<c180151c>] (arch_call_rest_init) from [<c1801990>] (start_kernel+0x3e0/0x424)
> [    7.336126] [<c18015b0>] (start_kernel) from [<00000000>] (0x0)
> [    7.342179]  r8:2eff9400 r7:00000c42 r6:30c0387d r5:00000000 r4:c1800334
> [    7.349000] Code: ebd9c790 e5954000 e2840008 ebd9c78d (e5943008) 
> [    7.355247] ---[ end trace 38b3df6838c109c3 ]---
> 
> Let me know if you need any other information, thanks!
> Maxime
> 

-- 
Florian