Re: Kernel Panic in skb_release_data using genet

From: Florian Fainelli
Date: Thu Aug 11 2022 - 23:34:08 EST



On 5/17/2022 12:52 AM, Maxime Ripard wrote:
It's not really 100% reliable, but happens 30%-50% of the time at boot
when KASAN is enabled. It seems like enabling KASAN increases that
likelihood though, it went unnoticed for some time before I started
having those issues again when I enabled it for something unrelated.

It looks like it happens in bursts though, so I would get 10-15 boots
fine, and then 4-5 boots with that crash.

Cold boot vs reboot doesn't seem to affect it in one way or the other.

What version of GCC did you build your kernel with?

The arm64 cross-compiler packaged by Fedora, which is GCC 11.2
at the moment.

How often does that happen? What config.txt file are you using
for your Pi4 B?

You'll find my config.txt and kernel .config attached

OK, so this is what I have been able to reproduce so far but this does not appear to be very reliable to reproduce, I will try my best to hold on to that lead though, thanks for your patience.

# udhcpc -i eth0
udhcpc: started, v1.35.0
[ 34.355086] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[ 34.363758] ==================================================================
[ 34.371106] BUG: KASAN: user-memory-access in put_page+0x10/0x64
[ 34.377227] Read of size 4 at addr 01000085 by task ifconfig/165
[ 34.383338]
[ 34.384857] CPU: 0 PID: 165 Comm: ifconfig Tainted: G W 5.19.0 #43
[ 34.392560] Hardware name: BCM2711
[ 34.396020] unwind_backtrace from show_stack+0x18/0x1c
[ 34.401354] show_stack from dump_stack_lvl+0x40/0x4c
[ 34.406502] dump_stack_lvl from kasan_report+0x8c/0xa4
[ 34.411825] kasan_report from put_page+0x10/0x64
[ 34.416615] put_page from skb_release_data+0x84/0x13c
[ 34.421847] skb_release_data from __kfree_skb+0x14/0x20
[ 34.427256] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8
[ 34.432846] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0
[ 34.439407] __napi_poll.constprop.0 from net_rx_action+0x278/0x488
[ 34.445787] net_rx_action from __do_softirq+0x268/0x390
[ 34.451197] __do_softirq from __irq_exit_rcu+0x88/0xf8
[ 34.456521] __irq_exit_rcu from irq_exit+0x10/0x18
[ 34.461492] irq_exit from call_with_stack+0x18/0x20
[ 34.466553] call_with_stack from __irq_svc+0x84/0x94
[ 34.471696] Exception stack(0xf0d337f8 to 0xf0d33840)
[ 34.476835] 37e0: c5548580 00000003
[ 34.485156] 3800: 00002000 f0a40808 c5548000 c5548580 00000000 c554b000 c5548580 c554bdd0
[ 34.493474] 3820: 00000000 00000004 c5548580 f0d33848 c094329c c09432bc 00070013 ffffffff
[ 34.501788] __irq_svc from bcmgenet_open+0xe1c/0x1094
[ 34.507023] bcmgenet_open from __dev_open+0x1e4/0x21c
[ 34.512258] __dev_open from __dev_change_flags+0x228/0x25c
[ 34.517931] __dev_change_flags from dev_change_flags+0x48/0x88
[ 34.523958] dev_change_flags from devinet_ioctl+0x3ac/0x834
[ 34.529723] devinet_ioctl from inet_ioctl+0x250/0x2a4
[ 34.534956] inet_ioctl from sock_ioctl+0x1dc/0x410
[ 34.539927] sock_ioctl from vfs_ioctl+0x50/0x64
[ 34.544632] vfs_ioctl from sys_ioctl+0x134/0xa7c
[ 34.549422] sys_ioctl from ret_fast_syscall+0x0/0x4c
[ 34.554565] Exception stack(0xf0d33fa8 to 0xf0d33ff0)
[ 34.559705] 3fa0: 0051fd98 0053f9dc 00000003 00008914 b6dc5c4c b6dc5bd0
[ 34.568025] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48 00000003 aed11d00 aed12010
[ 34.576341] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66
[ 34.581475] ==================================================================
[ 34.588882] Disabling lock debugging due to kernel taint
[ 34.594288] 8<--- cut here ---
[ 34.597412] Unable to handle kernel paging request at virtual address 01000085
[ 34.604775] [01000085] *pgd=01982003, *pmd=00000000
[ 34.609751] Internal error: Oops: 206 [#1] SMP ARM
[ 34.614624] Modules linked in:
[ 34.617734] CPU: 0 PID: 165 Comm: ifconfig Tainted: G B W 5.19.0 #43
[ 34.625435] Hardware name: BCM2711
[ 34.628892] PC is at put_page+0x14/0x64
[ 34.632800] LR is at kasan_report+0x98/0xa4
[ 34.637056] pc : [<c0b4bee4>] lr : [<c047ea5c>] psr: 60070113
[ 34.643427] sp : f0803d50 ip : 00000000 fp : c554bfd8
[ 34.648739] r10: 00007f5e r9 : c694f582 r8 : c1fef15e
[ 34.654052] r7 : c694f5b8 r6 : c694f580 r5 : 01000081 r4 : c1fef100
[ 34.660689] r3 : 00000000 r2 : c1f047c0 r1 : 00000004 r0 : 00000001
[ 34.667325] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 34.674582] Control: 30c5383d Table: 0606b700 DAC: fffffffd
[ 34.680422] Register r0 information: non-paged memory
[ 34.685565] Register r1 information: non-paged memory
[ 34.690705] Register r2 information: slab task_struct start c1f047c0 pointer offset 0
[ 34.698690] Register r3 information: NULL pointer
[ 34.703477] Register r4 information: slab skbuff_head_cache start c1fef100 pointer offset 0 size 48
[ 34.712699] Register r5 information: non-paged memory
[ 34.717839] Register r6 information: non-slab/vmalloc memory
[ 34.723595] Register r7 information: non-slab/vmalloc memory
[ 34.729352] Register r8 information: slab skbuff_head_cache start c1fef100 pointer offset 94 size 48
[ 34.738662] Register r9 information: non-slab/vmalloc memory
[ 34.744419] Register r10 information: non-paged memory
[ 34.749646] Register r11 information: non-slab/vmalloc memory
[ 34.755492] Register r12 information: NULL pointer
[ 34.760366] Process ifconfig (pid: 165, stack limit = 0xf517d551)
[ 34.766573] Stack: (0xf0803d50 to 0xf0804000)
[ 34.771005] 3d40: c1fef100 00000001 c694f580 c0b4dc74
[ 34.779325] 3d60: c1fef100 c5548000 c5548580 c1fef100 f0803e40 7f5e0001 00007f5e c0b4db24
[ 34.787644] 3d80: c554bdd0 c0940f84 0bc80000 b4c23195 c2cb12c0 c0efdab0 c2cb12c0 00000001
[ 34.795963] 3da0: 00000000 00000040 00000004 c554bec4 1e1007bc c554beb8 c5548588 00000004
[ 34.804282] 3dc0: c55498bc c554bec8 c02d5684 00000003 00000000 c02b6e10 e7df0980 c02bf390
[ 34.812601] 3de0: 41b58ab3 c15fec7a c0940a80 c1f047c0 00070113 257ac000 e7de97cc ffff982d
[ 34.820919] 3e00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 b4c23195
[ 34.829237] 3e20: c1f047c0 e7de8680 00000000 c1f047c0 00000000 c076733c e7de9ad8 00000000
[ 34.837556] 3e40: e7de97d4 c613e0a0 00000001 c554bdd0 00000001 00000040 f0803ef0 c554bdd8
[ 34.845875] 3e60: 257ac000 c2805d40 e7df0d00 c0b70f24 c554bdd0 f0803ef0 00000000 e7df0b40
[ 34.854195] 3e80: f0803f60 bd1007d8 c554bdd0 c2644b40 257ac000 c0b7130c 0000012c e7df0d0c
[ 34.862513] 3ea0: ffff9839 f0803ef0 81d99054 c554bdd4 0000002c 257ac000 c26433c8 c0840554
[ 34.870832] 3ec0: 41b58ab3 c1612850 c0b71094 c2cb12c0 e7df0980 c02d8a5c ea8ed400 c02d8ae0
[ 34.879150] 3ee0: 41b58ab3 c15f3580 c08403c4 00000010 c554bd00 c554bdd8 00000000 00000010
[ 34.887470] 3f00: f0803f00 f0803f00 c5548580 00002000 c554bdd0 c554b580 0000010a c093e0b8
[ 34.895788] 3f20: f0803f20 f0803f20 0000002c c093df98 c2806f18 c029f4ac 00000000 00000007
[ 34.904108] 3f40: e7de9780 c02a4218 00000104 c4dca800 00000001 c4dca824 c4dca86c c4dca86c
[ 34.912427] 3f60: c4dca848 f0803fc8 f0d337f0 b4c23195 c4dca800 c1f047c0 c280508c 00000008
[ 34.920747] 3f80: c2643dc0 c1f047c4 00000003 00000100 c1f049d4 c02014d8 c4dca800 c1f047c0
[ 34.929066] 3fa0: 00400100 0000000a ffff9838 00000004 c263c3c8 257ac000 c26433c0 c1f047c0
[ 34.937385] 3fc0: c2643dc0 c1f047c4 257ac000 257ac000 c1f047c0 00000000 f0d337f0 c02312c4
[ 34.945704] 3fe0: c09432bc 00070013 ffffffff f0d3382c c5548580 c0231418 c09432bc c07559fc
[ 34.954019] put_page from skb_release_data+0x84/0x13c
[ 34.959252] skb_release_data from __kfree_skb+0x14/0x20
[ 34.964660] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8
[ 34.970250] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0
[ 34.976812] __napi_poll.constprop.0 from net_rx_action+0x278/0x488
[ 34.983192] net_rx_action from __do_softirq+0x268/0x390
[ 34.988602] __do_softirq from __irq_exit_rcu+0x88/0xf8
[ 34.993927] __irq_exit_rcu from irq_exit+0x10/0x18
[ 34.998899] irq_exit from call_with_stack+0x18/0x20
[ 35.003958] call_with_stack from __irq_svc+0x84/0x94
[ 35.009101] Exception stack(0xf0d337f8 to 0xf0d33840)
[ 35.014238] 37e0: c5548580 00000003
[ 35.022557] 3800: 00002000 f0a40808 c5548000 c5548580 00000000 c554b000 c5548580 c554bdd0
[ 35.030877] 3820: 00000000 00000004 c5548580 f0d33848 c094329c c09432bc 00070013 ffffffff
[ 35.039192] __irq_svc from bcmgenet_open+0xe1c/0x1094
[ 35.044427] bcmgenet_open from __dev_open+0x1e4/0x21c
[ 35.049661] __dev_open from __dev_change_flags+0x228/0x25c
[ 35.055334] __dev_change_flags from dev_change_flags+0x48/0x88
[ 35.061361] dev_change_flags from devinet_ioctl+0x3ac/0x834
[ 35.067125] devinet_ioctl from inet_ioctl+0x250/0x2a4
[ 35.072359] inet_ioctl from sock_ioctl+0x1dc/0x410
[ 35.077330] sock_ioctl from vfs_ioctl+0x50/0x64
[ 35.082034] vfs_ioctl from sys_ioctl+0x134/0xa7c
[ 35.086825] sys_ioctl from ret_fast_syscall+0x0/0x4c
[ 35.091969] Exception stack(0xf0d33fa8 to 0xf0d33ff0)
[ 35.097109] 3fa0: 0051fd98 0053f9dc 00000003 00008914 b6dc5c4c b6dc5bd0
[ 35.105428] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48 00000003 aed11d00 aed12010
[ 35.113744] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66
[ 35.118883] Code: e1a05000 e2800004 ebe4cca7 e3a01004 (e5953004)
[ 35.125104] ---[ end trace 0000000000000000 ]---
[ 35.129801] Kernel panic - not syncing: Fatal exception in interrupt
[ 35.136260] CPU3: stopping
[ 35.139009] CPU: 3 PID: 27 Comm: migration/3 Tainted: G B D W 5.19.0 #43
[ 35.146872] Hardware name: BCM2711
[ 35.150318] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4
[ 35.158197] unwind_backtrace from show_stack+0x18/0x1c
[ 35.163509] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.168643] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.174218] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.179351] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.185454] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.191820] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.197393] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.203671] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.210038] call_with_stack from __irq_svc+0x84/0x94
[ 35.215168] Exception stack(0xf0913e98 to 0xf0913ee0)
[ 35.220293] 3e80: e7e20a10 00000000
[ 35.228594] 3ea0: 00000000 257dc000 e7e1ec68 f0913ee8 257dc000 00000000 c2806f18 60070013
[ 35.236896] 3ec0: f0863d70 f0863d74 f0863d70 f0913ee8 c02bebd4 c02bebe8 60070013 ffffffff
[ 35.245192] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c
[ 35.251296] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140
[ 35.257931] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8
[ 35.263947] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264
[ 35.270228] smpboot_thread_fn from kthread+0x12c/0x140
[ 35.275539] kthread from ret_from_fork+0x14/0x1c
[ 35.280317] Exception stack(0xf0913fb0 to 0xf0913ff8)
[ 35.285441] 3fa0: 00000000 00000000 00000000 00000000
[ 35.293739] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 35.302037] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 35.308746] CPU2: stopping
[ 35.311492] CPU: 2 PID: 22 Comm: migration/2 Tainted: G B D W 5.19.0 #43
[ 35.319355] Hardware name: BCM2711
[ 35.322803] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4
[ 35.330677] unwind_backtrace from show_stack+0x18/0x1c
[ 35.335988] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.341122] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.346697] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.351830] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.357932] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.364298] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.369870] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.376148] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.382515] call_with_stack from __irq_svc+0x84/0x94
[ 35.387646] Exception stack(0xf08ebea8 to 0xf08ebef0)
[ 35.392773] bea0: f0863d70 00000003 00000000 00000001 f0863d60 00000000
[ 35.401074] bec0: 00000001 00000000 c2806f18 600c0013 f0863d70 f0863d74 f0863d70 f08ebef8
[ 35.409372] bee0: c030acac c02bebbc 600c0013 ffffffff
[ 35.414495] __irq_svc from rcu_momentary_dyntick_idle+0x0/0x9c
[ 35.420511] rcu_momentary_dyntick_idle from 0xc31d0000
[ 35.425820] CPU1: stopping
[ 35.428568] CPU: 1 PID: 17 Comm: migration/1 Tainted: G B D W 5.19.0 #43
[ 35.436430] Hardware name: BCM2711
[ 35.439879] Stopper: multi_cpu_stop+0x0/0x140 <- stop_machine_cpuslocked+0x180/0x1e4
[ 35.447752] unwind_backtrace from show_stack+0x18/0x1c
[ 35.453064] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.458198] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.463772] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.468905] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.475006] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.481373] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.486945] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.493222] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.499590] call_with_stack from __irq_svc+0x84/0x94
[ 35.504721] Exception stack(0xf08c3e98 to 0xf08c3ee0)
[ 35.509847] 3e80: e7e00a10 00000000
[ 35.518148] 3ea0: 00000000 257bc000 e7dfec68 f08c3ee8 257bc000 00000000 c2806f18 600f0013
[ 35.526449] 3ec0: f0863d70 f0863d74 f0863d70 f08c3ee8 c02bebd4 c02bebe8 600f0013 ffffffff
[ 35.534745] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c
[ 35.540849] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140
[ 35.547483] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8
[ 35.553499] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264
[ 35.559780] smpboot_thread_fn from kthread+0x12c/0x140
[ 35.565090] kthread from ret_from_fork+0x14/0x1c
[ 35.569868] Exception stack(0xf08c3fb0 to 0xf08c3ff8)
[ 35.574992] 3fa0: 00000000 00000000 00000000 00000000
[ 35.583292] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 35.591589] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 35.599291] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
--
Florian