Re: Origen board hang with functionfs
From: John Stultz
Date: Thu Feb 21 2013 - 13:34:11 EST
On 02/20/2013 06:01 PM, John Stultz wrote:
Hey Kukjin, Andrzej,
I recently started playing around with functionfs, and have
noticed some strange behavior with my origen board.
If I enable the FunctionFS gadget driver, I see the board hang at boot
here:
[ 2.360000] USB Mass Storage support registered.
[ 2.365000] s3c-hsotg s3c-hsotg: regs f0040000, irq 103
[ 2.375000] s3c-hsotg s3c-hsotg: EPs:15
[ 2.380000] s3c-hsotg s3c-hsotg: dedicated fifos
[ 2.385000] g_ffs: file system registered
[ 2.390000] mousedev: PS/2 mouse device common for all mice
[ 2.395000] s5p-fimc exynos4-fimc.0: start latency exceeded, new
value 792 ns
[ 2.400000] s5p-fimc exynos4-fimc.0: state restore latency
exceeded, new vals
[ 2.410000] s5p-fimc exynos4-fimc.0: stop latency exceeded, new
value 667 ns
[ 2.420000] s5p-fimc exynos4-fimc.0: state save latency exceeded,
new value s
[ 2.425000] s5p-fimc exynos4-fimc.1: start latency exceeded, new
value 625 ns
[ 2.435000] s5p-fimc exynos4-fimc.1: state restore latency
exceeded, new vals
[ 2.440000] s5p-fimc exynos4-fimc.1: stop latency exceeded, new
value 750 ns
[ 2.450000] s5p-fimc exynos4-fimc.1: start latency exceeded, new
value 667 ns
[ 2.455000] s5p-fimc exynos4-fimc.1: state save latency exceeded,
new value s
And it gets no further. Trying to send a sysrq-t (or anything after a
break) doesn't seem to do anything either.
Trying to debug this a bit more, I added "initcall_debug
ignore_loglevel" to the boot arguments, but strangely, that allows it
to boot and everything seems to be fine. I added a number of printks
to the g_ffs driver, but it doesn't seem like the hang is happening in
that code (though de-configuring it does avoid the issue - so maybe
not?).
So I also enabled lockdep to see if it might show a locking problem even
when initcall_debug avoids the race. Unfortunately nothing shows up at
boot, but when I try to use FunctionFS I got the following lockdep
splat, which then triggered a warning and then finally an oops all in
succession.
The core lockdep parts are below, and the full log is attached. (Sorry
its cropped at 80 lines, I wasn't logging my screen session at the time,
so its all copy-pasted. Let me know if you'd need me to recreate the
problem.)
[ 109.290000] =========================================================
[ 109.295000] [ INFO: possible irq lock inversion dependency detected ]
[ 109.305000] 3.8.0-dirty #199 Not tainted
[ 109.310000] ---------------------------------------------------------
[ 109.315000] swapper/0/0 just changed the state of lock:
[ 109.320000] (&(&hsotg->lock)->rlock){-.....}, at: [<c029d35c>]
s3c_hsotg_ir8
[ 109.325000] but this lock took another, HARDIRQ-unsafe lock in the past:
[ 109.335000] (clocks_mutex){+.+...}
and interrupts could create inverse lock ordering between them.
[ 109.345000]
[ 109.345000] other info that might help us debug this:
[ 109.350000] Possible interrupt unsafe locking scenario:
[ 109.350000]
[ 109.355000] CPU0 CPU1
[ 109.360000] ---- ----
[ 109.365000] lock(clocks_mutex);
[ 109.370000] local_irq_disable();
[ 109.375000] lock(&(&hsotg->lock)->rlock);
[ 109.380000] lock(clocks_mutex);
[ 109.390000] <Interrupt>
[ 109.390000] lock(&(&hsotg->lock)->rlock);
[ 109.395000]
[ 109.395000] *** DEADLOCK ***
[ 109.395000]
[ 109.400000] no locks held by swapper/0/0.
[ 109.405000]
[ 109.405000] the shortest dependencies between 2nd lock and 1st lock:
[ 109.410000] -> (clocks_mutex){+.+...} ops: 221 {
[ 109.415000] HARDIRQ-ON-W at:
[ 109.420000] [<c0064b08>] __lock_acquire+0x560/0xa94
[ 109.425000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.435000] [<c045b010>]
mutex_lock_nested+0x54/0x3a8
[ 109.440000] [<c0310888>] clk_get_sys+0x24/0xf4
[ 109.445000] [<c059994c>]
exynos_pm_drvinit+0x2c/0x64
[ 109.455000] [<c05918e0>] do_one_initcall+0x50/0x164
[ 109.460000] [<c0591b0c>]
kernel_init_freeable+0x118/0xc
[ 109.470000] [<c0451494>] kernel_init+0x8/0xe4
[ 109.475000] [<c000e1b0>] ret_from_fork+0x14/0x24
[ 109.480000] SOFTIRQ-ON-W at:
[ 109.485000] [<c0064b2c>] __lock_acquire+0x584/0xa94
[ 109.490000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.495000] [<c045b010>]
mutex_lock_nested+0x54/0x3a8
[ 109.505000] [<c0310888>] clk_get_sys+0x24/0xf4
[ 109.510000] [<c059994c>]
exynos_pm_drvinit+0x2c/0x64
[ 109.515000] [<c05918e0>] do_one_initcall+0x50/0x164
[ 109.525000] [<c0591b0c>]
kernel_init_freeable+0x118/0xc
[ 109.530000] [<c0451494>] kernel_init+0x8/0xe4
[ 109.535000] [<c000e1b0>] ret_from_fork+0x14/0x24
[ 109.545000] INITIAL USE at:
[ 109.545000] [<c0064840>] __lock_acquire+0x298/0xa94
[ 109.555000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.560000] [<c045b010>]
mutex_lock_nested+0x54/0x3a8
[ 109.565000] [<c0310998>] clkdev_add+0x1c/0x3c
[ 109.575000] [<c001fcd4>]
s3c24xx_register_clock+0x34/0x0
[ 109.580000] [<c059b3b0>]
s3c24xx_register_baseclocks+0x4
[ 109.590000] [<c0598d1c>]
exynos4_init_clocks+0x1c/0x80
[ 109.595000] [<c059ae00>]
s3c24xx_init_clocks+0x44/0x54
[ 109.600000] [<c059a848>] origen_map_io+0x1c/0x34
[ 109.610000] [<c0597d6c>] paging_init+0x5ec/0x798
[ 109.615000] [<c0594c00>] setup_arch+0x3c8/0x65c
[ 109.620000] [<c0591588>] start_kernel+0x7c/0x384
[ 109.625000] [<40008078>] 0x40008078
[ 109.630000] }
[ 109.250000] g_ffs: read descriptors
[ 109.255000] g_ffs: read strings
[ 109.275000] gadget: g_ffs ready
[ 109.285000] s3c-hsotg s3c-hsotg: bound driver g_ffs
[ 109.290000]
[ 109.290000] =========================================================
[ 109.295000] [ INFO: possible irq lock inversion dependency detected ]
[ 109.305000] 3.8.0-dirty #199 Not tainted
[ 109.310000] ---------------------------------------------------------
[ 109.315000] swapper/0/0 just changed the state of lock:
[ 109.320000] (&(&hsotg->lock)->rlock){-.....}, at: [<c029d35c>] s3c_hsotg_ir8
[ 109.325000] but this lock took another, HARDIRQ-unsafe lock in the past:
[ 109.335000] (clocks_mutex){+.+...}
and interrupts could create inverse lock ordering between them.
[ 109.345000]
[ 109.345000] other info that might help us debug this:
[ 109.350000] Possible interrupt unsafe locking scenario:
[ 109.350000]
[ 109.355000] CPU0 CPU1
[ 109.360000] ---- ----
[ 109.365000] lock(clocks_mutex);
[ 109.370000] local_irq_disable();
[ 109.375000] lock(&(&hsotg->lock)->rlock);
[ 109.380000] lock(clocks_mutex);
[ 109.390000] <Interrupt>
[ 109.390000] lock(&(&hsotg->lock)->rlock);
[ 109.395000]
[ 109.395000] *** DEADLOCK ***
[ 109.395000]
[ 109.400000] no locks held by swapper/0/0.
[ 109.405000]
[ 109.405000] the shortest dependencies between 2nd lock and 1st lock:
[ 109.410000] -> (clocks_mutex){+.+...} ops: 221 {
[ 109.415000] HARDIRQ-ON-W at:
[ 109.420000] [<c0064b08>] __lock_acquire+0x560/0xa94
[ 109.425000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.435000] [<c045b010>] mutex_lock_nested+0x54/0x3a8
[ 109.440000] [<c0310888>] clk_get_sys+0x24/0xf4
[ 109.445000] [<c059994c>] exynos_pm_drvinit+0x2c/0x64
[ 109.455000] [<c05918e0>] do_one_initcall+0x50/0x164
[ 109.460000] [<c0591b0c>] kernel_init_freeable+0x118/0xc
[ 109.470000] [<c0451494>] kernel_init+0x8/0xe4
[ 109.475000] [<c000e1b0>] ret_from_fork+0x14/0x24
[ 109.480000] SOFTIRQ-ON-W at:
[ 109.485000] [<c0064b2c>] __lock_acquire+0x584/0xa94
[ 109.490000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.495000] [<c045b010>] mutex_lock_nested+0x54/0x3a8
[ 109.505000] [<c0310888>] clk_get_sys+0x24/0xf4
[ 109.510000] [<c059994c>] exynos_pm_drvinit+0x2c/0x64
[ 109.515000] [<c05918e0>] do_one_initcall+0x50/0x164
[ 109.525000] [<c0591b0c>] kernel_init_freeable+0x118/0xc
[ 109.530000] [<c0451494>] kernel_init+0x8/0xe4
[ 109.535000] [<c000e1b0>] ret_from_fork+0x14/0x24
[ 109.545000] INITIAL USE at:
[ 109.545000] [<c0064840>] __lock_acquire+0x298/0xa94
[ 109.555000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.560000] [<c045b010>] mutex_lock_nested+0x54/0x3a8
[ 109.565000] [<c0310998>] clkdev_add+0x1c/0x3c
[ 109.575000] [<c001fcd4>] s3c24xx_register_clock+0x34/0x0
[ 109.580000] [<c059b3b0>] s3c24xx_register_baseclocks+0x4
[ 109.590000] [<c0598d1c>] exynos4_init_clocks+0x1c/0x80
[ 109.595000] [<c059ae00>] s3c24xx_init_clocks+0x44/0x54
[ 109.600000] [<c059a848>] origen_map_io+0x1c/0x34
[ 109.610000] [<c0597d6c>] paging_init+0x5ec/0x798
[ 109.615000] [<c0594c00>] setup_arch+0x3c8/0x65c
[ 109.620000] [<c0591588>] start_kernel+0x7c/0x384
[ 109.625000] [<40008078>] 0x40008078
[ 109.630000] }
[ 109.635000] ... key at: [<c0624f7c>] clocks_mutex+0x38/0x48
[ 109.640000] ... acquired at:
[ 109.645000] [<c006498c>] __lock_acquire+0x3e4/0xa94
[ 109.650000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.655000] [<c045b010>] mutex_lock_nested+0x54/0x3a8
[ 109.660000] [<c0310888>] clk_get_sys+0x24/0xf4
[ 109.665000] [<c001ede0>] exynos4210_usb_phy_clkset+0x10/0x164
[ 109.670000] [<c001f128>] s5p_usb_phy_init+0x1f4/0x2b8
[ 109.675000] [<c029ce78>] s3c_hsotg_pullup+0x48/0x98
[ 109.680000] [<c0297e84>] usb_gadget_probe_driver+0x160/0x188
[ 109.685000] [<c02a02cc>] ffs_ep0_write+0x1f8/0x630
[ 109.690000] [<c00c1c1c>] vfs_write+0x9c/0x140
[ 109.695000] [<c00c1d6c>] sys_write+0x3c/0x70
[ 109.700000] [<c000e100>] ret_fast_syscall+0x0/0x3c
[ 109.705000]
[ 109.705000] -> (&(&hsotg->lock)->rlock){-.....} ops: 2 {
[ 109.710000] IN-HARDIRQ-W at:
[ 109.715000] [<c0064c14>] __lock_acquire+0x66c/0xa94
[ 109.720000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.730000] [<c045de94>] _raw_spin_lock+0x3c/0x4c
[ 109.735000] [<c029d35c>] s3c_hsotg_irq+0x28/0x7b8
[ 109.740000] [<c0071914>] handle_irq_event_percpu+0x50/0x8
[ 109.750000] [<c0071a98>] handle_irq_event+0x3c/0x5c
[ 109.755000] [<c0074364>] handle_fasteoi_irq+0x7c/0x138
[ 109.760000] [<c00712b8>] generic_handle_irq+0x24/0x38
[ 109.770000] [<c000efc4>] handle_IRQ+0x38/0x94
[ 109.775000] [<c000855c>] gic_handle_irq+0x34/0x68
[ 109.780000] [<c000dcc4>] __irq_svc+0x44/0x78
[ 109.785000] [<c000f248>] default_idle+0x20/0x30
[ 109.795000] [<c000f378>] cpu_idle+0x48/0xf4
[ 109.800000] [<c059183c>] start_kernel+0x330/0x384
[ 109.805000] INITIAL USE at:
[ 109.810000] [<c0064840>] __lock_acquire+0x298/0xa94
[ 109.815000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.820000] [<c045deec>] _raw_spin_lock_irqsave+0x48/0x5c
[ 109.830000] [<c029ce4c>] s3c_hsotg_pullup+0x1c/0x98
[ 109.835000] [<c0297e84>] usb_gadget_probe_driver+0x160/0x8
[ 109.840000] [<c02a02cc>] ffs_ep0_write+0x1f8/0x630
[ 109.850000] [<c00c1c1c>] vfs_write+0x9c/0x140
[ 109.855000] [<c00c1d6c>] sys_write+0x3c/0x70
[ 109.860000] [<c000e100>] ret_fast_syscall+0x0/0x3c
[ 109.865000] }
[ 109.870000] ... key at: [<c0b43060>] __key.23369+0x0/0x8
[ 109.875000] ... acquired at:
[ 109.875000] [<c00626f4>] check_usage_forwards+0xa0/0x10c
[ 109.880000] [<c00640ec>] mark_lock+0x1c0/0x67c
[ 109.885000] [<c0064c14>] __lock_acquire+0x66c/0xa94
[ 109.890000] [<c0066684>] lock_acquire+0x60/0x74
[ 109.895000] [<c045de94>] _raw_spin_lock+0x3c/0x4c
[ 109.900000] [<c029d35c>] s3c_hsotg_irq+0x28/0x7b8
[ 109.905000] [<c0071914>] handle_irq_event_percpu+0x50/0x198
[ 109.910000] [<c0071a98>] handle_irq_event+0x3c/0x5c
[ 109.920000] [<c0074364>] handle_fasteoi_irq+0x7c/0x138
[ 109.925000] [<c00712b8>] generic_handle_irq+0x24/0x38
[ 109.930000] [<c000efc4>] handle_IRQ+0x38/0x94
[ 109.935000] [<c000855c>] gic_handle_irq+0x34/0x68
[ 109.940000] [<c000dcc4>] __irq_svc+0x44/0x78
[ 109.940000] [<c000f248>] default_idle+0x20/0x30
[ 109.945000] [<c000f378>] cpu_idle+0x48/0xf4
[ 109.950000] [<c059183c>] start_kernel+0x330/0x384
[ 109.955000]
[ 109.960000]
[ 109.960000] stack backtrace:
[ 109.960000] [<c00143d8>] (unwind_backtrace+0x0/0xf8) from [<c00625ec>] (prin)
[ 109.970000] [<c00625ec>] (print_irq_inversion_bug+0x1a4/0x20c) from [<c00626)
[ 109.980000] [<c00626f4>] (check_usage_forwards+0xa0/0x10c) from [<c00640ec>])
[ 109.990000] [<c00640ec>] (mark_lock+0x1c0/0x67c) from [<c0064c14>] (__lock_a)
[ 110.000000] [<c0064c14>] (__lock_acquire+0x66c/0xa94) from [<c0066684>] (loc)
[ 110.010000] [<c0066684>] (lock_acquire+0x60/0x74) from [<c045de94>] (_raw_sp)
[ 110.020000] [<c045de94>] (_raw_spin_lock+0x3c/0x4c) from [<c029d35c>] (s3c_h)
[ 110.025000] [<c029d35c>] (s3c_hsotg_irq+0x28/0x7b8) from [<c0071914>] (handl)
[ 110.035000] [<c0071914>] (handle_irq_event_percpu+0x50/0x198) from [<c0071a9)
[ 110.045000] [<c0071a98>] (handle_irq_event+0x3c/0x5c) from [<c0074364>] (han)
[ 110.055000] [<c0074364>] (handle_fasteoi_irq+0x7c/0x138) from [<c00712b8>] ()
[ 110.065000] [<c00712b8>] (generic_handle_irq+0x24/0x38) from [<c000efc4>] (h)
[ 110.075000] [<c000efc4>] (handle_IRQ+0x38/0x94) from [<c000855c>] (gic_handl)
[ 110.080000] [<c000855c>] (gic_handle_irq+0x34/0x68) from [<c000dcc4>] (__irq)
[ 110.090000] Exception stack(0xc05c7f58 to 0xc05c7fa0)
[ 110.095000] 7f40: 00001
[ 110.105000] 7f60: 00000000 00000000 c05c6000 c05c6000 c062f648 c04616f8 c05c1
[ 110.110000] 7f80: c05ec448 00000000 c0605854 c05c7fa0 c00654c4 c000f248 2000f
[ 110.120000] [<c000dcc4>] (__irq_svc+0x44/0x78) from [<c000f248>] (default_id)
[ 110.130000] [<c000f248>] (default_idle+0x20/0x30) from [<c000f378>] (cpu_idl)
[ 110.135000] [<c000f378>] (cpu_idle+0x48/0xf4) from [<c059183c>] (start_kerne)
[ 110.145000] s3c-hsotg s3c-hsotg: new device is high-speed
[ 110.150000] s3c-hsotg s3c-hsotg: s3c_hsotg_irq: USBRst
[ 110.160000] s3c-hsotg s3c-hsotg: GINTSTS_USBSusp
[ 110.160000] s3c-hsotg s3c-hsotg: s3c_hsotg_rx_data: FIFO 8 bytes on ep0 but )
[ 110.170000] s3c-hsotg s3c-hsotg: s3c_hsotg_irq: USBRst
[ 110.175000] s3c-hsotg s3c-hsotg: s3c_hsotg_rx_data: FIFO 8 bytes on ep0 but )
[ 110.185000] s3c-hsotg s3c-hsotg: new device is high-speed
[ 110.190000] ------------[ cut here ]------------
[ 110.195000] WARNING: at drivers/usb/gadget/s3c-hsotg.c:1464 s3c_hsotg_rx_dat)
[ 110.205000] [<c00143d8>] (unwind_backtrace+0x0/0xf8) from [<c0023680>] (warn)
[ 110.210000] [<c0023680>] (warn_slowpath_common+0x48/0x60) from [<c0023760>] )
[ 110.220000] [<c0023760>] (warn_slowpath_null+0x1c/0x24) from [<c029c1f8>] (s)
[ 110.230000] [<c029c1f8>] (s3c_hsotg_rx_data+0xe8/0x104) from [<c029da34>] (s)
[ 110.240000] [<c029da34>] (s3c_hsotg_irq+0x700/0x7b8) from [<c0071914>] (hand)
[ 110.250000] [<c0071914>] (handle_irq_event_percpu+0x50/0x198) from [<c0071a9)
[ 110.260000] [<c0071a98>] (handle_irq_event+0x3c/0x5c) from [<c0074364>] (han)
[ 110.270000] [<c0074364>] (handle_fasteoi_irq+0x7c/0x138) from [<c00712b8>] ()
[ 110.280000] [<c00712b8>] (generic_handle_irq+0x24/0x38) from [<c000efc4>] (h)
[ 110.290000] [<c000efc4>] (handle_IRQ+0x38/0x94) from [<c000855c>] (gic_handl)
[ 110.295000] [<c000855c>] (gic_handle_irq+0x34/0x68) from [<c000dcc4>] (__irq)
[ 110.305000] Exception stack(0xc05c7f58 to 0xc05c7fa0)
[ 110.310000] 7f40: 00001
[ 110.320000] 7f60: 00000000 00000000 c05c6000 c05c6000 c062f648 c04616f8 c05c1
[ 110.325000] 7f80: c05ec448 00000000 c0605854 c05c7fa0 c00654c4 c000f248 2000f
[ 110.335000] [<c000dcc4>] (__irq_svc+0x44/0x78) from [<c000f248>] (default_id)
[ 110.340000] [<c000f248>] (default_idle+0x20/0x30) from [<c000f378>] (cpu_idl)
[ 110.350000] [<c000f378>] (cpu_idle+0x48/0xf4) from [<c059183c>] (start_kerne)
[ 110.360000] ---[ end trace 13036207479db903 ]---
[ 110.365000] s3c-hsotg s3c-hsotg: GINTSTS_USBSusp
[ 110.370000] Unable to handle kernel paging request at virtual address 00ec000
[ 110.375000] pgd = c0004000
[ 110.380000] [00ec0090] *pgd=00000000
[ 110.380000] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[ 110.385000] CPU: 0 Tainted: G W (3.8.0-dirty #199)
[ 110.395000] PC is at composite_suspend+0x88/0xa4
[ 110.395000] LR is at composite_suspend+0x10/0xa4
[ 110.400000] pc : [<c0298974>] lr : [<c02988fc>] psr: 60000193
[ 110.400000] sp : c05c7e90 ip : c060bc40 fp : 00000001
[ 110.415000] r10: c05ccf00 r9 : c05e7ff4 r8 : 00000067
[ 110.420000] r7 : 00000c10 r6 : ed319488 r5 : ecbb2400 r4 : ed319410
[ 110.425000] r3 : 00ec0080 r2 : 00000000 r1 : 00000001 r0 : ecbb2400
[ 110.430000] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kel
[ 110.440000] Control: 10c5387d Table: 6c56804a DAC: 00000015
[ 110.445000] Process swapper/0 (pid: 0, stack limit = 0xc05c6238)
[ 110.450000] Stack: (0xc05c7e90 to 0xc05c8000)
[ 110.455000] 7e80: ed319410 0000000f 00000
[ 110.465000] 7ea0: 00000102 00000004 c05c6000 ed31941c 00000001 ec84ea40 c05c0
[ 110.470000] 7ec0: 00000000 00000067 c062f4be c05ccf00 00000001 c0071914 00000
[ 110.480000] 7ee0: c05ccf50 c05ccf00 c05ccf50 ec84ea40 f8810000 c05c6000 412f8
[ 110.490000] 7f00: 00000000 c0071a98 c05ccf00 c05ccf50 00000000 c0074364 c0078
[ 110.495000] 7f20: 00000067 c00712b8 00000225 c000efc4 f881000c c05e8018 c05cc
[ 110.505000] 7f40: c00654c4 c000f248 20000013 ffffffff c05c7f8c c000dcc4 00001
[ 110.510000] 7f60: 00000000 00000000 c05c6000 c05c6000 c062f648 c04616f8 c05c1
[ 110.520000] 7f80: c05ec448 00000000 c0605854 c05c7fa0 c00654c4 c000f248 2000f
[ 110.530000] 7fa0: c000f228 c000f378 c05c6000 c05e811c c062f580 ffffffff c05b0
[ 110.535000] 7fc0: 00000000 c059183c ffffffff ffffffff c05912e8 00000000 00008
[ 110.545000] 7fe0: 10c5387d c05e7ff0 c05b7be4 c05ec43c 4000406a 40008078 00000
[ 110.555000] [<c0298974>] (composite_suspend+0x88/0xa4) from [<c029d970>] (s3)
[ 110.560000] [<c029d970>] (s3c_hsotg_irq+0x63c/0x7b8) from [<c0071914>] (hand)
[ 110.570000] [<c0071914>] (handle_irq_event_percpu+0x50/0x198) from [<c0071a9)
[ 110.580000] [<c0071a98>] (handle_irq_event+0x3c/0x5c) from [<c0074364>] (han)
[ 110.590000] [<c0074364>] (handle_fasteoi_irq+0x7c/0x138) from [<c00712b8>] ()
[ 110.600000] [<c00712b8>] (generic_handle_irq+0x24/0x38) from [<c000efc4>] (h)
[ 110.610000] [<c000efc4>] (handle_IRQ+0x38/0x94) from [<c000855c>] (gic_handl)
[ 110.620000] [<c000855c>] (gic_handle_irq+0x34/0x68) from [<c000dcc4>] (__irq)
[ 110.625000] Exception stack(0xc05c7f58 to 0xc05c7fa0)
[ 110.630000] 7f40: 00001
[ 110.640000] 7f60: 00000000 00000000 c05c6000 c05c6000 c062f648 c04616f8 c05c1
[ 110.645000] 7f80: c05ec448 00000000 c0605854 c05c7fa0 c00654c4 c000f248 2000f
[ 110.655000] [<c000dcc4>] (__irq_svc+0x44/0x78) from [<c000f248>] (default_id)
[ 110.665000] [<c000f248>] (default_idle+0x20/0x30) from [<c000f378>] (cpu_idl)
[ 110.670000] [<c000f378>] (cpu_idle+0x48/0xf4) from [<c059183c>] (start_kerne)
[ 110.680000] Code: e5d5300c e3833001 e5c5300c e5963000 (e5933010)
[ 110.685000] ---[ end trace 13036207479db904 ]---
[ 110.690000] Kernel panic - not syncing: Fatal exception in interrupt
[ 110.695000] CPU1: stopping
[ 110.695000] [<c00143d8>] (unwind_backtrace+0x0/0xf8) from [<c0012d54>] (hand)
[ 110.695000] [<c0012d54>] (handle_IPI+0x124/0x150) from [<c0008588>] (gic_han)
[ 110.695000] [<c0008588>] (gic_handle_irq+0x60/0x68) from [<c000dcc4>] (__irq)
[ 110.695000] Exception stack(0xed071f88 to 0xed071fd0)
[ 110.695000] 1f80: c000f244 ed071fd0 00000000 00000000 ed070
[ 110.695000] 1fa0: c062f648 c04616f8 ed070000 412fc091 c05ec448 00000000 c0600
[ 110.695000] 1fc0: c000f244 c000f248 60000013 ffffffff
[ 110.695000] [<c000dcc4>] (__irq_svc+0x44/0x78) from [<c000f248>] (default_id)
[ 110.695000] [<c000f248>] (default_idle+0x20/0x30) from [<c000f378>] (cpu_idl)
[ 110.695000] [<c000f378>] (cpu_idle+0x48/0xf4) from [<40452248>] (0x40452248)