Re: [PATCH v2] panic: Avoid the extra noise dmesg
From: Feng Tang
Date: Mon Nov 26 2018 - 22:25:59 EST
Hi Andrew,
On Mon, Nov 26, 2018 at 03:59:52PM -0800, Andrew Morton wrote:
> On Thu, 8 Nov 2018 21:05:03 +0800 Feng Tang <feng.tang@xxxxxxxxx> wrote:
>
> > Sometimes when debugging kernel panic, we saw many extra noisy error
> > messages after the expected end:
> >
> > [ 35.743249] ---[ end Kernel panic - not syncing: Fatal exception
> > [ 35.749975] ------------[ cut here ]------------
>
> What are these "noisy messages"? The above looks OK?
Yes, the above is the sane messages :) and contains the necessary context
info for debug.
The noisy messages are those following messages, I tried to capture one with
sysrq and paste it in the tail of this mail as it's big (for a 4 cores
machine)
> > When panic happens under console mode, the screen will first show the panic
> > call stack, but will immediately be overrided by noisy extra messages, like
> > this for every CPU except the panic one:
> >
> > WARNING: CPU: 1 PID: 280 at kernel/sched/core.c:1198 set_task_cpu+0x183/0x190
> > Call Trace:
> > <IRQ>
> > try_to_wake_up
> > default_wake_function
> > autoremove_wake_function
> > __wake_up_common
> > __wake_up_common_lock
> > __wake_up
> > wake_up_klogd_work_func
> > irq_work_run_list
> > irq_work_tick
> > update_process_times
> > tick_sched_timer
> > __hrtimer_run_queues
> > hrtimer_interrupt
> > smp_apic_timer_interrupt
> > apic_timer_interrupt
>
> And this output looks OK as well.
Yes, this is fine also. What I mean is it will be a little noisy if each and
every CPU will dump similar info, which will make it hard to find the real panic
context if a user has no serial console but a LCD.
> > Which make debugging more difficult, as the original context is
> > lost on screen.
> >
> > Keeping the CPU IRQ disabled will void these messages, as we code
> > run to this point, the user has chosed not to reboot, nor do
> > some extra handling with panic notifier, not much point in
> > re-enabling the interrupt.
>
> And please have another attempt at the above paragraph?
Sorry for the unclear info, will try to sanitize it.
> > --- a/kernel/panic.c
> > +++ b/kernel/panic.c
> > @@ -295,7 +295,6 @@ void panic(const char *fmt, ...)
> > }
> > #endif
> > pr_emerg("---[ end Kernel panic - not syncing: %s ]---\n", buf);
> > - local_irq_enable();
> > for (i = 0; ; i += PANIC_TIMER_STEP) {
> > touch_softlockup_watchdog();
> > if (i >= i_next) {
>
> The patch looks reasonable, but the description is just too hard for me
> to follow, sorry. Please spend a bit more time over that and resend?
Sure and thanks for your review!
- Feng
-------
Panic log:
#echo c > /proc/sysrq-trigger
[ 62.918474] sysrq: SysRq : Trigger a crash
[ 62.918643] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 62.918906] PGD 67e36067 P4D 67e36067 PUD 6189f067 PMD 0
[ 62.919092] Oops: 0002 [#1] PREEMPT SMP
[ 62.919226] CPU: 3 PID: 336 Comm: bash Tainted: G U W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 62.919563] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 62.919726] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 62.920332] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 62.920508] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 62.920744] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 62.920979] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 62.921216] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 62.921457] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 62.921693] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.921959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.922150] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.922385] Call Trace:
[ 62.922476] __handle_sysrq+0xef/0x140
[ 62.922605] write_sysrq_trigger+0x4c/0x50
[ 62.922746] proc_reg_write+0x40/0x70
[ 62.922874] __vfs_write+0x35/0x180
[ 62.922997] ? __this_cpu_preempt_check+0x13/0x20
[ 62.923156] vfs_write+0xc5/0x1c0
[ 62.923271] ksys_write+0x53/0xc0
[ 62.923388] ? __audit_syscall_exit+0x20d/0x2a0
[ 62.923541] __x64_sys_write+0x15/0x20
[ 62.923671] do_syscall_64+0x4e/0x100
[ 62.923800] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 62.923970] RIP: 0033:0x7f3d2fca0664
[ 62.924093] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 62.924699] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 62.924949] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 62.925184] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 62.925420] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 62.925655] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 62.925890] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 62.926126] Modules linked in:
[ 62.926234] CR2: 0000000000000000
[ 62.926349] ---[ end trace 416678827625f99b ]---
[ 62.936473] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 62.936644] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 62.937260] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 62.937438] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 62.937677] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 62.937917] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 62.938158] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 62.938396] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 62.938636] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.938905] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.939099] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.939339] Kernel panic - not syncing: Fatal exception
[ 62.939941] reboot: panic mode set: p,w
[ 62.940100] Kernel Offset: 0x38000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 62.950783] ---[ end Kernel panic - not syncing: Fatal exception ]---
[ 62.951034] ------------[ cut here ]------------
[ 62.951201] sched: Unexpected reschedule of offline CPU#1!
[ 62.951409] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 62.951732] Modules linked in:
[ 62.951848] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 62.952199] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 62.952394] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 62.953037] RSP: 0018:ffff9939b4b83cd8 EFLAGS: 00010082
[ 62.953222] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
[ 62.953472] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 62.953721] RBP: ffff9939b4b83cd8 R08: 0000000000000000 R09: 0000000000000c65
[ 62.953970] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000000
[ 62.954219] R13: ffff9939a46e28c8 R14: ffff9939a46e2140 R15: 0000000000020480
[ 62.954469] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 62.954753] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.954955] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 62.955205] Call Trace:
[ 62.955299] <IRQ>
[ 62.955380] try_to_wake_up+0x386/0x3f0
[ 62.955521] default_wake_function+0xd/0x10
[ 62.955673] autoremove_wake_function+0x11/0x60
[ 62.955836] __wake_up_common+0x82/0x150
[ 62.955979] ? __queue_work+0x125/0x330
[ 62.956119] __wake_up_common_lock+0x7b/0xc0
[ 62.956273] __wake_up+0xe/0x10
[ 62.956390] wake_up_klogd_work_func+0x47/0x70
[ 62.956552] irq_work_run_list+0x55/0x80
[ 62.956694] irq_work_tick+0x55/0x60
[ 62.956825] update_process_times+0x3d/0x50
[ 62.956977] tick_sched_handle+0x24/0x60
[ 62.957120] tick_sched_timer+0x37/0x80
[ 62.957259] ? tick_sched_do_timer+0x60/0x60
[ 62.957413] __hrtimer_run_queues+0xf3/0x190
[ 62.957567] hrtimer_interrupt+0x11d/0x250
[ 62.957718] smp_apic_timer_interrupt+0x60/0xa0
[ 63.577258] apic_timer_interrupt+0xf/0x20
[ 63.579677] </IRQ>
[ 63.579759] RIP: 0010:panic+0x19b/0x1f4
[ 63.579897] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 63.580536] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 63.580800] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 63.581048] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 63.581296] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 63.581544] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 63.581792] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 63.582046] oops_end+0x9d/0xb0
[ 63.582163] no_context+0x12d/0x320
[ 63.582290] __bad_area_nosemaphore+0x7c/0x1d0
[ 63.582450] bad_area_nosemaphore+0xf/0x20
[ 63.582598] __do_page_fault+0xa7/0x570
[ 63.582737] do_page_fault+0x22/0x30
[ 63.582869] page_fault+0x1e/0x30
[ 63.582992] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 63.583163] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 63.583802] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 63.583986] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 63.584234] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 63.584482] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 63.584730] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 63.584978] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 63.585229] ? sysrq_handle_crash+0x9/0x20
[ 63.585377] __handle_sysrq+0xef/0x140
[ 63.585513] write_sysrq_trigger+0x4c/0x50
[ 63.585662] proc_reg_write+0x40/0x70
[ 63.585795] __vfs_write+0x35/0x180
[ 63.585925] ? __this_cpu_preempt_check+0x13/0x20
[ 63.586093] vfs_write+0xc5/0x1c0
[ 63.586214] ksys_write+0x53/0xc0
[ 63.586336] ? __audit_syscall_exit+0x20d/0x2a0
[ 63.586498] __x64_sys_write+0x15/0x20
[ 63.586635] do_syscall_64+0x4e/0x100
[ 63.586768] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 63.586948] RIP: 0033:0x7f3d2fca0664
[ 63.587078] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 63.587716] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 63.587980] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 63.588228] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 63.588476] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 63.588724] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 63.588972] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 63.589222] ---[ end trace 416678827625f99c ]---
[ 63.589394] WARNING: CPU: 3 PID: 336 at kernel/sched/core.c:1163 set_task_cpu+0x145/0x150
[ 63.589680] Modules linked in:
[ 63.589793] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 63.590143] RIP: 0010:set_task_cpu+0x145/0x150
[ 63.590302] Code: 5e 5d c3 80 8b 94 04 00 00 04 e9 76 ff ff ff 0f 0b e9 00 ff ff ff f7 43 60 fd ff ff ff 0f 84 0a ff ff ff 0f 0b e9 03 ff ff ff <0f> 0b e9 12 ff ff ff 0f 1f 40 00 55 48 89 e5 41 56 49 89 f6 41 55
[ 63.590943] RSP: 0018:ffff9939b4b83c10 EFLAGS: 00010046
[ 63.591128] RAX: 0000000000000200 RBX: ffff9939a7d26180 RCX: ffff9939a7d26180
[ 63.591377] RDX: ffff9939b4b20401 RSI: 0000000000000002 RDI: ffff9939a7d26180
[ 63.591625] RBP: ffff9939b4b83c30 R08: ffff9939a7d2be78 R09: 0000000000000000
[ 63.591874] R10: 0000000000000046 R11: ffffffffba932f0d R12: 0000000000000002
[ 63.592122] R13: ffff9939a7d26908 R14: 0000000000000002 R15: ffff9939a7d264f0
[ 63.592371] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 63.592653] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 63.592855] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 63.593103] Call Trace:
[ 63.593194] <IRQ>
[ 63.593272] try_to_wake_up+0x13a/0x3f0
[ 63.593413] default_wake_function+0xd/0x10
[ 63.593563] __wake_up_common+0x82/0x150
[ 63.593706] __wake_up_locked+0x11/0x20
[ 63.593847] ep_poll_callback+0xcc/0x330
[ 63.593989] __wake_up_common+0x82/0x150
[ 63.594131] ? __queue_work+0x125/0x330
[ 63.594270] __wake_up_common_lock+0x7b/0xc0
[ 63.594424] __wake_up+0xe/0x10
[ 63.594540] wake_up_klogd_work_func+0x47/0x70
[ 63.594699] irq_work_run_list+0x55/0x80
[ 63.594841] irq_work_tick+0x55/0x60
[ 63.594971] update_process_times+0x3d/0x50
[ 63.595122] tick_sched_handle+0x24/0x60
[ 63.595264] tick_sched_timer+0x37/0x80
[ 63.595403] ? tick_sched_do_timer+0x60/0x60
[ 63.595557] __hrtimer_run_queues+0xf3/0x190
[ 63.595711] hrtimer_interrupt+0x11d/0x250
[ 63.595859] smp_apic_timer_interrupt+0x60/0xa0
[ 63.596021] apic_timer_interrupt+0xf/0x20
[ 63.596168] </IRQ>
[ 63.596249] RIP: 0010:panic+0x19b/0x1f4
[ 63.596387] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 63.597027] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 63.597292] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 63.597540] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 63.597789] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.191896] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.192145] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.192398] oops_end+0x9d/0xb0
[ 64.192514] no_context+0x12d/0x320
[ 64.192642] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.192802] bad_area_nosemaphore+0xf/0x20
[ 64.192949] __do_page_fault+0xa7/0x570
[ 64.193089] do_page_fault+0x22/0x30
[ 64.193220] page_fault+0x1e/0x30
[ 64.193342] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.193513] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.194154] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.194339] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.194588] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.194837] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.195085] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.195334] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.195585] ? sysrq_handle_crash+0x9/0x20
[ 64.195734] __handle_sysrq+0xef/0x140
[ 64.195870] write_sysrq_trigger+0x4c/0x50
[ 64.196018] proc_reg_write+0x40/0x70
[ 64.196151] __vfs_write+0x35/0x180
[ 64.196279] ? __this_cpu_preempt_check+0x13/0x20
[ 64.196447] vfs_write+0xc5/0x1c0
[ 64.196569] ksys_write+0x53/0xc0
[ 64.196690] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.196853] __x64_sys_write+0x15/0x20
[ 64.196988] do_syscall_64+0x4e/0x100
[ 64.197122] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.197302] RIP: 0033:0x7f3d2fca0664
[ 64.197431] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.394050] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 64.394315] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 64.394563] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 64.394812] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 64.395061] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 64.395309] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 64.395559] ---[ end trace 416678827625f99d ]---
[ 64.395735] ------------[ cut here ]------------
[ 64.395899] sched: Unexpected reschedule of offline CPU#2!
[ 64.396095] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 64.396417] Modules linked in:
[ 64.396531] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 64.396880] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 64.397074] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 64.397715] RSP: 0018:ffff9939b4b83ba0 EFLAGS: 00010 64.532707] irq_work_tick+0x55/0x60
[ 64.532838] update_process_times+0x3d/0x50
[ 64.532988] tick_sched_handle+0x24/0x60
[ 64.533130] tick_sched_timer+0x37/0x80
[ 64.533270] ? tick_sched_do_timer+0x60/0x60
[ 64.533423] __hrtimer_run_queues+0xf3/0x190
[ 64.533577] hrtimer_interrupt+0x11d/0x250
[ 64.533725] smp_apic_timer_interrupt+0x60/0xa0
[ 64.533887] apic_timer_interrupt+0xf/0x20
[ 64.534034] </IRQ>
[ 64.534115] RIP: 0010:panic+0x19b/0x1f4
[ 64.534253] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 64.534894] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 64.535158] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 64.535407] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.535655] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.535904] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.536153] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.536405] oops_end+0x9d/0xb0
[ 64.536521] no_context+0x12d/0x320
[ 64.536649] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.536809] bad_area_nosemaphore+0xf/0x20
[ 64.536956] __do_page_fault+0xa7/0x570
[ 64.537096] do_page_fault+0x22/0x30
[ 64.537227] page_fault+0x1e/0x30
[ 64.537349] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.537520] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.538164] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.538349] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.538598] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.538846] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.539095] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.539343] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.539594] ? sysrq_handle_crash+0x9/0x20
[ 64.539743] __handle_sysrq+0xef/0x140
[ 64.539879] write_sysrq_trigger+0x4c/0x50
[ 64.540027] proc_reg_write+0x40/0x70
[ 64.540160] __vfs_write+0x35/0x180
[ 64.540289] ? __this_cpu_preempt_check+0x13/0x20
[ 64.540457] vfs_write+0xc5/0x1c0
[ 64.540578] ksys_write+0x53/0xc0
[ 64.540700] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.540862] __x64_sys_write+0x15/0x20
[ 64.540998] do_syscall_64+0x4e/0x100
[ 64.541132] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.541311] RIP: 0033:0x7f3d2fca0664
[ 64.541441] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.542080] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 64.542345] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 64.542593] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 64.542842] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 64.543090] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 64.543339] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 64.543589] ---[ end trace 416678827625f99e ]---
[ 64.543775] ------------[ cut here ]------------
[ 64.543942] sched: Unexpected reschedule of offline CPU#0!
[ 64.544138] WARNING: CPU: 3 PID: 336 at arch/x86/kernel/smp.c:141 native_smp_send_reschedule+0x3a/0x40
[ 64.544460] Modules linked in:
[ 64.544573] CPU: 3 PID: 336 Comm: bash Tainted: G UD W 4.19.0-19.iot-lts2018-sos-g6b95688-dirty #44
[ 64.544923] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 64.545117] Code: c0 84 c0 74 17 48 8b 05 c4 40 54 01 be fd 00 00 00 48 8b 40 30 e8 f6 a8 dc 00 5d c3 89 fe 48 c7 c7 00 7a 45 ba e8 e6 e2 05 00 <0f> 0b 5d c3 66 90 55 48 89 e5 41 54 53 48 83 ec 20 65 48 8b 04 25
[ 64.545755] RSP: 0018:ffff9939b4b83e28 EFLAGS: 00010086
[ 64.545939] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000
[ 64.546187] RDX: 0000000000000007 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.546434] RBP: ffff9939b4b83e28 R08: 0000000000000000 R09: 0000000000000d5e
[ 64.546682] R10: 0000000000000400 R11: ffffffffba932f0d R12: 0000000000000000
[ 64.546930] R13: 0000000000000003 R14: ffff9939a3650280 R15: ffff9939b4b9be00
[ 64.547178] FS: 00007f3d2fb94740(0000) GS:ffff9939b4b80000(0000) knlGS:0000000000000000
[ 64.547458] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 64.547659] CR2: 0000000000000000 CR3: 000000006185d000 CR4: 00000000003406e0
[ 64.547906] Call Trace:
[ 64.547997] <IRQ>
[ 64.548074] kick_ilb+0x8b/0xa0
[ 64.548190] trigger_load_balance+0xb1/0x1a0
[ 64.548344] scheduler_tick+0xa4/0xd0
[ 64.548477] update_process_times+0x42/0x50
[ 64.548627] tick_sched_handle+0x24/0x60
[ 64.548769] tick_sched_timer+0x37/0x80
[ 64.548908] ? tick_sched_do_timer+0x60/0x60
[ 64.549060] __hrtimer_run_queues+0xf3/0x190
[ 64.549214] hrtimer_interrupt+0x11d/0x250
[ 64.549362] smp_apic_timer_interrupt+0x60/0xa0
[ 64.549523] apic_timer_interrupt+0xf/0x20
[ 64.549670] </IRQ>
[ 64.549750] RIP: 0010:panic+0x19b/0x1f4
[ 64.549888] Code: 89 01 90 69 09 b9 48 c7 c6 80 c3 92 ba 48 c7 c7 e0 e1 45 ba c7 05 15 d7 59 01 00 00 00 00 e8 ee 56 05 00 fb 66 0f 1f 44 00 00 <45> 31 e4 45 31 f6 31 db e8 9c d6 0a 00 4c 39 f3 7c 1d 41 83 f4 01
[ 64.550525] RSP: 0018:ffff9939a5ed3af0 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
[ 64.550789] RAX: 0000000000000039 RBX: 0000000000000000 RCX: 0000000000000000
[ 64.551036] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 00000000ffffffff
[ 64.551284] RBP: ffff9939a5ed3b68 R08: 0000000000000000 R09: 0000000000000c63
[ 64.551532] R10: 000000000000064b R11: ffffffffba932f0d R12: ffffffffba450f06
[ 64.551779] R13: ffff9939a5ed3b78 R14: 0000000000000046 R15: 0000000000000000
[ 64.552031] oops_end+0x9d/0xb0
[ 64.552146] no_context+0x12d/0x320
[ 64.552274] __bad_area_nosemaphore+0x7c/0x1d0
[ 64.552433] bad_area_nosemaphore+0xf/0x20
[ 64.552580] __do_page_fault+0xa7/0x570
[ 64.552720] do_page_fault+0x22/0x30
[ 64.552850] page_fault+0x1e/0x30
[ 64.552972] RIP: 0010:sysrq_handle_crash+0x16/0x20
[ 64.553141] Code: d5 e8 8e c2 bc ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 e8 77 31 c3 ff c7 05 65 9e 16 01 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 55 48 89 e5 fb 66 0f 1f 44 00 00 e8
[ 64.553779] RSP: 0018:ffff9939a5ed3d98 EFLAGS: 00010246
[ 64.553964] RAX: 0000000000000000 RBX: 0000000000000063 RCX: 0000000000000000
[ 64.554212] RDX: 0000000000000000 RSI: ffffffffba47f75b RDI: 0000000000000063
[ 64.554460] RBP: ffff9939a5ed3d98 R08: 0000000000000000 R09: 0000000000000c2e
[ 64.554709] R10: 0000000000000000 R11: ffffffffba932f0d R12: 0000000000000003
[ 64.554957] R13: 0000000000000000 R14: ffffffffba6807e0 R15: 0000000000000000
[ 64.555207] ? sysrq_handle_crash+0x9/0x20
[ 64.555355] __handle_sysrq+0xef/0x140
[ 64.555491] write_sysrq_trigger+0x4c/0x50
[ 64.555638] proc_reg_write+0x40/0x70
[ 64.555771] __vfs_write+0x35/0x180
[ 64.555898] ? __this_cpu_preempt_check+0x13/0x20
[ 64.556066] vfs_write+0xc5/0x1c0
[ 64.556187] ksys_write+0x53/0xc0
[ 64.556308] ? __audit_syscall_exit+0x20d/0x2a0
[ 64.556470] __x64_sys_write+0x15/0x20
[ 64.556606] do_syscall_64+0x4e/0x100
[ 64.556739] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 64.556918] RIP: 0033:0x7f3d2fca0664
[ 64.557047] Code: 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 48 8d 05 99 20 0e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 41 54 49 89 d4 55 48 89 f5 53
[ 64.557685] RSP: 002b:00007ffcf9071e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 65.374953] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f3d2fca0664
[ 65.376235] RDX: 0000000000000002 RSI: 0000000000ee8190 RDI: 0000000000000001
[ 65.376482] RBP: 0000000000ee8190 R08: 000000000000000a R09: 00007f3d2fb94740
[ 65.376730] R10: 00000000004b8f41 R11: 0000000000000246 R12: 00007f3d2fd7e760
[ 65.376978] R13: 0000000000000002 R14: 00007f3d2fd7a280 R15: 00007f3d2fd79740
[ 65.377228] ---[ end trace 416678827625f99f ]---