printk() from NMI backtrace can delay a lot
From: Tetsuo Handa
Date: Mon Jul 02 2018 - 06:26:51 EST
Hello.
Today I was testing conditions when/how stall watchdog fires. I noticed that
printing NMI backtraces to consoles is delayed till IRQ is enabled or somebody
else schedules printk(). This is not a welcomed behavior when the cause of
lock up is doing nearly-infinite loop with IRQ disabled. Can we improve this?
----------
static int __init ccs_init(void)
{
unsigned long stamp = jiffies;
local_irq_disable();
while (time_before(jiffies, stamp + 130 * HZ)) /* Emulating a lockup bug. */
;
local_irq_enable();
return -EINVAL;
}
----------
Below samples used netconsole in order to show the lag between printk() saved uptime
and printk() wrote the pending lines.
Loading the module after
echo 25 > /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout
----------
2018-07-02 12:11:40 192.168.159.129:6666 [ 58.090624] netconsole: network logging started
2018-07-02 12:12:11 192.168.159.129:6666 [ 89.017794] akari_test: loading out-of-tree module taints kernel.
2018-07-02 12:12:41 192.168.159.129:6666 [ 119.024961] INFO: rcu_sched detected stalls on CPUs/tasks:
2018-07-02 12:12:41 192.168.159.129:6666 [ 119.026970] 2-...0: (0 ticks this GP) idle=0be/1/4611686018427387904 softirq=5544/5544 fqs=7077
2018-07-02 12:12:41 192.168.159.129:6666 [ 119.030076] (detected by 3, t=30005 jiffies, g=2553, c=2552, q=94)
2018-07-02 12:12:41 192.168.159.129:6666 [ 119.032142] Sending NMI from CPU 3 to CPUs 2:
/* Here is a 32 seconds of lag which is flushed by */
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033877] NMI backtrace for cpu 2
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033884] CPU: 2 PID: 10038 Comm: insmod Tainted: G O 4.18.0-rc3 #643
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033886] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033895] RIP: 0010:__sanitizer_cov_trace_pc+0x20/0x50
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033896] Code: 5d c3 90 90 90 90 90 90 90 90 55 65 48 8b 04 25 40 5e 01 00 48 89 e5 65 8b 15 6c f6 d1 7e 81 e2 00 01 1f 00 48 8b 4d 08 75 2b <8b> 90 80 12 00 00 83 fa 02 75 20 48 8b b0 88 12 00 00 8b 80 84 12
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033942] RSP: 0018:ffffc9000205fc60 EFLAGS: 00000046
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033945] RAX: ffff880136312080 RBX: 00000000fffcc7e1 RCX: ffffffffa000d038
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033947] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa000d033
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033949] RBP: ffffc9000205fc60 R08: 0000000000000001 R09: 0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033951] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa000d000
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033953] R13: ffff8801351cd940 R14: 0000000000000001 R15: ffffc9000205fe78
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033956] FS: 00007f90d3e62740(0000) GS:ffff88013a680000(0000) knlGS:0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033958] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033961] CR2: 00000000015051d8 CR3: 0000000136a66004 CR4: 00000000000606e0
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.033998] Call Trace:
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034006] ccs_init+0x38/0x79 [akari_test]
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034011] do_one_initcall+0x7a/0x460
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034016] ? trace_hardirqs_on+0xd/0x10
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034021] do_init_module+0x5a/0x1e0
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034024] load_module+0x2144/0x2800
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034030] __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034033] ? __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034038] __x64_sys_finit_module+0x15/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034041] do_syscall_64+0x81/0x260
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034047] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034050] RIP: 0033:0x7f90d3335ec9
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034050] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 cf 2c 00 f7 d8 64 89 01 48
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034096] RSP: 002b:00007ffce0ff8338 EFLAGS: 00000202 ORIG_RAX: 0000000000000139
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034100] RAX: ffffffffffffffda RBX: 00000000015051e0 RCX: 00007f90d3335ec9
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034102] RDX: 0000000000000000 RSI: 000000000041aa0e RDI: 0000000000000003
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034104] RBP: 000000000041aa0e R08: 0000000000000000 R09: 00007ffce0ff84d8
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034105] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 119.034107] R13: 0000000001504120 R14: 0000000000000000 R15: 0000000000000000
/* somebody else's printk(). */
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.518735] BUG: workqueue lockup - pool cpus=2 node=0 flags=0x0 nice=0 stuck for 61s!
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.598965] Showing busy workqueues and worker pools:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.601027] workqueue mm_percpu_wq: flags=0x8
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.602688] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.605458] pending: vmstat_update
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606825]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606827] ======================================================
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606829] WARNING: possible circular locking dependency detected
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606831] 4.18.0-rc3 #643 Tainted: G O
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606832] ------------------------------------------------------
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606834] swapper/0/0 is trying to acquire lock:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606835] 00000000316e1432 (console_owner){-.-.}, at: console_unlock+0x1ce/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606840]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606841] but task is already holding lock:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606842] 000000009b45dcb4 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state+0x3b2/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606847]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606848] which lock already depends on the new lock.
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606849]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606849]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606851] the existing dependency chain (in reverse order) is:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606852]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606852] -> #4 (&(&pool->lock)->rlock){-.-.}:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606857] _raw_spin_lock+0x2a/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606858] __queue_work+0xfb/0x780
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606859] queue_work_on+0xde/0x110
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606861] put_pwq+0x85/0x90
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606862] put_pwq_unlocked.part.29+0x1d/0x30
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606863] destroy_workqueue+0x2df/0x320
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606865] tipc_topsrv_stop+0x1a0/0x290
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606866] tipc_exit_net+0x15/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606867] ops_exit_list.isra.8+0x4e/0x90
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606869] cleanup_net+0x219/0x390
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606870] process_one_work+0x2ec/0x800
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606871] worker_thread+0x45/0x4d0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606872] kthread+0x14c/0x170
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606874] ret_from_fork+0x3a/0x50
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606874]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606875] -> #3 (&pool->lock/1){-.-.}:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606880] _raw_spin_lock+0x2a/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606881] __queue_work+0xfb/0x780
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606882] queue_work_on+0xde/0x110
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606884] tty_flip_buffer_push+0x2f/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606885] pty_write+0xb0/0xc0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606886] n_tty_write+0x244/0x620
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606887] tty_write+0x1e3/0x3f0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606888] __vfs_write+0x54/0x270
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606890] vfs_write+0xf0/0x230
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606891] redirected_tty_write+0x61/0xb0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606892] do_iter_write+0x1ce/0x220
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606893] vfs_writev+0xb0/0x140
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606895] do_writev+0x6d/0x120
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606896] __x64_sys_writev+0x20/0x30
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606897] do_syscall_64+0x81/0x260
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606899] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606899]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606900] -> #2 (&(&port->lock)->rlock){-.-.}:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606905] _raw_spin_lock_irqsave+0x56/0x70
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606906] tty_port_tty_get+0x20/0x60
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606907] tty_port_default_wakeup+0x15/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606909] tty_port_tty_wakeup+0x20/0x30
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606910] uart_write_wakeup+0x26/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606911] serial8250_tx_chars+0x1ae/0x2d0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606913] serial8250_handle_irq.part.23+0x108/0x130
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606915] serial8250_default_handle_irq+0x7a/0x90
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606916] serial8250_interrupt+0x77/0x110
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606918] __handle_irq_event_percpu+0x80/0x480
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606919] handle_irq_event_percpu+0x35/0x90
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606920] handle_irq_event+0x39/0x60
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606922] handle_edge_irq+0xa5/0x230
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606923] handle_irq+0xae/0x120
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606924] do_IRQ+0x53/0x110
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606925] ret_from_intr+0x0/0x1e
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606927] free_unref_page_list+0x333/0x590
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606928] release_pages+0x44a/0x5c0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606930] free_pages_and_swap_cache+0x107/0x150
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606931] tlb_flush_mmu_free+0x4e/0x70
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606932] arch_tlb_finish_mmu+0x61/0xa0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606934] tlb_finish_mmu+0x36/0x50
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606935] exit_mmap+0x10a/0x230
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606936] mmput+0x92/0x1c0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606937] do_exit+0x365/0x10b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606938] do_group_exit+0x60/0x100
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606940] __x64_sys_exit_group+0x1c/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606941] do_syscall_64+0x81/0x260
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606943] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606943]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606944] -> #1 (&port_lock_key){-.-.}:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606948] _raw_spin_lock_irqsave+0x56/0x70
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606950] serial8250_console_write+0x30c/0x3b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606951] univ8250_console_write+0x35/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606953] console_unlock+0x650/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606954] register_console+0x339/0x540
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606955] univ8250_console_init+0x30/0x3a
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606957] console_init+0x2c6/0x424
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606958] start_kernel+0x466/0x61f
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606959] x86_64_start_reservations+0x2a/0x2c
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606961] x86_64_start_kernel+0x72/0x75
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606962] secondary_startup_64+0xa5/0xb0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606963]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606963] -> #0 (console_owner){-.-.}:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606968] lock_acquire+0xbf/0x220
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606969] console_unlock+0x23e/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606970] vprintk_emit+0x394/0x680
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606971] vprintk_default+0x28/0x30
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606973] vprintk_func+0x57/0xc0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606974] printk+0x5e/0x7d
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606975] show_workqueue_state+0x448/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606977] wq_watchdog_timer_fn+0x1df/0x320
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606978] call_timer_fn+0xb0/0x420
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606979] run_timer_softirq+0x2da/0x870
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606981] __do_softirq+0xd0/0x4fe
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606982] irq_exit+0xd4/0xe0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606983] smp_apic_timer_interrupt+0x9a/0x2b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606985] apic_timer_interrupt+0xf/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606986] native_safe_halt+0x6/0x10
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606987] default_idle+0x2e/0x1a0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606988] arch_cpu_idle+0x10/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606990] default_idle_call+0x1e/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606991] do_idle+0x205/0x2e0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606992] cpu_startup_entry+0x6e/0x70
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606993] rest_init+0xc7/0xd0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606995] start_kernel+0x5fc/0x61f
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606996] x86_64_start_reservations+0x2a/0x2c
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606997] x86_64_start_kernel+0x72/0x75
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606999] secondary_startup_64+0xa5/0xb0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.606999]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607001] other info that might help us debug this:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607001]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607002] Chain exists of:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607003] console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607009]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607010] Possible unsafe locking scenario:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607011]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607012] CPU0 CPU1
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607014] ---- ----
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607014] lock(&(&pool->lock)->rlock);
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607017] lock(&pool->lock/1);
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607021] lock(&(&pool->lock)->rlock);
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607024] lock(console_owner);
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607026]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607027] *** DEADLOCK ***
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607027]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607028] 4 locks held by swapper/0/0:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607029] #0: 00000000acb744a0 ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0x0/0x420
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607034] #1: 00000000918be876 (rcu_read_lock_sched){....}, at: show_workqueue_state+0x0/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607040] #2: 000000009b45dcb4 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state+0x3b2/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607045] #3: 000000005fd39989 (console_lock){+.+.}, at: vprintk_emit+0x386/0x680
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607050]
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607051] stack backtrace:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607053] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G O 4.18.0-rc3 #643
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607056] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607057] Call Trace:
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607058] <IRQ>
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607059] dump_stack+0xe9/0x148
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607061] print_circular_bug.isra.39+0x1f3/0x201
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607062] __lock_acquire+0x135f/0x1400
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607063] lock_acquire+0xbf/0x220
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607065] ? console_unlock+0x215/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607066] ? lock_acquire+0xbf/0x220
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607067] ? console_unlock+0x1ce/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607068] console_unlock+0x23e/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607070] ? console_unlock+0x1ce/0x8b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607071] vprintk_emit+0x394/0x680
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607072] vprintk_default+0x28/0x30
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607073] vprintk_func+0x57/0xc0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607074] printk+0x5e/0x7d
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607076] show_workqueue_state+0x448/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607077] ? wq_watchdog_timer_fn+0x1a6/0x320
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607078] wq_watchdog_timer_fn+0x1df/0x320
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607080] ? show_workqueue_state+0x900/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607081] ? show_workqueue_state+0x900/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607082] call_timer_fn+0xb0/0x420
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607084] ? show_workqueue_state+0x900/0x900
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607085] run_timer_softirq+0x2da/0x870
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607086] __do_softirq+0xd0/0x4fe
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607087] irq_exit+0xd4/0xe0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607089] smp_apic_timer_interrupt+0x9a/0x2b0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607090] apic_timer_interrupt+0xf/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607091] </IRQ>
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607092] RIP: 0010:native_safe_halt+0x6/0x10
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607093] Code: 53 18 fe 65 48 8b 04 25 40 5e 01 00 f0 80 48 02 20 48 8b 00 a8 08 0f 84 57 ff ff ff eb a6 90 90 90 90 90 90 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 90 90 90 90 90
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607135] RSP: 0018:ffffffff84003df8 EFLAGS: 00000282 ORIG_RAX: ffffffffffffff13
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607138] RAX: ffffffff8406d7c0 RBX: 0000000000000000 RCX: 0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607140] RDX: ffffffff8406d7c0 RSI: 0000000000000001 RDI: ffffffff8406d7c0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607142] RBP: ffffffff84003df8 R08: 0000000000000001 R09: 0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607144] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607146] R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff84098920
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607148] ? trace_hardirqs_on+0xd/0x10
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607149] default_idle+0x2e/0x1a0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607150] arch_cpu_idle+0x10/0x20
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607151] default_idle_call+0x1e/0x40
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607152] do_idle+0x205/0x2e0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607154] ? do_idle+0xf9/0x2e0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607155] cpu_startup_entry+0x6e/0x70
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607156] rest_init+0xc7/0xd0
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607158] ? trace_event_define_fields_vector_alloc_managed+0x8b/0x8b
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607159] start_kernel+0x5fc/0x61f
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607161] x86_64_start_reservations+0x2a/0x2c
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607162] x86_64_start_kernel+0x72/0x75
2018-07-02 12:13:13 192.168.159.129:6666 [ 151.607163] secondary_startup_64+0xa5/0xb0
2018-07-02 12:13:44 192.168.159.129:6666 [ 182.238928] BUG: workqueue lockup - pool cpus=2 node=0 flags=0x0 nice=0 stuck for 92s!
2018-07-02 12:13:44 192.168.159.129:6666 [ 182.241803] Showing busy workqueues and worker pools:
2018-07-02 12:13:44 192.168.159.129:6666 [ 182.243616] workqueue mm_percpu_wq: flags=0x8
2018-07-02 12:13:44 192.168.159.129:6666 [ 182.245141] pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=1/256
2018-07-02 12:13:44 192.168.159.129:6666 [ 182.247130] pending: vmstat_update
2018-07-02 12:14:11 192.168.159.129:6666 [ 209.030165] INFO: rcu_sched detected stalls on CPUs/tasks:
2018-07-02 12:14:11 192.168.159.129:6666 [ 209.032175] 2-...0: (0 ticks this GP) idle=0be/1/4611686018427387904 softirq=5544/5544 fqs=27833
2018-07-02 12:14:11 192.168.159.129:6666 [ 209.035069] (detected by 3, t=120007 jiffies, g=2553, c=2552, q=1155)
2018-07-02 12:14:11 192.168.159.129:6666 [ 209.037220] Sending NMI from CPU 3 to CPUs 2:
/* Here is a 10 seconds of lag which is flushed by local_irq_enable(). */
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038805] NMI backtrace for cpu 2
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038813] CPU: 2 PID: 10038 Comm: insmod Tainted: G O 4.18.0-rc3 #643
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038815] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038827] RIP: 0010:__sanitizer_cov_trace_pc+0x26/0x50
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038828] Code: 90 90 90 90 55 65 48 8b 04 25 40 5e 01 00 48 89 e5 65 8b 15 6c f6 d1 7e 81 e2 00 01 1f 00 48 8b 4d 08 75 2b 8b 90 80 12 00 00 <83> fa 02 75 20 48 8b b0 88 12 00 00 8b 80 84 12 00 00 48 8b 16 48
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038881] RSP: 0018:ffffc9000205fc60 EFLAGS: 00000046
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038884] RAX: ffff880136312080 RBX: 00000000fffcc7e1 RCX: ffffffffa000d038
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038886] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffa000d033
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038888] RBP: ffffc9000205fc60 R08: 0000000000000001 R09: 0000000000000000
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038890] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa000d000
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038892] R13: ffff8801351cd940 R14: 0000000000000001 R15: ffffc9000205fe78
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038895] FS: 00007f90d3e62740(0000) GS:ffff88013a680000(0000) knlGS:0000000000000000
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038899] CR2: 00000000015051d8 CR3: 0000000136a66004 CR4: 00000000000606e0
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038941] Call Trace:
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038950] ccs_init+0x38/0x79 [akari_test]
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038956] do_one_initcall+0x7a/0x460
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038961] ? trace_hardirqs_on+0xd/0x10
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038966] do_init_module+0x5a/0x1e0
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038969] load_module+0x2144/0x2800
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038975] __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038978] ? __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038983] __x64_sys_finit_module+0x15/0x20
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038986] do_syscall_64+0x81/0x260
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038992] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038995] RIP: 0033:0x7f90d3335ec9
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.038996] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 cf 2c 00 f7 d8 64 89 01 48
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039041] RSP: 002b:00007ffce0ff8338 EFLAGS: 00000202 ORIG_RAX: 0000000000000139
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039045] RAX: ffffffffffffffda RBX: 00000000015051e0 RCX: 00007f90d3335ec9
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039047] RDX: 0000000000000000 RSI: 000000000041aa0e RDI: 0000000000000003
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039049] RBP: 000000000041aa0e R08: 0000000000000000 R09: 00007ffce0ff84d8
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039051] R10: 0000000000000003 R11: 0000000000000202 R12: 0000000000000000
2018-07-02 12:14:21 192.168.159.129:6666 [ 209.039053] R13: 0000000001504120 R14: 0000000000000000 R15: 0000000000000000
----------
Loading the module after
echo 180 > /sys/module/workqueue/parameters/watchdog_thresh
echo 25 > /sys/module/rcupdate/parameters/rcu_cpu_stall_timeout
----------
2018-07-02 12:15:57 192.168.159.129:6666 [ 315.368812] INFO: rcu_sched detected stalls on CPUs/tasks:
2018-07-02 12:15:57 192.168.159.129:6666 [ 315.370959] 1-...0: (2 ticks this GP) idle=d0a/1/4611686018427387904 softirq=4587/4587 fqs=7128
2018-07-02 12:15:57 192.168.159.129:6666 [ 315.373828] (detected by 7, t=30007 jiffies, g=2649, c=2648, q=64)
2018-07-02 12:15:57 192.168.159.129:6666 [ 315.375881] Sending NMI from CPU 7 to CPUs 1:
/* Here is a 90 seconds of lag which is flushed by */
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377376] NMI backtrace for cpu 1
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377382] CPU: 1 PID: 10062 Comm: insmod Tainted: G O 4.18.0-rc3 #643
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377384] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377394] RIP: 0010:__sanitizer_cov_trace_pc+0x4b/0x50
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377395] Code: 90 80 12 00 00 83 fa 02 75 20 48 8b b0 88 12 00 00 8b 80 84 12 00 00 48 8b 16 48 83 c2 01 48 39 d0 76 07 48 89 0c d6 48 89 16 <5d> c3 0f 1f 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 89 f3 48
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377451] RSP: 0018:ffffc90000cffc60 EFLAGS: 00000093
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377454] RAX: ffff88013091a600 RBX: 00000000ffffc6dd RCX: ffffffffa0012038
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377456] RDX: 0000000000000000 RSI: 0000000000000bfa RDI: ffffffffa0012033
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377458] RBP: ffffc90000cffc60 R08: 0000000000000000 R09: 0000000000000000
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377460] R10: ffffc90000cffbe8 R11: ffff88013091a600 R12: ffffffffa0012000
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377462] R13: ffff88012f7734e0 R14: 0000000000000001 R15: ffffc90000cffe78
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377466] FS: 00007fafb5b7b740(0000) GS:ffff88013a640000(0000) knlGS:0000000000000000
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377468] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377470] CR2: 00007fafb50bf910 CR3: 0000000130321006 CR4: 00000000000606e0
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377507] Call Trace:
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377515] ccs_init+0x38/0x79 [akari_test]
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377521] do_one_initcall+0x7a/0x460
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377526] ? trace_hardirqs_on+0xd/0x10
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377531] do_init_module+0x5a/0x1e0
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377534] load_module+0x2144/0x2800
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377540] __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377543] ? __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377548] __x64_sys_finit_module+0x15/0x20
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377552] do_syscall_64+0x81/0x260
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377557] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377560] RIP: 0033:0x7fafb504eec9
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377561] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 cf 2c 00 f7 d8 64 89 01 48
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377606] RSP: 002b:00007ffcd3167148 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377610] RAX: ffffffffffffffda RBX: 00000000012ab1e0 RCX: 00007fafb504eec9
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377612] RDX: 0000000000000000 RSI: 000000000041aa0e RDI: 0000000000000003
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377614] RBP: 000000000041aa0e R08: 0000000000000000 R09: 00007ffcd31672e8
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377616] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000000
2018-07-02 12:17:27 192.168.159.129:6666 [ 315.377618] R13: 00000000012aa120 R14: 0000000000000000 R15: 0000000000000000
/* somebody else's printk(). */
2018-07-02 12:17:27 192.168.159.129:6666 [ 405.373020] INFO: rcu_sched detected stalls on CPUs/tasks:
2018-07-02 12:17:27 192.168.159.129:6666 [ 405.449297] 1-...0: (2 ticks this GP) idle=d0a/1/4611686018427387904 softirq=4587/4587 fqs=27972
2018-07-02 12:17:27 192.168.159.129:6666 [ 405.452234] (detected by 2, t=120086 jiffies, g=2649, c=2648, q=537)
2018-07-02 12:17:27 192.168.159.129:6666 [ 405.454419] Sending NMI from CPU 2 to CPUs 1:
/* Here is a 10 seconds of lag which is flushed by local_irq_enable(). */
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.455947] NMI backtrace for cpu 1
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.455953] CPU: 1 PID: 10062 Comm: insmod Tainted: G O 4.18.0-rc3 #643
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.455956] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.455964] RIP: 0010:__sanitizer_cov_trace_pc+0x20/0x50
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.455965] Code: 5d c3 90 90 90 90 90 90 90 90 55 65 48 8b 04 25 40 5e 01 00 48 89 e5 65 8b 15 6c f6 d1 7e 81 e2 00 01 1f 00 48 8b 4d 08 75 2b <8b> 90 80 12 00 00 83 fa 02 75 20 48 8b b0 88 12 00 00 8b 80 84 12
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456012] RSP: 0018:ffffc90000cffc60 EFLAGS: 00000046
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456014] RAX: ffff88013091a600 RBX: 00000000ffffc6dd RCX: ffffffffa0012038
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456017] RDX: 0000000000000000 RSI: 0000000000000bfa RDI: ffffffffa0012033
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456018] RBP: ffffc90000cffc60 R08: 0000000000000000 R09: 0000000000000000
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456021] R10: ffffc90000cffbe8 R11: ffff88013091a600 R12: ffffffffa0012000
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456023] R13: ffff88012f7734e0 R14: 0000000000000001 R15: ffffc90000cffe78
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456026] FS: 00007fafb5b7b740(0000) GS:ffff88013a640000(0000) knlGS:0000000000000000
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456028] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456030] CR2: 00007fafb50bf910 CR3: 0000000130321006 CR4: 00000000000606e0
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456069] Call Trace:
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456077] ccs_init+0x38/0x79 [akari_test]
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456083] do_one_initcall+0x7a/0x460
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456088] ? trace_hardirqs_on+0xd/0x10
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456093] do_init_module+0x5a/0x1e0
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456096] load_module+0x2144/0x2800
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456102] __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456105] ? __do_sys_finit_module+0xd1/0xf0
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456110] __x64_sys_finit_module+0x15/0x20
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456113] do_syscall_64+0x81/0x260
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456119] entry_SYSCALL_64_after_hwframe+0x49/0xbe
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456122] RIP: 0033:0x7fafb504eec9
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456123] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 97 cf 2c 00 f7 d8 64 89 01 48
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456168] RSP: 002b:00007ffcd3167148 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456172] RAX: ffffffffffffffda RBX: 00000000012ab1e0 RCX: 00007fafb504eec9
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456174] RDX: 0000000000000000 RSI: 000000000041aa0e RDI: 0000000000000003
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456176] RBP: 000000000041aa0e R08: 0000000000000000 R09: 00007ffcd31672e8
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456177] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000000
2018-07-02 12:17:37 192.168.159.129:6666 [ 405.456179] R13: 00000000012aa120 R14: 0000000000000000 R15: 0000000000000000
----------