"rcu_sched self-detected stall on CPU" on highload dns servers
From: ya dmesg
Date: Tue Apr 16 2013 - 14:03:19 EST
Hi everyone.
We areexperiencing 100% reproducable kernel lockups (4 servers crashes 2-3 times a day each) on a highload DNS servers (30-40kpps udp requests).
Hardware is pretty different, I can provide lspci if needed. We are using Intel ethernet adapters with Jumbo Frames enabled and we are using no swap at all.
e1000e driver:
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
08:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
igb driver:
01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
All hardware affected.
My system is Ubuntu-12.04 Server LTS. But this problem occures on many kernel versions:
linux-3.2 ubuntu lts
linux-3.5 quantal ubuntu lts
linux-3.4.* kernel.org LTS
linux-3.8.* kernel.org LTS
I've attached crash I've catched with ipmitool. If there is anything I can do to help to debug this problem, please, let me know!
Thanks.
[110804.419268] INFO: rcu_sched self-detected stall on CPU
[110804.424588] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[110804.430547] (t=15000 jiffies)
[110804.434541] INFO: rcu_sched detected stalls on CPUs/tasks:
[110804.434544] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[110804.434545] (detected by 1, t=15003 jiffies)
[110804.434547] INFO: Stall ended before state dump start
[110805.078303] INFO: rcu_bh detected stalls on CPUs/tasks:
[110805.083710] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[110805.089671] (detected by 6, t=15005 jiffies)
[110805.094207] INFO: Stall ended before state dump start
[110950.191379] INFO: task md1_raid10:361 blocked for more than 120 seconds.
[110950.198225] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110950.206394] INFO: task jbd2/md1-8:377 blocked for more than 120 seconds.
[110950.213220] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110950.221457] INFO: task debian-sa1:23250 blocked for more than 120 seconds.
[110950.228495] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110950.236650] INFO: task date:23252 blocked for more than 120 seconds.
[110950.243135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110950.251230] INFO: task ifconfig:23253 blocked for more than 120 seconds.
[110950.258152] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110950.266291] INFO: task job_converter.p:23257 blocked for more than 120 seconds.
[110950.273778] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[110984.167042] INFO: rcu_sched self-detected stall on CPU
[110984.172386] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[110984.178372] (t=60006 jiffies)
[110984.182481] INFO: rcu_sched detected stalls on CPUs/tasks:
[110984.182489] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[110984.182491] (detected by 6, t=60009 jiffies)
[110984.182495] INFO: Stall ended before state dump start
[110984.822148] INFO: rcu_bh detected stalls on CPUs/tasks:
[110984.827589] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[110984.833576] (detected by 7, t=60010 jiffies)
[110984.838134] INFO: Stall ended before state dump start
[111070.095039] INFO: task md1_raid10:361 blocked for more than 120 seconds.
[111070.101925] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[111070.110252] INFO: task jbd2/md1-8:377 blocked for more than 120 seconds.
[111070.117156] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[111070.125637] INFO: task rs:main Q:Reg:1052 blocked for more than 120 seconds.
[111070.132937] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[111070.141070] INFO: task flush-9:1:4108 blocked for more than 120 seconds.
[111070.147952] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[111163.914422] INFO: rcu_sched self-detected stall on CPU
[111163.919767] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111163.925753] (t=105012 jiffies)
[111163.929853] INFO: rcu_sched detected stalls on CPUs/tasks: linear
[111163.936177] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111163.942160] (detected by 1, t=105019 jiffies)
[111163.946806] INFO: Stall ended before state dump start
[111164.565547] INFO: rcu_bh detected stalls on CPUs/tasks:
[111164.570981] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111164.576968] (detected by 6, t=105015 jiffies)
[111164.581615] INFO: Stall ended before state dump start
[111343.662005] INFO: rcu_sched self-detected stall on CPU
[111343.667349] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111343.673334] (t=150018 jiffies)
[111343.677418] INFO: rcu_sched detected stalls on CPUs/tasks: async_tx raid1 raid0 multipath linear
[111343.686460] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111343.692442] (detected by 1, t=150025 jiffies)
[111343.697089] INFO: Stall ended before state dump start
[111344.309009] INFO: rcu_bh detected stalls on CPUs/tasks:
[111344.314444] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111344.320431] (detected by 2, t=150020 jiffies)
[111344.325077] INFO: Stall ended before state dump start
[111523.409590] INFO: rcu_sched self-detected stall on CPU
[111523.414938] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111523.420924] (t=195024 jiffies)
[111523.425297] INFO: rcu_sched detected stalls on CPUs/tasks:
[111523.430966] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111523.436950] (detected by 2, t=195031 jiffies)
[111523.441598] INFO: Stall ended before state dump start
[111524.052678] INFO: rcu_bh detected stalls on CPUs/tasks:
[111524.058114] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111524.064101] (detected by 6, t=195025 jiffies)
[111524.068746] INFO: Stall ended before state dump start
[111703.157255] INFO: rcu_sched self-detected stall on CPU
[111703.162597] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111703.168583] (t=240030 jiffies)
[111703.172947] INFO: rcu_sched detected stalls on CPUs/tasks:
[111703.172950] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111703.172952] (detected by 2, t=240033 jiffies)
[111703.172954] INFO: Stall ended before state dump start
[111703.796252] INFO: rcu_bh detected stalls on CPUs/tasks:
[111703.801687] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111703.807674] (detected by 2, t=240030 jiffies)
[111703.812321] INFO: Stall ended before state dump start
[111882.904766] INFO: rcu_sched self-detected stall on CPU
[111882.910107] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[111882.916086] (t=285036 jiffies)
[111882.920173] INFO: rcu_sched detected stalls on CPUs/tasks: multipath linear
[111882.927367] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111882.933351] (detected by 1, t=285043 jiffies)
[111882.937996] INFO: Stall ended before state dump start
[111883.539885] INFO: rcu_bh detected stalls on CPUs/tasks:
[111883.545321] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[111883.551310] (detected by 6, t=285035 jiffies)
[111883.555955] INFO: Stall ended before state dump start
[112062.652392] INFO: rcu_sched self-detected stall on CPU
[112062.657742] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112062.663727] (t=330042 jiffies)
[112062.667856] INFO: rcu_sched detected stalls on CPUs/tasks:
[112062.667859] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112062.667861] (detected by 1, t=330045 jiffies)
[112062.667862] INFO: Stall ended before state dump start
[112063.283472] INFO: rcu_bh detected stalls on CPUs/tasks:
[112063.288906] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112063.294894] (detected by 6, t=330040 jiffies)
[112063.299538] INFO: Stall ended before state dump start
[112242.399956] INFO: rcu_sched self-detected stall on CPU
[112242.405305] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112242.411290] (t=375048 jiffies)
[112242.415393] INFO: rcu_sched detected stalls on CPUs/tasks: multipath linear
[112242.422593] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112242.428574] (detected by 1, t=375055 jiffies)
[112242.433220] INFO: Stall ended before state dump start
[112243.027095] INFO: rcu_bh detected stalls on CPUs/tasks:
[112243.032533] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112243.038519] (detected by 7, t=375045 jiffies)
[112243.043165] INFO: Stall ended before state dump start
[112422.147588] INFO: rcu_sched self-detected stall on CPU
[112422.152930] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112422.158915] (t=420054 jiffies)
[112422.163011] INFO: rcu_sched detected stalls on CPUs/tasks: raid0 multipath linear
[112422.170758] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112422.176742] (detected by 1, t=420061 jiffies)
[112422.181389] INFO: Stall ended before state dump start
[112422.770722] INFO: rcu_bh detected stalls on CPUs/tasks:
[112422.776158] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112422.782146] (detected by 6, t=420050 jiffies)
[112422.786792] INFO: Stall ended before state dump start
[112601.895159] INFO: rcu_sched self-detected stall on CPU
[112601.900502] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112601.906488] (t=465060 jiffies)
[112601.910589] INFO: rcu_sched detected stalls on CPUs/tasks:
[112601.910592] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112601.910594] (detected by 1, t=465063 jiffies)
[112601.910595] INFO: Stall ended before state dump start
[112602.514229] INFO: rcu_bh detected stalls on CPUs/tasks:
[112602.519663] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112602.525650] (detected by 6, t=465055 jiffies)
[112602.530298] INFO: Stall ended before state dump start
[112781.642947] INFO: rcu_sched self-detected stall on CPU
[112781.648290] 0: (1 GPs behind) idle=ccf/2/0 drain=0 . timer=-1
[112781.654278] (t=510066 jiffies)
[112781.658643] INFO: rcu_sched detected stalls on CPUs/tasks: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112781.687911] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112781.693894] (detected by 2, t=510079 jiffies)
[112781.698539] INFO: Stall ended before state dump start
[112782.257865] INFO: rcu_bh detected stalls on CPUs/tasks:
[112782.263300] 0: (1 GPs behind) idle=ccf/1/0 drain=0 . timer=-1
[112782.269287] (detected by 1, t=510060 jiffies)
[112782.273933] INFO: Stall ended before state dump start
[112899.146651] SysRq : Show backtrace of all active CPUs
[112912.006981] SysRq : Trigger a crash
[112912.010673] BUG: unable to handle kernel NULL pointer dereference at (null)
[112912.018789] IP: [<ffffffff8132c3f1>] sysrq_handle_crash+0x11/0x20
[112912.025087] PGD 232ebc067 PUD 232083067 PMD 0
[112912.029840] Oops: 0002 [#1] SMP
[112912.033318] CPU 0
[112912.035281] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112912.063864]
[112912.065508] Pid: 0, comm: swapper/0 Not tainted 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112912.075755] RIP: 0010:[<ffffffff8132c3f1>] [<ffffffff8132c3f1>] sysrq_handle_crash+0x11/0x20
[112912.084512] RSP: 0018:ffff880237c03360 EFLAGS: 00010086
[112912.089972] RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000002928
[112912.097284] RDX: 0000000000000000 RSI: 0000000000000086 RDI: 0000000000000063
[112912.104597] RBP: ffff880237c03360 R08: 0000000000000000 R09: 0000000000000080
[112912.111909] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff818411c0
[112912.119219] R13: 0000000000000086 R14: 0000000000000004 R15: 0000000000000001
[112912.126533] FS: 0000000000000000(0000) GS:ffff880237c00000(0000) knlGS:0000000000000000
[112912.134797] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112912.140689] CR2: 0000000000000000 CR3: 0000000233bcb000 CR4: 00000000000007f0
[112912.148928] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112912.156241] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112912.163551] Process swapper/0 (pid: 0, threadinfo ffffffff81800000, task ffffffff81813020)
[112912.171988] Stack:
[112912.174153] ffff880237c033a0 ffffffff8132ca99 ffffea0008671700 ffff880231cb59a0
[112912.181931] 0000000000000001 0000000000000001 0000000000000001 000000000000002e
[112912.189721] ffff880237c033e0 ffffffff8132cbe6 ffff880200000001 ffffffff0000002e
[112912.197499] Call Trace:
[112912.200096] <IRQ>
[112912.202398] [<ffffffff8132ca99>] __handle_sysrq+0x119/0x180
[112912.208204] [<ffffffff8132cbe6>] sysrq_filter+0x96/0x1a0
[112912.213752] [<ffffffff81402c76>] input_pass_event+0x76/0xd0
[112912.219568] [<ffffffff8140433f>] input_handle_event+0x8f/0x4a0
[112912.225642] [<ffffffff81404884>] input_event+0x84/0xa0
[112912.231017] [<ffffffffa00dc39d>] hidinput_hid_event+0x27d/0x3e0 [hid]
[112912.237697] [<ffffffffa00d7292>] hid_process_event+0x112/0x140 [hid]
[112912.244291] [<ffffffffa00d7a9d>] hid_report_raw_event+0x36d/0x3e0 [hid]
[112912.251143] [<ffffffffa00d7cf6>] hid_input_report+0x1e6/0x2b0 [hid]
[112912.257651] [<ffffffffa0113105>] hid_irq_in+0x1a5/0x1f0 [usbhid]
[112912.263899] [<ffffffff813cdd8f>] usb_hcd_giveback_urb+0x5f/0xe0
[112912.270059] [<ffffffff813e1968>] ehci_urb_done+0xd8/0x120
[112912.275694] [<ffffffff813e1d29>] qh_completions+0x379/0x460
[112912.281507] [<ffffffff813e2bde>] ehci_work+0x35e/0xbb0
[112912.286881] [<ffffffff813e3c52>] ehci_irq+0x202/0x260
[112912.292169] [<ffffffff810682b4>] ? scheduler_tick+0xf4/0x130
[112912.298070] [<ffffffff813cd397>] usb_hcd_irq+0x37/0x60
[112912.303445] [<ffffffff810a7bd3>] handle_irq_event_percpu+0x43/0x160
[112912.309951] [<ffffffff810a7d33>] handle_irq_event+0x43/0x70
[112912.315758] [<ffffffff810aac55>] handle_fasteoi_irq+0x55/0xf0
[112912.321737] [<ffffffff8100c19d>] handle_irq+0x1d/0x30
[112912.327025] [<ffffffff8100be05>] do_IRQ+0x55/0xd0
[112912.331967] [<ffffffff815632aa>] common_interrupt+0x6a/0x6a
[112912.337781] [<ffffffff814bdbc4>] ? udp4_lib_lookup2.isra.34+0x54/0x250
[112912.344548] [<ffffffff814bdfc7>] __udp4_lib_lookup+0x207/0x310
[112912.350622] [<ffffffff814bed24>] __udp4_lib_rcv+0x1b4/0x6c0
[112912.356431] [<ffffffff81491426>] ? ip_route_input_common+0xcd6/0xe70
[112912.363022] [<ffffffff814bf245>] udp_rcv+0x15/0x20
[112912.368050] [<ffffffff814935b5>] ip_local_deliver_finish+0xd5/0x270
[112912.374557] [<ffffffff814938d5>] ip_local_deliver+0x45/0x80
[112912.380364] [<ffffffff81493285>] ip_rcv_finish+0x105/0x360
[112912.386085] [<ffffffff81493b01>] ip_rcv+0x1f1/0x2f0
[112912.391199] [<ffffffff81102505>] ? init_object+0x45/0x80
[112912.396746] [<ffffffff81463326>] __netif_receive_skb+0x506/0x5c0
[112912.402993] [<ffffffff814633fe>] netif_receive_skb+0x1e/0x80
[112912.408886] [<ffffffff81463831>] ? dev_gro_receive+0x1b1/0x2b0
[112912.414961] [<ffffffff81463588>] napi_skb_finish+0x48/0x60
[112912.420680] [<ffffffff81464735>] napi_gro_receive+0xf5/0x140
[112912.426575] [<ffffffffa0059f16>] e1000_receive_skb+0x56/0x70 [e1000e]
[112912.433255] [<ffffffffa005d629>] e1000_clean_rx_irq_ps+0x439/0x720 [e1000e]
[112912.440455] [<ffffffffa005b32d>] e1000_clean+0x6d/0x2b0 [e1000e]
[112912.446703] [<ffffffff81463c89>] net_rx_action+0x139/0x1d0
[112912.452430] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112912.457976] [<ffffffff81562f39>] ? _raw_spin_lock+0x9/0x10
[112912.463699] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112912.469157] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112912.474444] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112912.479558] [<ffffffff8100be0e>] do_IRQ+0x5e/0xd0
[112912.484501] [<ffffffff815632aa>] common_interrupt+0x6a/0x6a
[112912.490313] <EOI>
[112912.492616] [<ffffffff812d8165>] ? intel_idle+0xe5/0x150
[112912.498164] [<ffffffff812d8147>] ? intel_idle+0xc7/0x150
[112912.503711] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112912.509256] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112912.515237] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112912.520438] [<ffffffff8153a179>] rest_init+0x6d/0x74
[112912.525638] [<ffffffff818a0c18>] start_kernel+0x3c0/0x3cd
[112912.531271] [<ffffffff818a0658>] ? repair_env_string+0x5a/0x5a
[112912.537338] [<ffffffff818a0346>] x86_64_start_reservations+0x131/0x135
[112912.544105] [<ffffffff818a0140>] ? early_idt_handlers+0x140/0x140
[112912.550438] [<ffffffff818a044c>] x86_64_start_kernel+0x102/0x111
[112912.556687] Code: 8a 45 01 f4 45 39 65 54 75 e5 4c 89 ef e8 28 f7 ff ff eb db 90 90 90 90 90 90 55 c7 05 c1 25 63 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 48 89 e5
[112912.579063] RIP [<ffffffff8132c3f1>] sysrq_handle_crash+0x11/0x20
[112912.585450] RSP <ffff880237c03360>
[112912.589085] CR2: 0000000000000000
[112912.592555] ---[ end trace 6552bded36af03a9 ]---
[112912.597322] Kernel panic - not syncing: Fatal exception in interrupt
[112912.603911] ------------[ cut here ]------------
[112912.608765] WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0x25f/0x270()
[112912.616268] Hardware name: System Product Name
[112912.620939] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out
[112912.627723] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112912.656807] Pid: 7, comm: watchdog/0 Tainted: G D 3.4.39-21 #8
[112912.663761] Call Trace:
[112912.666469] [<ffffffff8103f8da>] warn_slowpath_common+0x7a/0xb0
[112912.672726] [<ffffffff8103f9b1>] warn_slowpath_fmt+0x41/0x50
[112912.678700] [<ffffffff8128aaad>] ? do_raw_spin_unlock+0x5d/0xb0
[112912.684971] [<ffffffff8147e6cf>] dev_watchdog+0x25f/0x270
[112912.690715] [<ffffffff8104b56f>] run_timer_softirq+0xff/0x230
[112912.696798] [<ffffffff81011819>] ? read_tsc+0x9/0x20
[112912.702106] [<ffffffff8147e470>] ? pfifo_fast_init+0x90/0x90
[112912.708106] [<ffffffff810276e8>] ? lapic_next_event+0x18/0x20
[112912.714198] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112912.719861] [<ffffffff8108171f>] ? tick_program_event+0x1f/0x30
[112912.726119] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112912.731684] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112912.737070] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112912.742294] [<ffffffff81027b79>] smp_apic_timer_interrupt+0x69/0xa0
[112912.748908] [<ffffffff8156454a>] apic_timer_interrupt+0x6a/0x70
[112912.755171] [<ffffffff81562eed>] ? _raw_spin_unlock_irq+0xd/0x20
[112912.761522] [<ffffffff810647a7>] finish_task_switch+0x57/0xe0
[112912.767615] [<ffffffff8100d9fb>] ? register_nmi_handler+0x4b/0x180
[112912.774140] [<ffffffff81561cbb>] __schedule+0x3ab/0x720
[112912.779711] [<ffffffff810a6e60>] ? watchdog_enable+0x240/0x240
[112912.785891] [<ffffffff815622f4>] schedule+0x24/0x70
[112912.791112] [<ffffffff810a6eda>] watchdog+0x7a/0xd0
[112912.796333] [<ffffffff8105bd2e>] kthread+0x8e/0xa0
[112912.801467] [<ffffffff81564da4>] kernel_thread_helper+0x4/0x10
[112912.807644] [<ffffffff8105bca0>] ? flush_kthread_work+0x150/0x150
[112912.814086] [<ffffffff81564da0>] ? gs_change+0x13/0x13
[112912.819567] ---[ end trace 6552bded36af03aa ]---
[112912.824447] Clocksource tsc unstable (delta = 1000253115943 ns)
[112935.209819] ------------[ cut here ]------------
[112935.214592] WARNING: at kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0()
[112935.222508] Hardware name: System Product Name
[112935.227104] Watchdog detected hard LOCKUP on cpu 0
[112935.231844] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112935.260436] Pid: 7, comm: watchdog/0 Tainted: G D W 3.4.39-21 #8
[112935.267291] Call Trace:
[112935.269886] <NMI> [<ffffffff8103f8da>] warn_slowpath_common+0x7a/0xb0
[112935.276706] [<ffffffff8103f9b1>] warn_slowpath_fmt+0x41/0x50
[112935.282607] [<ffffffff810a70ca>] watchdog_overflow_callback+0x9a/0xc0
[112935.289289] [<ffffffff810ba2bc>] __perf_event_overflow+0x9c/0x220
[112935.295622] [<ffffffff8101a38a>] ? x86_perf_event_set_period+0xda/0x150
[112935.302476] [<ffffffff810babb4>] perf_event_overflow+0x14/0x20
[112935.308551] [<ffffffff8101f3fc>] intel_pmu_handle_irq+0x17c/0x2f0
[112935.314885] [<ffffffff81305ec0>] ? ghes_read_estatus+0x90/0x180
[112935.321047] [<ffffffff8128aaad>] ? do_raw_spin_unlock+0x5d/0xb0
[112935.327208] [<ffffffff81019141>] perf_event_nmi_handler+0x21/0x30
[112935.333542] [<ffffffff8100dc4a>] do_nmi+0x11a/0x3a0
[112935.338656] [<ffffffff815638bc>] end_repeat_nmi+0x1a/0x1e
[112935.344289] [<ffffffff81012496>] ? native_read_tsc+0x6/0x20
[112935.350096] [<ffffffff81012496>] ? native_read_tsc+0x6/0x20
[112935.355902] [<ffffffff81012496>] ? native_read_tsc+0x6/0x20
[112935.361707] <<EOE>> [<ffffffff8128362a>] delay_tsc+0x4a/0x80
[112935.367748] [<ffffffff8128354a>] __delay+0xa/0x10
[112935.372690] [<ffffffff8128a9bc>] do_raw_spin_lock+0xcc/0x120
[112935.378592] [<ffffffff815630ef>] _raw_spin_lock_irqsave+0x1f/0x30
[112935.384926] [<ffffffff81404126>] input_repeat_key+0x26/0xd0
[112935.390742] [<ffffffff8104b56f>] run_timer_softirq+0xff/0x230
[112935.396730] [<ffffffff81011819>] ? read_tsc+0x9/0x20
[112935.401930] [<ffffffff81404100>] ? input_dev_resume+0x20/0x20
[112935.407918] [<ffffffff810276e8>] ? lapic_next_event+0x18/0x20
[112935.413907] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112935.419452] [<ffffffff8108171f>] ? tick_program_event+0x1f/0x30
[112935.425616] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112935.431074] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112935.436361] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112935.441477] [<ffffffff81027b79>] smp_apic_timer_interrupt+0x69/0xa0
[112935.447983] [<ffffffff8156454a>] apic_timer_interrupt+0x6a/0x70
[112935.454144] [<ffffffff81562eed>] ? _raw_spin_unlock_irq+0xd/0x20
[112935.460391] [<ffffffff810647a7>] finish_task_switch+0x57/0xe0
[112935.466371] [<ffffffff8100d9fb>] ? register_nmi_handler+0x4b/0x180
[112935.472793] [<ffffffff81561cbb>] __schedule+0x3ab/0x720
[112935.478254] [<ffffffff810a6e60>] ? watchdog_enable+0x240/0x240
[112935.484326] [<ffffffff815622f4>] schedule+0x24/0x70
[112935.489443] [<ffffffff810a6eda>] watchdog+0x7a/0xd0
[112935.494557] [<ffffffff8105bd2e>] kthread+0x8e/0xa0
[112935.499583] [<ffffffff81564da4>] kernel_thread_helper+0x4/0x10
[112935.505650] [<ffffffff8105bca0>] ? flush_kthread_work+0x150/0x150
[112935.511983] [<ffffffff81564da0>] ? gs_change+0x13/0x13
[112935.517356] ---[ end trace 6552bded36af03ab ]---
[112992.222406] BUG: spinlock lockup on CPU#0, watchdog/0/7
[112992.227780] lock: ffff8804346d5d20, .magic: dead4ead, .owner: swapper/0/0, .owner_cpu: 0
[112992.237075] Pid: 7, comm: watchdog/0 Tainted: G D W 3.4.39-21 #8
[112992.243928] Call Trace:
[112992.246526] [<ffffffff8155cf14>] spin_dump+0x8a/0x8f
[112992.251727] [<ffffffff8128a9d8>] do_raw_spin_lock+0xe8/0x120
[112992.257627] [<ffffffff815630ef>] _raw_spin_lock_irqsave+0x1f/0x30
[112992.263962] [<ffffffff81404126>] input_repeat_key+0x26/0xd0
[112992.269770] [<ffffffff8104b56f>] run_timer_softirq+0xff/0x230
[112992.275757] [<ffffffff81011819>] ? read_tsc+0x9/0x20
[112992.280957] [<ffffffff81404100>] ? input_dev_resume+0x20/0x20
[112992.286946] [<ffffffff810276e8>] ? lapic_next_event+0x18/0x20
[112992.292933] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112992.298481] [<ffffffff8108171f>] ? tick_program_event+0x1f/0x30
[112992.304642] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112992.310103] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112992.315390] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112992.320504] [<ffffffff81027b79>] smp_apic_timer_interrupt+0x69/0xa0
[112992.327012] [<ffffffff8156454a>] apic_timer_interrupt+0x6a/0x70
[112992.333173] [<ffffffff81562eed>] ? _raw_spin_unlock_irq+0xd/0x20
[112992.339421] [<ffffffff810647a7>] finish_task_switch+0x57/0xe0
[112992.345408] [<ffffffff8100d9fb>] ? register_nmi_handler+0x4b/0x180
[112992.351830] [<ffffffff81561cbb>] __schedule+0x3ab/0x720
[112992.357289] [<ffffffff810a6e60>] ? watchdog_enable+0x240/0x240
[112992.363365] [<ffffffff815622f4>] schedule+0x24/0x70
[112992.368479] [<ffffffff810a6eda>] watchdog+0x7a/0xd0
[112992.373593] [<ffffffff8105bd2e>] kthread+0x8e/0xa0
[112992.378620] [<ffffffff81564da4>] kernel_thread_helper+0x4/0x10
[112992.384694] [<ffffffff8105bca0>] ? flush_kthread_work+0x150/0x150
[112992.391029] [<ffffffff81564da0>] ? gs_change+0x13/0x13
[112992.396403] sending NMI to all CPUs:
[112992.400133] NMI backtrace for cpu 0
[112992.403775] CPU 0
[112992.405739] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112992.434321]
[112992.435965] Pid: 7, comm: watchdog/0 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112992.447259] RIP: 0010:[<ffffffff81012496>] [<ffffffff81012496>] native_read_tsc+0x6/0x20
[112992.455670] RSP: 0018:ffff8802348bfaa8 EFLAGS: 00000046
[112992.461130] RAX: 00000000fb5c0731 RBX: 0000000000000000 RCX: 00000000fb5c0707
[112992.468441] RDX: 000000000000f72f RSI: 0000000000000080 RDI: 000000000003a1fe
[112992.475754] RBP: ffff8802348bfaa8 R08: ffffffff81885f10 R09: 0000000000000000
[112992.483066] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000fb5c068c
[112992.490379] R13: 0000000000000000 R14: 000000000003a1fe R15: 0000000000000096
[112992.497690] FS: 0000000000000000(0000) GS:ffff880237c00000(0000) knlGS:0000000000000000
[112992.505954] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112992.511846] CR2: 0000000000000000 CR3: 0000000233bcb000 CR4: 00000000000007f0
[112992.519159] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112992.526473] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112992.533783] Process watchdog/0 (pid: 7, threadinfo ffff8802348be000, task ffff880234899890)
[112992.542307] Stack:
[112992.544470] ffff8802348bfad8 ffffffff8128362a 00000000000003e9 0000000000001000
[112992.552258] ffffffff81885f10 0000000000000400 ffff8802348bfae8 ffffffff8128357c
[112992.560046] ffff8802348bfb08 ffffffff81027a32 0000000000000002 000000000000dc42
[112992.567835] Call Trace:
[112992.570433] [<ffffffff8128362a>] delay_tsc+0x4a/0x80
[112992.575634] [<ffffffff8128357c>] __const_udelay+0x2c/0x30
[112992.581276] [<ffffffff81027a32>] native_safe_apic_wait_icr_idle+0x22/0x50
[112992.588302] [<ffffffff81028bdf>] default_send_IPI_mask_sequence_phys+0xcf/0xe0
[112992.595788] [<ffffffff8102cd02>] physflat_send_IPI_all+0x12/0x20
[112992.602034] [<ffffffff81028df0>] arch_trigger_all_cpu_backtrace+0x60/0xa0
[112992.609060] [<ffffffff8128a9dd>] do_raw_spin_lock+0xed/0x120
[112992.614953] [<ffffffff815630ef>] _raw_spin_lock_irqsave+0x1f/0x30
[112992.621288] [<ffffffff81404126>] input_repeat_key+0x26/0xd0
[112992.627095] [<ffffffff8104b56f>] run_timer_softirq+0xff/0x230
[112992.633082] [<ffffffff81011819>] ? read_tsc+0x9/0x20
[112992.638283] [<ffffffff81404100>] ? input_dev_resume+0x20/0x20
[112992.644264] [<ffffffff810276e8>] ? lapic_next_event+0x18/0x20
[112992.650251] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112992.655797] [<ffffffff8108171f>] ? tick_program_event+0x1f/0x30
[112992.661951] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112992.667412] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112992.672699] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112992.677813] [<ffffffff81027b79>] smp_apic_timer_interrupt+0x69/0xa0
[112992.684319] [<ffffffff8156454a>] apic_timer_interrupt+0x6a/0x70
[112992.690480] [<ffffffff81562eed>] ? _raw_spin_unlock_irq+0xd/0x20
[112992.696728] [<ffffffff810647a7>] finish_task_switch+0x57/0xe0
[112992.702709] [<ffffffff8100d9fb>] ? register_nmi_handler+0x4b/0x180
[112992.709131] [<ffffffff81561cbb>] __schedule+0x3ab/0x720
[112992.714589] [<ffffffff810a6e60>] ? watchdog_enable+0x240/0x240
[112992.720664] [<ffffffff815622f4>] schedule+0x24/0x70
[112992.725780] [<ffffffff810a6eda>] watchdog+0x7a/0xd0
[112992.730893] [<ffffffff8105bd2e>] kthread+0x8e/0xa0
[112992.735920] [<ffffffff81564da4>] kernel_thread_helper+0x4/0x10
[112992.741996] [<ffffffff8105bca0>] ? flush_kthread_work+0x150/0x150
[112992.748329] [<ffffffff81564da0>] ? gs_change+0x13/0x13
[112992.753703] Code: c3 90 90 90 90 55 89 f8 48 89 e5 e6 70 e4 71 5d c3 0f 1f 40 00 55 89 f0 48 89 e5 e6 70 89 f8 e6 71 5d c3 66 90 55 48 89 e5 0f 31 <89> c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 66 66 66 2e 0f 1f 84
[112992.776079] Call Trace:
[112992.778678] [<ffffffff8128362a>] delay_tsc+0x4a/0x80
[112992.783878] [<ffffffff8128357c>] __const_udelay+0x2c/0x30
[112992.789511] [<ffffffff81027a32>] native_safe_apic_wait_icr_idle+0x22/0x50
[112992.796537] [<ffffffff81028bdf>] default_send_IPI_mask_sequence_phys+0xcf/0xe0
[112992.804024] [<ffffffff8102cd02>] physflat_send_IPI_all+0x12/0x20
[112992.810272] [<ffffffff81028df0>] arch_trigger_all_cpu_backtrace+0x60/0xa0
[112992.817297] [<ffffffff8128a9dd>] do_raw_spin_lock+0xed/0x120
[112992.823198] [<ffffffff815630ef>] _raw_spin_lock_irqsave+0x1f/0x30
[112992.829532] [<ffffffff81404126>] input_repeat_key+0x26/0xd0
[112992.835339] [<ffffffff8104b56f>] run_timer_softirq+0xff/0x230
[112992.841319] [<ffffffff81011819>] ? read_tsc+0x9/0x20
[112992.846520] [<ffffffff81404100>] ? input_dev_resume+0x20/0x20
[112992.852509] [<ffffffff810276e8>] ? lapic_next_event+0x18/0x20
[112992.858496] [<ffffffff81045da8>] __do_softirq+0x98/0x120
[112992.864044] [<ffffffff8108171f>] ? tick_program_event+0x1f/0x30
[112992.870205] [<ffffffff81564e9c>] call_softirq+0x1c/0x30
[112992.875664] [<ffffffff8100c215>] do_softirq+0x65/0xa0
[112992.880951] [<ffffffff810460ae>] irq_exit+0x8e/0xb0
[112992.886065] [<ffffffff81027b79>] smp_apic_timer_interrupt+0x69/0xa0
[112992.892574] [<ffffffff8156454a>] apic_timer_interrupt+0x6a/0x70
[112992.898736] [<ffffffff81562eed>] ? _raw_spin_unlock_irq+0xd/0x20
[112992.904983] [<ffffffff810647a7>] finish_task_switch+0x57/0xe0
[112992.910971] [<ffffffff8100d9fb>] ? register_nmi_handler+0x4b/0x180
[112992.917393] [<ffffffff81561cbb>] __schedule+0x3ab/0x720
[112992.922851] [<ffffffff810a6e60>] ? watchdog_enable+0x240/0x240
[112992.928927] [<ffffffff815622f4>] schedule+0x24/0x70
[112992.934042] [<ffffffff810a6eda>] watchdog+0x7a/0xd0
[112992.939155] [<ffffffff8105bd2e>] kthread+0x8e/0xa0
[112992.944182] [<ffffffff81564da4>] kernel_thread_helper+0x4/0x10
[112992.950257] [<ffffffff8105bca0>] ? flush_kthread_work+0x150/0x150
[112992.956591] [<ffffffff81564da0>] ? gs_change+0x13/0x13
[112992.962023] NMI backtrace for cpu 1
[112992.965669] CPU 1
[112992.967633] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112992.996215]
[112992.997858] Pid: 0, comm: swapper/1 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112993.009066] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112993.017217] RSP: 0018:ffff88023499de48 EFLAGS: 00000046
[112993.022677] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[112993.029990] RDX: 0000000000000000 RSI: ffff88023499dfd8 RDI: ffffffff81815880
[112993.037300] RBP: ffff88023499dea8 R08: 0000000000000000 R09: 000000000000020f
[112993.044612] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000003
[112993.051925] R13: ffff880237c39470 R14: 0000000000000020 R15: 12f57b20d8479d09
[112993.059237] FS: 0000000000000000(0000) GS:ffff880237c20000(0000) knlGS:0000000000000000
[112993.067501] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112993.073395] CR2: 00007f8d804d2000 CR3: 000000000180b000 CR4: 00000000000007e0
[112993.080708] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112993.088018] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112993.095331] Process swapper/1 (pid: 0, threadinfo ffff88023499c000, task ffff88023489c9b0)
[112993.103768] Stack:
[112993.105930] 0000000000000000 000000000011bf06 ffff88023499deb8 ffffffff814375b0
[112993.113718] 0000000000000000 000000010011bf06 0000000000000000 ffff880237c39470
[112993.121507] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112993.129295] Call Trace:
[112993.132812] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.138452] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.143999] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.149987] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.155188] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.161089] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112993.183467] Call Trace:
[112993.186063] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.191696] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.197244] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.203231] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.208433] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.214335] NMI backtrace for cpu 2
[112993.215370] INFO: rcu_sched detected stalls on CPUs/tasks:
[112993.215374] 0: (1 GPs behind) idle=ccf/3/0 drain=0 . timer=-1
[112993.215376] (detected by 1, t=563040 jiffies)
[112993.215377] INFO: Stall ended before state dump start
[112993.215379] INFO: rcu_bh detected stalls on CPUs/tasks:
[112993.215381] 0: (1 GPs behind) idle=ccf/3/0 drain=0 . timer=-1
[112993.215383] (detected by 1, t=562877 jiffies)
[112993.215384] INFO: Stall ended before state dump start
[112993.260534] CPU 2
[112993.262498] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112993.291080]
[112993.292725] Pid: 0, comm: swapper/2 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112993.303932] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112993.312083] RSP: 0018:ffff8802349b5e48 EFLAGS: 00000046
[112993.317544] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[112993.324854] RDX: 0000000000000000 RSI: ffff8802349b5fd8 RDI: ffffffff81815880
[112993.332166] RBP: ffff8802349b5ea8 R08: 0000000000000000 R09: 0000000000000199
[112993.339479] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000002
[112993.346792] R13: ffff880237c59470 R14: 0000000000000010 R15: 12f57b20d846a366
[112993.354155] FS: 0000000000000000(0000) GS:ffff880237c40000(0000) knlGS:0000000000000000
[112993.362419] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112993.368311] CR2: 00007f6c0453ebd0 CR3: 000000000180b000 CR4: 00000000000007e0
[112993.375625] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112993.382938] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112993.390248] Process swapper/2 (pid: 0, threadinfo ffff8802349b4000, task ffff8802349bf2a0)
[112993.398685] Stack:
[112993.400848] 0000000000000000 000000000010000c ffff8802349b5eb8 ffffffff814375b0
[112993.408636] 0000000000000000 000000020010000c 0000000000000000 ffff880237c59470
[112993.416426] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112993.424214] Call Trace:
[112993.426811] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.432443] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.437991] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.443979] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.449180] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.455079] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112993.477457] Call Trace:
[112993.480055] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.485697] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.491245] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.497233] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.502433] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.508336] NMI backtrace for cpu 3
[112993.511976] CPU 3
[112993.513940] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112993.542523]
[112993.544168] Pid: 0, comm: swapper/3 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112993.555374] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112993.563525] RSP: 0018:ffff8802349d7e48 EFLAGS: 00000046
[112993.568984] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[112993.576297] RDX: 0000000000000000 RSI: ffff8802349d7fd8 RDI: ffffffff81815880
[112993.583609] RBP: ffff8802349d7ea8 R08: 0000000000000000 R09: 0000000000000682
[112993.590922] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000002
[112993.598235] R13: ffff880237c79470 R14: 0000000000000010 R15: 12f57b20d840513c
[112993.605546] FS: 0000000000000000(0000) GS:ffff880237c60000(0000) knlGS:0000000000000000
[112993.613810] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112993.619704] CR2: 00007f8390134098 CR3: 000000000180b000 CR4: 00000000000007e0
[112993.627015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112993.634328] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112993.641640] Process swapper/3 (pid: 0, threadinfo ffff8802349d6000, task ffff8802349b9890)
[112993.650085] Stack:
[112993.652248] 0000000000000000 0000000000190ba0 ffff8802349d7eb8 ffffffff814375b0
[112993.660027] 0000000000000000 0000000300190ba0 0000000000000000 ffff880237c79470
[112993.667816] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112993.675604] Call Trace:
[112993.678202] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.683844] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.689391] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.695378] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.700579] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.706471] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112993.728849] Call Trace:
[112993.731445] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.737079] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.742626] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.748615] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.753815] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.759717] NMI backtrace for cpu 4
[112993.763358] CPU 4
[112993.765323] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112993.793914]
[112993.795559] Pid: 0, comm: swapper/4 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112993.806763] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112993.814915] RSP: 0018:ffff8802349dbe48 EFLAGS: 00000046
[112993.820377] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[112993.827689] RDX: 0000000000000000 RSI: ffff8802349dbfd8 RDI: ffffffff81815880
[112993.835001] RBP: ffff8802349dbea8 R08: 0000000000000000 R09: 0000000000000000
[112993.842311] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003
[112993.849625] R13: ffff88043fc19470 R14: 0000000000000020 R15: 12f57b20d83d416f
[112993.856938] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
[112993.865209] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112993.871104] CR2: 00007f8390133028 CR3: 000000000180b000 CR4: 00000000000007e0
[112993.878414] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112993.885728] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112993.893039] Process swapper/4 (pid: 0, threadinfo ffff8802349da000, task ffff8802349ba0c0)
[112993.901476] Stack:
[112993.903639] 0000000000000000 000000000ee4bff3 ffff8802349dbeb8 ffffffff814375b0
[112993.911427] 0000000000000000 000000040ee4bff3 0000000000000000 ffff88043fc19470
[112993.919216] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112993.927004] Call Trace:
[112993.929601] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.935233] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.940781] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112993.946771] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112993.951971] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112993.957870] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112993.981182] Call Trace:
[112993.983780] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112993.989413] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112993.994960] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.000948] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.006149] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.012052] NMI backtrace for cpu 5
[112994.015693] CPU 5
[112994.017657] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112994.046257]
[112994.047902] Pid: 0, comm: swapper/5 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112994.059098] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112994.067249] RSP: 0018:ffff8802349dde48 EFLAGS: 00000046
[112994.072712] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
[112994.080022] RDX: 0000000000000000 RSI: ffff8802349ddfd8 RDI: ffffffff81815880
[112994.087335] RBP: ffff8802349ddea8 R08: 0000000000000000 R09: 00000000001a76e6
[112994.094648] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000002
[112994.101959] R13: ffff88043fc39470 R14: 0000000000000010 R15: 12f57b20d8445d94
[112994.109273] FS: 0000000000000000(0000) GS:ffff88043fc20000(0000) knlGS:0000000000000000
[112994.117543] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112994.123436] CR2: 00007f838c2c3058 CR3: 000000000180b000 CR4: 00000000000007e0
[112994.130748] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112994.138061] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112994.145374] Process swapper/5 (pid: 0, threadinfo ffff8802349dc000, task ffff8802349bd1e0)
[112994.153818] Stack:
[112994.155981] 0000000000000000 000000000015014a ffff8802349ddeb8 ffffffff814375b0
[112994.163770] 0000000000000000 000000050015014a 0000000000000000 ffff88043fc39470
[112994.171559] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112994.179347] Call Trace:
[112994.181945] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.187586] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.193133] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.199122] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.204323] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.210222] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112994.232600] Call Trace:
[112994.235196] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.240831] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.246376] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.252356] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.257559] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.263459] NMI backtrace for cpu 6
[112994.267100] CPU 6
[112994.269065] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112994.297648]
[112994.299292] Pid: 0, comm: swapper/6 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112994.310490] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112994.318640] RSP: 0018:ffff8802349f1e48 EFLAGS: 00000046
[112994.324102] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[112994.331412] RDX: 0000000000000000 RSI: ffff8802349f1fd8 RDI: ffffffff81815880
[112994.338725] RBP: ffff8802349f1ea8 R08: 0000000000000000 R09: 0000000000000000
[112994.346037] R10: 0000000000000002 R11: 0000000000000000 R12: 0000000000000003
[112994.353349] R13: ffff88043fc59470 R14: 0000000000000020 R15: 12f57b20d83d8930
[112994.360663] FS: 0000000000000000(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
[112994.368927] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112994.374820] CR2: 00007f0aceea5bd0 CR3: 000000000180b000 CR4: 00000000000007e0
[112994.382131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112994.389444] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112994.396757] Process swapper/6 (pid: 0, threadinfo ffff8802349f0000, task ffff8802349ba8f0)
[112994.405201] Stack:
[112994.407364] 0000000000000000 00000000001bd26d ffff8802349f1eb8 ffffffff814375b0
[112994.415153] 0000000000000000 00000006001bd26d 0000000000000000 ffff88043fc59470
[112994.422941] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112994.430729] Call Trace:
[112994.433326] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.438959] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.444506] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.450495] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.455695] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.461595] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112994.483972] Call Trace:
[112994.486571] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.492213] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.497760] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.503749] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.508949] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.514850] NMI backtrace for cpu 7
[112994.518492] CPU 7
[112994.520456] Modules linked in: coretemp psmouse ioatdma i7core_edac edac_core microcode dca serio_raw tcp_htcp raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov usbhid hid ahci libahci mptsas mptscsih mptbase scsi_transport_sas e1000e raid6_pq async_tx raid1 raid0 multipath linear
[112994.549047]
[112994.550692] Pid: 0, comm: swapper/7 Tainted: G D W 3.4.39-21 #8 System manufacturer System Product Name/Z8NR-D12
[112994.561889] RIP: 0010:[<ffffffff812d8137>] [<ffffffff812d8137>] intel_idle+0xb7/0x150
[112994.570040] RSP: 0018:ffff8802349f5e48 EFLAGS: 00000046
[112994.575502] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[112994.582812] RDX: 0000000000000000 RSI: ffff8802349f5fd8 RDI: ffffffff81815880
[112994.590125] RBP: ffff8802349f5ea8 R08: 0000000000000000 R09: 0000000000000000
[112994.597438] R10: 0000000000000001 R11: 0000000000000ffe R12: 0000000000000003
[112994.604748] R13: ffff88043fc79470 R14: 0000000000000020 R15: 12f57b20d83d4e60
[112994.612061] FS: 0000000000000000(0000) GS:ffff88043fc60000(0000) knlGS:0000000000000000
[112994.620325] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[112994.626218] CR2: 00007f8d804d2000 CR3: 000000000180b000 CR4: 00000000000007e0
[112994.633530] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[112994.640843] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[112994.648156] Process swapper/7 (pid: 0, threadinfo ffff8802349f4000, task ffff8802349bc9b0)
[112994.656600] Stack:
[112994.658763] 0000000000000000 000000002c61ee04 ffff8802349f5eb8 ffffffff814375b0
[112994.666552] 0000000000000000 000000072c61ee04 0000000000000000 ffff88043fc79470
[112994.674340] ffffffff8183abc0 00000000fffffff0 0000000000000000 0000000000000000
[112994.682127] Call Trace:
[112994.684726] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.690358] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.695905] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.701893] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.707096] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd
[112994.712995] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> d4 26 da ff 4c 29 f8 48 89 c7 e8 59 cf d6 ff 4c 69 f0 40 42
[112994.735373] Call Trace:
[112994.737970] [<ffffffff814375b0>] ? menu_select+0xf0/0x3a0
[112994.743603] [<ffffffff81435dd9>] cpuidle_enter+0x19/0x20
[112994.749151] [<ffffffff8143639b>] cpuidle_idle_call+0x8b/0x100
[112994.755138] [<ffffffff81013535>] cpu_idle+0xc5/0x120
[112994.760339] [<ffffffff8154fb47>] start_secondary+0x1f6/0x1fd