[4.2-rc7][Haswell] NMI soft lockup - try_to_del_timer_sync

From: Shawn Starr
Date: Thu Aug 20 2015 - 22:49:34 EST

Hello folks,

I was just using KVM and doing some VM work on my laptop when it locked up system fully, here is the kernel trace below (it repeats after the second dump)

Since we're soon 'close' to 4.2 final, maybe someone can check if something regressed? Never saw this in eariler -rcX builds.

Thanks,
Shawn.


[69399.505355] kvm: zapping shadow pages for mmio generation wraparound
[69405.611747] kvm [24415]: vcpu0 unhandled rdmsr: 0x641
[69411.477997] Bridge: port 2(vnet0) entered forwarding state
[70844.574880] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0]
[70844.575264] Modules linked in: vhost_net vhost macvtap macvlan tun ebtable_filter ebtables ip6table_filter ip6_tables bridge vfat fat intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel uvcvideo videobuf2_vmalloc videobuf2_core videobuf2_memops arc4 iwlmvm v4l2_common mac80211 videodev snd_usb_audio snd_usbmidi_lib snd_rawmidi media iwlwifi dell_laptop cfg80211 snd_hda_codec_realtek dell_wmi snd_soc_rt5640 iTCO_wdt iTCO_vendor_support joydev lpc_ich dcdbas sparse_keymap ppdev dell_smm_hwmon mei_me snd_hda_codec_generic snd_soc_rl6231 snd_hda_codec_hdmi mei ie31200_edac edac_core dw_dmac snd_soc_core parport_pc parport snd_compress snd_pcm_dmaengine shpchp video ac97_bus dw_dmac_core i2c_i801 i2c_designware_platform i2c_designware_core
[70844.579376] snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_soc_sst_acpi dell_rbtn rfkill wmi dell_smo8800 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc 8021q garp stp llc mrp amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper ttm e1000e drm serio_raw sdhci_pci ptp sdhci_acpi pps_core sdhci mmc_core i2c_hid
[70844.581351] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.2.0-0.rc7.git0.1.fc23.x86_64 #1
[70844.581804] Hardware name: Dell Inc. Precision M6800/05NG6V, BIOS A14 05/24/2015
[70844.582198] task: ffff880809f6c4c0 ti: ffff880809900000 task.ti: ffff880809900000
[70844.582555] RIP: 0010:[<ffffffff811036cb>] [<ffffffff811036cb>] try_to_del_timer_sync+0xb/0x90
[70844.583042] RSP: 0018:ffff88082dcc3da0 EFLAGS: 00000286
[70844.583296] RAX: 00000000ffffffff RBX: 0000000000000296 RCX: 0000000000000000
[70844.583637] RDX: 0000000000000001 RSI: 0000000000000296 RDI: ffff8800bc0612e0
[70844.584037] RBP: ffff88082dcc3da8 R08: 0000000000000101 R09: 0000000000000003
[70844.584373] R10: 0000000000000002 R11: 0000000000000004 R12: ffff88082dcc3d18
[70844.584771] R13: ffffffff8177818b R14: ffff88082dcc3da8 R15: ffff8800bc0612e0
[70844.585139] FS: 0000000000000000(0000) GS:ffff88082dcc0000(0000) knlGS:0000000000000000
[70844.585524] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[70844.585833] CR2: 0000140a84048000 CR3: 0000000001c0b000 CR4: 00000000001426e0
[70844.586200] Stack:
[70844.586301] ffff8807650fe800 ffff88082dcc3dc8 ffffffff8110379c ffff88082dcc3dc8
[70844.586687] ffff8800bc061248 ffff88082dcc3df8 ffffffff816ac35c 00000000000167c0
[70844.587187] ffff8800bc061248 ffff8807650fe800 ffff8807ec9c5000 ffff88082dcc3e38
[70844.587631] Call Trace:
[70844.587784] <IRQ>
[70844.587884] [<ffffffff8110379c>] del_timer_sync+0x4c/0x60
[70844.588188] [<ffffffff816ac35c>] inet_csk_reqsk_queue_drop+0x6c/0x1d0
[70844.588499] [<ffffffff816ac599>] reqsk_timer_handler+0xd9/0x2a0
[70844.588817] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70844.589158] [<ffffffff811034f9>] call_timer_fn+0x39/0xf0
[70844.589432] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70844.589774] [<ffffffff81103b2f>] run_timer_softirq+0x20f/0x2c0
[70844.590105] [<ffffffff810a280b>] __do_softirq+0xfb/0x290
[70844.590365] [<ffffffff810a2bb9>] irq_exit+0x119/0x120
[70844.590618] [<ffffffff81779ff6>] smp_apic_timer_interrupt+0x46/0x60
[70844.590929] [<ffffffff8177818b>] apic_timer_interrupt+0x6b/0x70
[70844.591222] <EOI>
[70844.591322] [<ffffffff8110653f>] ? hrtimer_start_range_ns+0x1bf/0x3b0
[70844.591656] [<ffffffff816136e0>] ? cpuidle_enter_state+0x130/0x270
[70844.591961] [<ffffffff816136bb>] ? cpuidle_enter_state+0x10b/0x270
[70844.592265] [<ffffffff81613857>] cpuidle_enter+0x17/0x20
[70844.592527] [<ffffffff810df402>] call_cpuidle+0x32/0x60
[70844.592787] [<ffffffff81613833>] ? cpuidle_select+0x13/0x20
[70844.593060] [<ffffffff810df698>] cpu_startup_entry+0x268/0x320
[70844.593347] [<ffffffff8104cbd6>] start_secondary+0x186/0x1c0
[70844.593623] Code: 00 00 44 89 e0 75 0b 48 83 c4 18 5b 41 5c 41 5d 5d c3 e8 79 ab f9 ff 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 <53> 48 8d 75 e0 48 89 fb 48 83 c7 28 48 83 ec 10 65 48 8b 04 25
[70872.581083] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0]
[70872.581462] Modules linked in: vhost_net vhost macvtap macvlan tun ebtable_filter ebtables ip6table_filter ip6_tables bridge vfat fat intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel uvcvideo videobuf2_vmalloc videobuf2_core videobuf2_memops arc4 iwlmvm v4l2_common mac80211 videodev snd_usb_audio snd_usbmidi_lib snd_rawmidi media iwlwifi dell_laptop cfg80211 snd_hda_codec_realtek dell_wmi snd_soc_rt5640 iTCO_wdt iTCO_vendor_support joydev lpc_ich dcdbas sparse_keymap ppdev dell_smm_hwmon mei_me snd_hda_codec_generic snd_soc_rl6231 snd_hda_codec_hdmi mei ie31200_edac edac_core dw_dmac snd_soc_core parport_pc parport snd_compress snd_pcm_dmaengine shpchp video ac97_bus dw_dmac_core i2c_i801 i2c_designware_platform i2c_designware_core
[70872.585614] snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_soc_sst_acpi dell_rbtn rfkill wmi dell_smo8800 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc 8021q garp stp llc mrp amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper ttm e1000e drm serio_raw sdhci_pci ptp sdhci_acpi pps_core sdhci mmc_core i2c_hid
[70872.587536] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 4.2.0-0.rc7.git0.1.fc23.x86_64 #1
[70872.587974] Hardware name: Dell Inc. Precision M6800/05NG6V, BIOS A14 05/24/2015
[70872.588345] task: ffff880809f6c4c0 ti: ffff880809900000 task.ti: ffff880809900000
[70872.588708] RIP: 0010:[<ffffffff811036c0>] [<ffffffff811036c0>] try_to_del_timer_sync+0x0/0x90
[70872.589144] RSP: 0018:ffff88082dcc3db0 EFLAGS: 00000286
[70872.589446] RAX: 00000000ffffffff RBX: 0000000000000003 RCX: 0000000000000000
[70872.589801] RDX: 0000000000000001 RSI: 0000000000000296 RDI: ffff8800bc0612e0
[70872.590173] RBP: ffff88082dcc3dc8 R08: 0000000000000101 R09: 0000000000000003
[70872.590543] R10: 0000000000000002 R11: 0000000000000004 R12: ffff88082dcc3d28
[70872.590882] R13: ffffffff8177818b R14: ffff88082dcc3dc8 R15: ffff8800bc0612e0
[70872.591255] FS: 0000000000000000(0000) GS:ffff88082dcc0000(0000) knlGS:0000000000000000
[70872.591659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[70872.591938] CR2: 0000140a84048000 CR3: 0000000001c0b000 CR4: 00000000001426e0
[70872.592291] Stack:
[70872.592395] ffffffff8110379c ffff88082dcc3dc8 ffff8800bc061248 ffff88082dcc3df8
[70872.592782] ffffffff816ac35c 00000000000167c0 ffff8800bc061248 ffff8807650fe800
[70872.593171] ffff8807ec9c5000 ffff88082dcc3e38 ffffffff816ac599 00000000000167c0
[70872.593560] Call Trace:
[70872.593684] <IRQ>
[70872.593784] [<ffffffff8110379c>] ? del_timer_sync+0x4c/0x60
[70872.594076] [<ffffffff816ac35c>] inet_csk_reqsk_queue_drop+0x6c/0x1d0
[70872.594406] [<ffffffff816ac599>] reqsk_timer_handler+0xd9/0x2a0
[70872.594698] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70872.595029] [<ffffffff811034f9>] call_timer_fn+0x39/0xf0
[70872.595304] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70872.595662] [<ffffffff81103b2f>] run_timer_softirq+0x20f/0x2c0
[70872.595948] [<ffffffff810a280b>] __do_softirq+0xfb/0x290
[70872.596315] [<ffffffff810a2bb9>] irq_exit+0x119/0x120
[70872.596589] [<ffffffff81779ff6>] smp_apic_timer_interrupt+0x46/0x60
[70872.596894] [<ffffffff8177818b>] apic_timer_interrupt+0x6b/0x70
[70872.597211] <EOI>
[70872.597323] [<ffffffff8110653f>] ? hrtimer_start_range_ns+0x1bf/0x3b0
[70872.597655] [<ffffffff816136e0>] ? cpuidle_enter_state+0x130/0x270
[70872.597956] [<ffffffff816136bb>] ? cpuidle_enter_state+0x10b/0x270
[70872.598261] [<ffffffff81613857>] cpuidle_enter+0x17/0x20
[70872.598546] [<ffffffff810df402>] call_cpuidle+0x32/0x60
[70872.598850] [<ffffffff81613833>] ? cpuidle_select+0x13/0x20
[70872.599182] [<ffffffff810df698>] cpu_startup_entry+0x268/0x320
[70872.599471] [<ffffffff8104cbd6>] start_secondary+0x186/0x1c0
[70872.599778] Code: 48 8b 4d d8 65 48 33 0c 25 28 00 00 00 44 89 e0 75 0b 48 83 c4 18 5b 41 5c 41 5d 5d c3 e8 79 ab f9 ff 66 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 41 54 53 48 8d 75 e0 48 89 fb 48 83
[70877.828815] INFO: rcu_sched detected stalls on CPUs/tasks: { 3} (detected by 4, t=60002 jiffies, g=7274139, c=7274138, q=0)
[70877.829393] Task dump for CPU 3:
[70877.829553] swapper/3 R running task 0 0 1 0x00000008
[70877.829910] 00004064a7466c97 000000002dcd3880 ffffffff81cb86f8 0000000000000003
[70877.830300] ffffffff81d2e2c0 ffff880809904000 ffff88082dcde200 ffffffff81cb8500
[70877.830686] ffff880809900000 ffff880809903ea8 ffffffff81613857 ffff880809903ec8
[70877.831071] Call Trace:
[70877.831200] [<ffffffff81613857>] ? cpuidle_enter+0x17/0x20
[70877.831475] [<ffffffff810df402>] ? call_cpuidle+0x32/0x60
[70877.831739] [<ffffffff81613833>] ? cpuidle_select+0x13/0x20
[70877.832011] [<ffffffff810df698>] ? cpu_startup_entry+0x268/0x320
[70877.832308] [<ffffffff8104cbd6>] ? start_secondary+0x186/0x1c0
[70900.587287] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0]
[70900.587658] Modules linked in: vhost_net vhost macvtap macvlan tun ebtable_filter ebtables ip6table_filter ip6_tables bridge vfat fat intel_rapl iosf_mbi x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel uvcvideo videobuf2_vmalloc videobuf2_core videobuf2_memops arc4 iwlmvm v4l2_common mac80211 videodev snd_usb_audio snd_usbmidi_lib snd_rawmidi media iwlwifi dell_laptop cfg80211 snd_hda_codec_realtek dell_wmi snd_soc_rt5640 iTCO_wdt iTCO_vendor_support joydev lpc_ich dcdbas sparse_keymap ppdev dell_smm_hwmon mei_me snd_hda_codec_generic snd_soc_rl6231 snd_hda_codec_hdmi mei ie31200_edac edac_core dw_dmac snd_soc_core parport_pc parport snd_compress snd_pcm_dmaengine shpchp video ac97_bus dw_dmac_core i2c_i801 i2c_designware_platform i2c_designware_core
[70900.591765] snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_soc_sst_acpi dell_rbtn rfkill wmi dell_smo8800 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc 8021q garp stp llc mrp amdkfd amd_iommu_v2 radeon i2c_algo_bit drm_kms_helper ttm e1000e drm serio_raw sdhci_pci ptp sdhci_acpi pps_core sdhci mmc_core i2c_hid
[70900.593863] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 4.2.0-0.rc7.git0.1.fc23.x86_64 #1
[70900.594402] Hardware name: Dell Inc. Precision M6800/05NG6V, BIOS A14 05/24/2015
[70900.594801] task: ffff880809f6c4c0 ti: ffff880809900000 task.ti: ffff880809900000
[70900.595186] RIP: 0010:[<ffffffff81776d41>] [<ffffffff81776d41>] _raw_write_unlock_irqrestore+0x11/0x20
[70900.595677] RSP: 0018:ffff88082dcc3d68 EFLAGS: 00000296
[70900.595962] RAX: ffff88082dccec00 RBX: ffff88082dcc3da8 RCX: 0000000000000000
[70900.596364] RDX: 0000000000000001 RSI: 0000000000000296 RDI: 0000000000000296
[70900.596702] RBP: ffff88082dcc3d68 R08: 0000000000000101 R09: 0000000000000003
[70900.597043] R10: 0000000000000002 R11: 0000000000000004 R12: ffff88082dcc3cd8
[70900.597409] R13: ffffffff8177818b R14: ffff88082dcc3d68 R15: 00000000ffffffff
[70900.597751] FS: 0000000000000000(0000) GS:ffff88082dcc0000(0000) knlGS:0000000000000000
[70900.598137] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[70900.598412] CR2: 0000140a84048000 CR3: 0000000001c0b000 CR4: 00000000001426e0
[70900.598755] Stack:
[70900.598857] ffff88082dcc3d78 ffffffff81776d5e ffff88082dcc3da8 ffffffff8110371e
[70900.599279] 0000000000000296 2eb4abc0c48f26fd ffff8800bc0612e0 ffff8807650fe800
[70900.599670] ffff88082dcc3dc8 ffffffff8110379c ffff88082dcc3dc8 ffff8800bc061248
[70900.600057] Call Trace:
[70900.600181] <IRQ>
[70900.600312] [<ffffffff81776d5e>] _raw_spin_unlock_irqrestore+0xe/0x10
[70900.600644] [<ffffffff8110371e>] try_to_del_timer_sync+0x5e/0x90
[70900.600937] [<ffffffff8110379c>] del_timer_sync+0x4c/0x60
[70900.601203] [<ffffffff816ac35c>] inet_csk_reqsk_queue_drop+0x6c/0x1d0
[70900.601546] [<ffffffff816ac599>] reqsk_timer_handler+0xd9/0x2a0
[70900.601885] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70900.602256] [<ffffffff811034f9>] call_timer_fn+0x39/0xf0
[70900.602544] [<ffffffff816ac4c0>] ? inet_csk_reqsk_queue_drop+0x1d0/0x1d0
[70900.602868] [<ffffffff81103b2f>] run_timer_softirq+0x20f/0x2c0
[70900.603154] [<ffffffff810a280b>] __do_softirq+0xfb/0x290
[70900.603443] [<ffffffff810a2bb9>] irq_exit+0x119/0x120
[70900.603693] [<ffffffff81779ff6>] smp_apic_timer_interrupt+0x46/0x60
[70900.603997] [<ffffffff8177818b>] apic_timer_interrupt+0x6b/0x70
[70900.604299] <EOI>
[70900.604398] [<ffffffff8110653f>] ? hrtimer_start_range_ns+0x1bf/0x3b0
[70900.604733] [<ffffffff816136e0>] ? cpuidle_enter_state+0x130/0x270
[70900.605036] [<ffffffff816136bb>] ? cpuidle_enter_state+0x10b/0x270
[70900.605406] [<ffffffff81613857>] cpuidle_enter+0x17/0x20
[70900.605667] [<ffffffff810df402>] call_cpuidle+0x32/0x60
[70900.605924] [<ffffffff81613833>] ? cpuidle_select+0x13/0x20
[70900.606196] [<ffffffff810df698>] cpu_startup_entry+0x268/0x320
[70900.606539] [<ffffffff8104cbd6>] start_secondary+0x186/0x1c0
[70900.606815] Code: 00 fb 66 0f 1f 44 00 00 eb cc 31 c0 eb d0 e8 07 75 92 ff 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/