[Question] suse11sp3 32bit guest os panic on kvm

From: Gonglei (Arei)
Date: Wed Jul 02 2014 - 05:48:48 EST

Next message: Philipp Reisner: "Re: [Drbd-dev] [PATCH 15/20] drbd: cosmetic: change all printk(level, ...) to pr_<level>(...)"
Previous message: Dmitry Kasatkin: "IMA: kernel reading files opened with O_DIRECT"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi, all

Recently I encounter a confused issue that described by the title.
I get the dump core according "virsh dump", and this below is the core
Information. It seem as that the vcpu deadlock between irqbalance
process(_raw_spin_lock_irqsave) and an interrupt handle
(_raw_spin_lock), I'm so confused.

Any suggestions will be appreciated!

crash> log
[...]
[71434.017632] sd 0:0:0:0: [sda] abort
[71428.004077] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 2
[71428.004077] Pid: 9846, comm: irqbalance Tainted: G N 3.0.76-0.11-default #1
[71428.004077] Call Trace:
[71428.004077] [<c0205741>] try_stack_unwind+0x1a1/0x1b0
[71428.004077] [<c0204727>] dump_trace+0x47/0x110
[71428.004077] [<c02052cb>] show_trace_log_lvl+0x4b/0x60
[71428.004077] [<c02052f8>] show_trace+0x18/0x20
[71428.004077] [<c05ca598>] dump_stack+0x6d/0x72
[71428.004077] [<c05ca5f4>] panic+0x57/0x196
[71428.004077] [<c02a3d8e>] watchdog_overflow_callback+0xae/0xb0
[71428.004077] [<c02cba77>] __perf_event_overflow+0xc7/0x290
[71428.004077] [<c02cc302>] perf_event_overflow+0x12/0x20
[71428.004077] [<c0214c29>] intel_pmu_handle_irq+0x199/0x320
[71428.004077] [<c05cef56>] perf_event_nmi_handler+0x26/0x80
[71428.004077] [<c05d0a96>] notifier_call_chain+0x36/0x60
[71428.004077] [<c05d0adb>] __atomic_notifier_call_chain+0x1b/0x20
[71428.004077] [<c05d0af7>] atomic_notifier_call_chain+0x17/0x20
[71428.004077] [<c05d0b30>] notify_die+0x30/0x40
[71428.004077] [<c05ce4a0>] default_do_nmi+0x30/0x280
[71428.004077] [<c05ce74f>] do_nmi+0x5f/0x70
[71428.004077] [<c05cdeb9>] nmi_stack_correct+0x28/0x2d
[71428.004077] [<c05ccebe>] _raw_spin_lock_irqsave+0x2e/0x50
[71428.004077] [<c02a76df>] show_interrupts+0xff/0x2f0
[71428.004077] [<c0337cd0>] seq_read+0x150/0x360
[71428.004077] [<c0366c50>] proc_reg_read+0x60/0x90
[71428.004077] [<c031b9fb>] vfs_read+0x9b/0x110
[71428.004077] [<c031bb41>] sys_read+0x41/0x80
[71428.004077] [<c05d401c>] sysenter_do_call+0x12/0x28
[71428.004077] [<ffffe430>] 0xffffe42f
[71428.004077] ------------[ cut here ]------------
[71428.004077] kernel BUG at /usr/src/packages/BUILD/kernel-default-3.0.76/linux-3.0/mm/vmalloc.c:1314!
[71428.004077] invalid opcode: 0000 [#1] SMP
[71428.004077] Modules linked in: edd mperf af_packet fuse loop dm_mod joydev usbhid hid sr_mod acpiphp ipv6 ipv6_lib floppy i2c_pii
x4 cdrom pci_hotplug rtc_cmos pcspkr sg button ext3 jbd mbcache ttm drm_kms_helper drm i2c_core sysimgblt sysfillrect syscopyarea uh
ci_hcd ehci_hcd processor sd_mod crc_t10dif usbcore intel_agp thermal_sys hwmon usb_common intel_gtt scsi_dh_alua scsi_dh_emc scsi_d
h_hp_sw scsi_dh_rdac scsi_dh ata_generic ata_piix libata pv_channel(N) virtio_net virtio_blk virtio_pci virtio_scsi scsi_mod virtio_
console virtio virtio_ring kvm_ivshmem(N)
[71428.004077] Supported: No, Unsupported modules are loaded
[71428.004077]
[71428.004077] Pid: 9846, comm: irqbalance Tainted: G N 3.0.76-0.11-default #1 Bochs Bochs
[71428.004077] EIP: 0060:[<c02fceec>] EFLAGS: 00010006 CPU: 2
[71428.004077] EIP is at __get_vm_area_node+0x18c/0x190
[71428.004077] EAX: f2680000 EBX: 00001000 ECX: 00000022 EDX: 00000001
[71428.004077] ESI: ff7fe000 EDI: 00000001 EBP: 00000022 ESP: f2681a54
[71428.004077] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
[71428.004077] Process irqbalance (pid: 9846, ti=f2680000 task=f2406670 task.ti=f2680000)
[71428.004077] Stack:
[71428.004077] c03fcd44 00000001 ffffffff 00001000 ff7fe000 000000d2 00000000 c02fd15c
[71428.004077] f7ffe000 ff7fe000 ffffffff 000000d2 c044ee36 00000163 ff7fe000 c02fd239
[71428.004077] ff7fe000 000000d2 00000163 ffffffff c044ee36 f38fa920 f4010000 f38f3000
[71428.004077] Call Trace:
[71428.004077] [<c02fd15c>] __vmalloc_node_range+0x6c/0xf0
[71428.004077] [<c02fd239>] __vmalloc_node+0x59/0x70
[71428.004077] [<c02fd579>] vmalloc+0x29/0x30
[71428.004077] [<c044ee36>] splash_prepare+0x136/0x560
[71428.004077] [<c044441a>] fbcon_switch+0x4a/0x530
[71428.004077] [<c04a945b>] redraw_screen+0x11b/0x280
[71428.004077] [<c04442c9>] fbcon_blank+0x1c9/0x2d0
[71428.004077] [<c04aa7ab>] do_unblank_screen+0x9b/0x1a0
[71428.004077] [<c0401a25>] bust_spinlocks+0x15/0x40
[71428.004077] [<c05ca654>] panic+0xb7/0x196
[71428.004077] [<c02a3d8e>] watchdog_overflow_callback+0xae/0xb0
[71428.004077] [<c02cba77>] __perf_event_overflow+0xc7/0x290
[71428.004077] [<c02cc302>] perf_event_overflow+0x12/0x20
[71428.004077] [<c0214c29>] intel_pmu_handle_irq+0x199/0x320
[71428.004077] [<c05cef56>] perf_event_nmi_handler+0x26/0x80
[71428.004077] [<c05d0a96>] notifier_call_chain+0x36/0x60
[71428.004077] [<c05d0adb>] __atomic_notifier_call_chain+0x1b/0x20
[71428.004077] [<c05d0af7>] atomic_notifier_call_chain+0x17/0x20
[71428.004077] [<c05d0b30>] notify_die+0x30/0x40
[71428.004077] [<c05ce4a0>] default_do_nmi+0x30/0x280
[71428.004077] [<c05ce74f>] do_nmi+0x5f/0x70
[71428.004077] [<c05cdeb9>] nmi_stack_correct+0x28/0x2d
[71428.004077] [<c05ccebe>] _raw_spin_lock_irqsave+0x2e/0x50
[71428.004077] [<c02a76df>] show_interrupts+0xff/0x2f0
[71428.004077] [<c0337cd0>] seq_read+0x150/0x360
[71428.004077] [<c0366c50>] proc_reg_read+0x60/0x90
[71428.004077] [<c031b9fb>] vfs_read+0x9b/0x110
[71428.004077] [<c031bb41>] sys_read+0x41/0x80
[71428.004077] [<c05d401c>] sysenter_do_call+0x12/0x28
[71428.004077] [<ffffe430>] 0xffffe42f
[71428.004077] Code: 85 c0 75 f3 89 03 89 1a f0 81 05 54 6f 85 c0 00 00 00 01 83 c4 0c 89 d8 5b 5e 5f 5d c3 89 d8 31 db e8 e9 e7 00
00 e9 4e ff ff ff <0f> 0b eb fe 53 89 d3 83 ec 14 8b 15 88 df 84 c0 89 4c 24 10 89
[71428.004077] EIP: [<c02fceec>] __get_vm_area_node+0x18c/0x190 SS:ESP 0068:f2681a54
[71428.004077] ---[ end trace a538072f67b7bdaa ]---
[71428.004077] Kernel panic - not syncing: Fatal exception in interrupt
[71428.004077] Pid: 9846, comm: irqbalance Tainted: G D N 3.0.76-0.11-default #1
[71428.004077] Call Trace:
[71428.004077] [<c0205741>] try_stack_unwind+0x1a1/0x1b0
[71428.004077] [<c0204727>] dump_trace+0x47/0x110
[71428.004077] [<c02052cb>] show_trace_log_lvl+0x4b/0x60
[71428.004077] [<c02052f8>] show_trace+0x18/0x20
[71428.004077] [<c05ca598>] dump_stack+0x6d/0x72
[71428.004077] [<c05ca5f4>] panic+0x57/0x196
[71428.004077] [<c05ceacc>] oops_end+0xbc/0xd0
[71428.004077] [<c020322f>] do_invalid_op+0x7f/0x90
[71428.004077] [<c05cde0a>] error_code+0x5a/0x60
[71428.004077] [<c02fceec>] __get_vm_area_node+0x18c/0x190
[71428.004077] [<c02fd15c>] __vmalloc_node_range+0x6c/0xf0
[71428.004077] [<c02fd239>] __vmalloc_node+0x59/0x70
[71428.004077] [<c02fd579>] vmalloc+0x29/0x30
[71428.004077] [<c044ee36>] splash_prepare+0x136/0x560
[71428.004077] [<c044441a>] fbcon_switch+0x4a/0x530
[71428.004077] [<c04a945b>] redraw_screen+0x11b/0x280
[71428.004077] [<c04442c9>] fbcon_blank+0x1c9/0x2d0
[71428.004077] [<c04aa7ab>] do_unblank_screen+0x9b/0x1a0
[71428.004077] [<c0401a25>] bust_spinlocks+0x15/0x40
[71428.004077] [<c05ca654>] panic+0xb7/0x196
[71428.004077] [<c02a3d8e>] watchdog_overflow_callback+0xae/0xb0
[71428.004077] [<c02cba77>] __perf_event_overflow+0xc7/0x290
[71428.004077] [<c02cc302>] perf_event_overflow+0x12/0x20
[71428.004077] [<c0214c29>] intel_pmu_handle_irq+0x199/0x320
[71428.004077] [<c05cef56>] perf_event_nmi_handler+0x26/0x80
[71428.004077] [<c05d0a96>] notifier_call_chain+0x36/0x60
[71428.004077] [<c05d0adb>] __atomic_notifier_call_chain+0x1b/0x20
[71428.004077] [<c05d0af7>] atomic_notifier_call_chain+0x17/0x20
[71428.004077] [<c05d0b30>] notify_die+0x30/0x40
[71428.004077] [<c05ce4a0>] default_do_nmi+0x30/0x280
[71428.004077] [<c05ce74f>] do_nmi+0x5f/0x70
[71428.004077] [<c05cdeb9>] nmi_stack_correct+0x28/0x2d
[71428.004077] [<c05ccebe>] _raw_spin_lock_irqsave+0x2e/0x50
[71428.004077] [<c02a76df>] show_interrupts+0xff/0x2f0
[71428.004077] [<c0337cd0>] seq_read+0x150/0x360
[71428.004077] [<c0366c50>] proc_reg_read+0x60/0x90
[71428.004077] [<c031b9fb>] vfs_read+0x9b/0x110
[71428.004077] [<c031bb41>] sys_read+0x41/0x80
[71428.004077] [<c05d401c>] sysenter_do_call+0x12/0x28
[71428.004077] [<ffffe430>] 0xffffe42f
[71418.009628] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 5
[71418.009628] Pid: 0, comm: kworker/0:1 Tainted: G D N 3.0.76-0.11-default #1
[71418.009628] Call Trace:
[71418.009628] [<c0205741>] try_stack_unwind+0x1a1/0x1b0
[71418.009628] [<c0204727>] dump_trace+0x47/0x110
[71418.009628] [<c02052cb>] show_trace_log_lvl+0x4b/0x60
[71418.009628] [<c02052f8>] show_trace+0x18/0x20
[71418.009628] [<c05ca598>] dump_stack+0x6d/0x72
[71418.009628] [<c05ca5f4>] panic+0x57/0x196
[71418.009628] [<c02a3d8e>] watchdog_overflow_callback+0xae/0xb0
[71418.009628] [<c02cba77>] __perf_event_overflow+0xc7/0x290
[71418.009628] [<c02cc302>] perf_event_overflow+0x12/0x20
[71418.009628] [<c0214c29>] intel_pmu_handle_irq+0x199/0x320
[71418.009628] [<c05cef56>] perf_event_nmi_handler+0x26/0x80
[71418.009628] [<c05d0a96>] notifier_call_chain+0x36/0x60
[71418.009628] [<c05d0adb>] __atomic_notifier_call_chain+0x1b/0x20
[71418.009628] [<c05d0af7>] atomic_notifier_call_chain+0x17/0x20
[71418.009628] [<c05d0b30>] notify_die+0x30/0x40
[71418.009628] [<c05ce4a0>] default_do_nmi+0x30/0x280
[71418.009628] [<c05ce74f>] do_nmi+0x5f/0x70
[71418.009628] [<c05cdeb9>] nmi_stack_correct+0x28/0x2d
[71418.009628] [<c05cce80>] _raw_spin_lock+0x10/0x20
[71418.009628] [<c02a4925>] handle_irq_event+0x35/0x50
[71418.009628] [<c02a6ec4>] handle_edge_irq+0x54/0xf0
[71418.009628] [<c02042e1>] handle_irq+0x61/0x90
[71418.009628] [<00000005>] 0x4

crash> bt -a
PID: 0 TASK: c08464a0 CPU: 0 COMMAND: "swapper"
#0 [c083bf30] __schedule at c05caf78
#1 [c083bfd8] cpu_idle at c0201e54

PID: 25972 TASK: f2f94ed0 CPU: 1 COMMAND: "kworker/1:0"
#0 [f7235d30] stop_this_cpu at c02095d9
#1 [f7235d40] reboot_interrupt at c05cd475
EAX: 00000296 EBX: 00000000 ECX: f539aa84 EDX: 00000296 EBP: 00000296
DS: 007b ESI: f25c53c0 ES: 007b EDI: f3080000 GS: 0000
CS: 0060 EIP: c05ccf06 ERR: ffffff07 EFLAGS: 00000296
#2 [f7235d74] _raw_spin_unlock_irqrestore at c05ccf06
#3 [f7235d80] ata_scsi_queuecmd at f82d7627 [libata]
#4 [f7235d94] scsi_dispatch_cmd at f829bcf6 [scsi_mod]
#5 [f7235db4] scsi_request_fn at f82a2ab4 [scsi_mod]
#6 [f7235de8] __blk_run_queue at c03cc7c3
#7 [f7235df0] blk_execute_rq_nowait at c03d2591
#8 [f7235e04] blk_execute_rq at c03d2658
#9 [f7235e90] scsi_execute at f82a3e63 [scsi_mod]
#10 [f7235eb0] scsi_execute_req at f82a4026 [scsi_mod]
#11 [f7235ee8] sr_check_events at f94f8315 [sr_mod]
#12 [f7235f38] cdrom_check_events at f951a043 [cdrom]
#13 [f7235f44] disk_events_workfn at c03d5865
#14 [f7235f68] process_one_work at c0262a95
#15 [f7235f9c] worker_thread at c026521d
#16 [f7235fc0] kthread at c0268b12
#17 [f7235fe8] kernel_thread_helper at c05d46a4

PID: 9846 TASK: f2406670 CPU: 2 COMMAND: "irqbalance"
#0 [f26818f8] crash_kexec at c028f7d6
#1 [f268194c] panic at c05ca696
#2 [f2681968] oops_end at c05ceac7
#3 [f268197c] do_invalid_op at c020322a
#4 [f2681a14] error_code (via invalid_op) at c05cde08
EAX: f2680000 EBX: 00001000 ECX: 00000022 EDX: 00000001 EBP: 00000022
DS: 007b ESI: ff7fe000 ES: 007b EDI: 00000001 GS: 31b0
CS: 0060 EIP: c02fceec ERR: ffffffff EFLAGS: 00010006
#5 [f2681a48] __get_vm_area_node at c02fceec
#6 [f2681a70] __vmalloc_node_range at c02fd157
#7 [f2681a90] __vmalloc_node at c02fd234
#8 [f2681ab4] vmalloc at c02fd574
#9 [f2681ac4] splash_prepare at c044ee31
#10 [f2681b14] fbcon_switch at c0444415
#11 [f2681be4] redraw_screen at c04a9458
#12 [f2681c00] fbcon_blank at c04442c4
#13 [f2681cd4] do_unblank_screen at c04aa7a8
#14 [f2681cec] bust_spinlocks at c0401a20
#15 [f2681cf0] panic at c05ca64f
#16 [f2681d0c] watchdog_overflow_callback at c02a3d89
#17 [f2681d20] __perf_event_overflow at c02cba75
#18 [f2681d6c] perf_event_overflow at c02cc2fd
#19 [f2681d74] intel_pmu_handle_irq at c0214c24
#20 [f2681e20] perf_event_nmi_handler at c05cef50
#21 [f2681e24] notifier_call_chain at c05d0a94
#22 [f2681e40] __atomic_notifier_call_chain at c05d0ad6
#23 [f2681e50] atomic_notifier_call_chain at c05d0af2
#24 [f2681e5c] notify_die at c05d0b2b
#25 [f2681e74] default_do_nmi at c05ce49b
#26 [f2681e9c] do_nmi at c05ce74a
#27 [f2681ea4] nmi at c05cdeb4
EAX: 00000296 EBX: ffffffff ECX: 00005150 EDX: 0000003e EBP: f345da40
DS: 007b ESI: 00000296 ES: 007b EDI: f364b80c GS: 0000
CS: 0060 EIP: c05ccebe ERR: 00000296 EFLAGS: 00000097
#28 [f2681ed8] _raw_spin_lock_irqsave at c05ccebe
#29 [f2681ef0] show_interrupts at c02a76da
#30 [f2681f24] seq_read at c0337ccd
#31 [f2681f5c] proc_reg_read at c0366c4e
#32 [f2681f7c] vfs_read at c031b9f9
#33 [f2681f94] sys_read at c031bb3c
#34 [f2681fb0] ia32_sysenter_target at c05d4015
EAX: 00000003 EBX: 00000003 ECX: b76f3000 EDX: 00000400
DS: 007b ESI: b7722490 ES: 007b EDI: b76f3000
SS: 007b ESP: bfa64bbc EBP: bfa64bf4 GS: 0000
CS: 0073 EIP: ffffe430 ERR: 00000003 EFLAGS: 00000246

PID: 0 TASK: f40bc2b0 CPU: 3 COMMAND: "kworker/0:1"
#0 [f40e1f40] stop_this_cpu at c02095d9
#1 [f40e1f50] reboot_interrupt at c05cd475
EAX: 00000000 EBX: 00000003 ECX: f53b2700 EDX: 00000000 EBP: 00000000
DS: 007b ESI: c0885888 ES: 007b EDI: 00000003 GS: 0000
CS: 0060 EIP: c022a6c2 ERR: ffffff07 EFLAGS: 00000246
#2 [f40e1f84] native_safe_halt at c022a6c2
#3 [f40e1f90] default_idle at c0209dd0
#4 [f40e1fa4] cpu_idle at c0201e54

PID: 0 TASK: f40f5010 CPU: 4 COMMAND: "kworker/0:1"
#0 [f40f7f40] stop_this_cpu at c02095d9
#1 [f40f7f50] reboot_interrupt at c05cd475
EAX: 00000000 EBX: 00000004 ECX: f53be700 EDX: 00000000 EBP: 00000000
DS: 007b ESI: c0885888 ES: 007b EDI: 00000004 GS: 0000
CS: 0060 EIP: c022a6c2 ERR: ffffff07 EFLAGS: 00000246
#2 [f40f7f84] native_safe_halt at c022a6c2
#3 [f40f7f90] default_idle at c0209dd0
#4 [f40f7fa4] cpu_idle at c0201e54

PID: 0 TASK: f41063f0 CPU: 5 COMMAND: "kworker/0:1"
#0 [f410befc] __schedule at c05caf78
#1 [f410bfa4] cpu_idle at c0201e54
crash>

Best regards,
-Gonglei

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Philipp Reisner: "Re: [Drbd-dev] [PATCH 15/20] drbd: cosmetic: change all printk(level, ...) to pr_<level>(...)"
Previous message: Dmitry Kasatkin: "IMA: kernel reading files opened with O_DIRECT"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]