System deadlock on 2.6.24-rt20

From: Remy Bohmer
Date: Fri Apr 03 2009 - 05:49:14 EST

Next message: Jens Axboe: "Re: [PATCH] osdblk: a Linux block device for OSD objects"
Previous message: Peter Zijlstra: "Re: [Announce] 2.6.29-rt2"
Next in thread: Thomas Gleixner: "Re: System deadlock on 2.6.24-rt20"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hello All,

We encountered a nasty system lockup on an X86 system running
2.6.24.7-rt20. (Intel Core2Duo system)
What happens is that the system freezes completely after some time.
After a lot of testing we are now able to reproduce this problem
within 5 minutes of testing.

The backtrace was not printed to the serial terminal, but when we
accidentally used the VGA
terminal the backtrace was there. So we used/hacked the earlyprintk
mechanism to get the logging out.

Does anybody recognise these backtraces? And if so, is there a fix for
it already?

There are several independent processes involved, all with RT-priorities.

Kind regards,

Remy

============= Trace 1 ========================
<0>BUG: spinlock lockup on CPU#0, A_EngineNode/2487, f78c5004
<0>BUG: spinlock lockup on CPU#1, A_EngineNode/2296, c3e13400
<4>Pid: 2487, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4> [<c010657a>] Pid: 2296, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4> [<c010657a>] show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] show_trace+0x12/0x20
<4> [<c010782c>] show_trace+0x12/0x20
<4> [<c010782c>] dump_stack+0x6c/0x80
<4> [<c04c2511>] dump_stack+0x6c/0x80
<4>_raw_spin_lock+0x141/0x180
<4> [<c04c2511>] [<c04c16da>] _raw_spin_lock+0x141/0x180
<4> [<c04c19cd>] __spin_lock_irqsave+0x4a/0x60
<4> [<c016e5ea>] __spin_lock+0x3d/0x50
<4> [<c01270bc>] cpupri_set+0x4a/0xd0
<4> [<c0129f18>] task_rq_lock+0x3c/0x70
<4> [<c01273ac>] dequeue_task_rt+0xd8/0x1f0
<4> [<c01247f5>] try_to_wake_up+0x2c/0x110
<4> [<c01274a8>] dequeue_task+0x15/0x30
<4> [<c0124878>] default_wake_function+0x18/0x20
<4> [<c0124bd8>] deactivate_task+0x28/0x40
<4> [<c04bee0f>] __wake_up_common+0x48/0x70
<4> [<c012ba93>] __schedule+0x2ff/0x780
<4> [<c04bf3d3>] __wake_up+0x33/0x60
<4> [<c015380e>] schedule+0x33/0x120
<4> [<c015369f>] wake_futex+0x3e/0x50
<4>futex_wait+0x26f/0x3a0
<4> [<c015490c>] [<c01538b8>] futex_wake+0x98/0x100
<4>do_futex+0x8c/0xb40
<4> [<c0155445>] [<c0154c74>] do_futex+0x3f4/0xb40
<4>sys_futex+0x85/0x100
<4> [<c010547e>] [<c0155445>] sys_futex+0x85/0x100
<4>sysenter_past_esp+0x5f/0x99
<4> =======================
<4> [<c010547e>] sysenter_past_esp+0x5f/0x99
<4> =======================

============= Trace 2 ========================
<0>BUG: spinlock lockup on CPU#0, A_EngineNode/3069, f78c5004
<4>Pid: 3069, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4> [<c010657a>] show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] show_trace+0x12/0x20
<4> [<c010782c>] dump_stack+0x6c/0x80
<4> [<c04c2511>] _raw_spin_lock+0x141/0x180
<0>BUG: spinlock lockup on CPU#1, usbpui/2208, c3e13400
<4>Pid: 2208, comm: usbpui Not tainted 2.6.24.7-rt20 #4
<4> [<c04c16da>] [<c010657a>] __spin_lock_irqsave+0x4a/0x60
<4> [<c016e5ea>] show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] cpupri_set+0x4a/0xd0
<4>show_trace+0x12/0x20
<4> [<c010782c>] [<c0129f18>] dump_stack+0x6c/0x80
<4> [<c04c2511>] dequeue_task_rt+0xd8/0x1f0
<4> [<c01247f5>] _raw_spin_lock+0x141/0x180
<4>dequeue_task+0x15/0x30
<4> [<c0124878>] [<c04c19cd>] __spin_lock+0x3d/0x50
<4>deactivate_task+0x28/0x40
<4> [<c01270bc>] [<c04bee0f>] task_rq_lock+0x3c/0x70
<4> [<c01273ac>] __schedule+0x2ff/0x780
<4>try_to_wake_up+0x2c/0x110
<4> [<c04bf3d3>] [<c01274a8>] schedule+0x33/0x120
<4> [<c015369f>] default_wake_function+0x18/0x20
<4> [<c0124bd8>] futex_wait+0x26f/0x3a0
<4> [<c015490c>] __wake_up_common+0x48/0x70
<4>do_futex+0x8c/0xb40
<4> [<c0155445>] [<c012ba93>] __wake_up+0x33/0x60
<4>sys_futex+0x85/0x100
<4> [<c010547e>] [<c015380e>] sysenter_past_esp+0x5f/0x99
<4> =======================
<4>wake_futex+0x3e/0x50
<4> [<c01538b8>] futex_wake+0x98/0x100
<4> [<c0154c74>] do_futex+0x3f4/0xb40
<4> [<c0155445>] sys_futex+0x85/0x100
<4> [<c010547e>] sysenter_past_esp+0x5f/0x99
<4> =======================

============= Trace 3 ========================
<0>BUG: spinlock lockup on CPU#0, A_EngineNode/2525, f78c5004
<0>BUG: spinlock lockup on CPU#1, A_EngineNode/2498, c3e13400
<4>Pid: 2498, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4>Pid: 2525, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4> [<c010657a>] [<c010657a>] show_trace_log_lvl+0x1a/0x30
<4>show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] [<c0107012>] show_trace+0x12/0x20
<4>show_trace+0x12/0x20
<4> [<c010782c>] [<c010782c>] dump_stack+0x6c/0x80
<4>dump_stack+0x6c/0x80
<4> [<c04c2511>] [<c04c2511>] _raw_spin_lock+0x141/0x180
<4> [<c04c19cd>] _raw_spin_lock+0x141/0x180
<4> [<c04c16da>] __spin_lock+0x3d/0x50
<4>__spin_lock_irqsave+0x4a/0x60
<4> [<c016e5ea>] [<c01270bc>] task_rq_lock+0x3c/0x70
<4>cpupri_set+0x4a/0xd0
<4> [<c0129f18>] [<c01273ac>] dequeue_task_rt+0xd8/0x1f0
<4>try_to_wake_up+0x2c/0x110
<4> [<c01274a8>] [<c01247f5>] dequeue_task+0x15/0x30
<4>default_wake_function+0x18/0x20
<4> [<c0124bd8>] [<c0124878>] deactivate_task+0x28/0x40
<4>__wake_up_common+0x48/0x70
<4> [<c012ba93>] [<c04bee0f>] __wake_up+0x33/0x60
<4>__schedule+0x2ff/0x780
<4> [<c04bf3d3>] [<c015380e>] schedule+0x33/0x120
<4> [<c015369f>] wake_futex+0x3e/0x50
<4> [<c01538b8>] futex_wait+0x26f/0x3a0
<4> [<c015490c>] futex_wake+0x98/0x100
<4>do_futex+0x8c/0xb40
<4> [<c0155445>] [<c0154c74>] sys_futex+0x85/0x100
<4>do_futex+0x3f4/0xb40
<4> [<c0155445>] [<c010547e>] sysenter_past_esp+0x5f/0x99
<4>sys_futex+0x85/0x100
<4> [<c010547e>] =======================
<4>sysenter_past_esp+0x5f/0x99
<4> =======================

============= Trace 4 ========================
<0>BUG: spinlock lockup on CPU#0, A_EngineNode/3044, f78c5004
<0>BUG: spinlock lockup on CPU#1, A_EngineNode/3001, c3e13400
<4>Pid: 3044, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4>Pid: 3001, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4
<4> [<c010657a>] [<c010657a>] show_trace_log_lvl+0x1a/0x30
<4>show_trace_log_lvl+0x1a/0x30
<4> [<c0107012>] [<c0107012>] show_trace+0x12/0x20
<4>show_trace+0x12/0x20
<4> [<c010782c>] [<c010782c>] dump_stack+0x6c/0x80
<4>dump_stack+0x6c/0x80
<4> [<c04c2511>] [<c04c2511>] _raw_spin_lock+0x141/0x180
<4>_raw_spin_lock+0x141/0x180
<4> [<c04c16da>] [<c04c19cd>] __spin_lock+0x3d/0x50
<4>__spin_lock_irqsave+0x4a/0x60
<4> [<c01270bc>] [<c016e5ea>] task_rq_lock+0x3c/0x70
<4>cpupri_set+0x4a/0xd0
<4> [<c0129f18>] [<c01273ac>] try_to_wake_up+0x2c/0x110
<4>dequeue_task_rt+0xd8/0x1f0
<4> [<c0127509>] [<c01247f5>] wake_up_process_mutex+0x19/0x20
<4> [<c0155c6c>] dequeue_task+0x15/0x30
<4> [<c0124878>] wakeup_next_waiter+0x11c/0x1c0
<4> [<c04bfdbc>] deactivate_task+0x28/0x40
<4>rt_spin_lock_slowunlock+0x4c/0x70
<4> [<c04c1122>] [<c04bee0f>] rt_spin_unlock+0x52/0x60
<4>__schedule+0x2ff/0x780
<4> [<c04bf3d3>] [<c01538cb>] schedule+0x33/0x120
<4> [<c04c0b7c>] futex_wake+0xab/0x100
<4> [<c0154c74>] rt_spin_lock_slowlock+0x13c/0x1f4
<4> [<c04c101a>] do_futex+0x3f4/0xb40
<4>__lock_text_start+0x2a/0x50
<4> [<c0155445>] [<c04c10c2>] sys_futex+0x85/0x100
<4> [<c010547e>] rt_spin_lock+0x32/0x40
<4> [<c01533a8>] sysenter_past_esp+0x5f/0x99
<4> =======================
<4>unqueue_me+0x18/0xa0
<4> [<c01536da>] futex_wait+0x2aa/0x3a0
<4> [<c015490c>] do_futex+0x8c/0xb40
<4> [<c0155445>] sys_futex+0x85/0x100
<4> [<c010547e>] sysenter_past_esp+0x5f/0x99
<4> =======================
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Jens Axboe: "Re: [PATCH] osdblk: a Linux block device for OSD objects"
Previous message: Peter Zijlstra: "Re: [Announce] 2.6.29-rt2"
Next in thread: Thomas Gleixner: "Re: System deadlock on 2.6.24-rt20"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]