Re: Deadlock?

From: Jack Stone
Date: Mon Feb 13 2012 - 12:37:37 EST

Next message: Casey Schaufler: "Re: [PATCH 0/5] IPC: checkpoint/restore in userspace enhancements"
Previous message: Johannes Berg: "Re: [PATCH v3] printk: add console output tracing"
In reply to: Markus: "Deadlock?"
Next in thread: Jan Kara: "Re: Deadlock?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Adding CCs

On 02/13/2012 11:17 AM, Markus wrote:
> Hi!
>
> I noted some kind of deadlock, where I was not able to write to the raid6,
> while writing to each disk would still work.
> This caused many processes to "wait" in d-state, thus making it impossible to
> unmount, cleanly reboot, sync, ...
>
> So I enabled the detection of hung tasks and this deplock option.
> After running a 3.2.2 for about 5 days:
> http://pastebin.com/gy1kaYmS
>
> I dont know if its a bug or nothing. Or if it has anything to do with my
> problem as there was no hungtask detected and the raid still seems to work.
>
> Can anybody enlight me?! ;)
>
> Markus

[493061.240015] [ INFO: RECLAIM_FS-safe -> RECLAIM_FS-unsafe lock order detected ]
[493061.240015] 3.2.2 #1
[493061.240015] ------------------------------------------------------
[493061.240015] linuxdcpp/11649 [HC0[0]:SC0[0]:HE1:SE1] is trying to acquire:
[493061.240015] (&tty->atomic_write_lock){+.+.+.}, at: [<ffffffff811bf40a>] tty_write_message+0x2a/0x80
[493061.240015]
[493061.240015] and this task is already holding:
[493061.240015] (&s->s_dquot.dqptr_sem){++++-.}, at: [<ffffffff8111888d>] __dquot_alloc_space+0x9d/0x230
[493061.240015] which would create a new lock dependency:
[493061.240015] (&s->s_dquot.dqptr_sem){++++-.} -> (&tty->atomic_write_lock){+.+.+.}
[493061.240015]
[493061.240015] but this new dependency connects a RECLAIM_FS-irq-safe lock:
[493061.240015] (&s->s_dquot.dqptr_sem){++++-.}
[493061.240015] ... which became RECLAIM_FS-irq-safe at:
[493061.240015] [<ffffffff81069914>] __lock_acquire+0x6e4/0x1eb0
[493061.240015] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493061.240015] [<ffffffff81375e4c>] down_write+0x2c/0x50
[493061.240015] [<ffffffff81119c97>] __dquot_drop+0x27/0x80
[493061.240015] [<ffffffff81119d15>] dquot_drop+0x25/0x30
[493061.240015] [<ffffffff81138278>] ext3_evict_inode+0x158/0x260
[493061.240015] [<ffffffff810e0167>] evict+0xa7/0x1b0
[493061.240015] [<ffffffff810e0807>] dispose_list+0x47/0x60
[493061.240015] [<ffffffff810e1222>] prune_icache_sb+0x192/0x370
[493061.240015] [<ffffffff810c9e03>] prune_super+0x153/0x1b0
[493061.240015] [<ffffffff8109c355>] shrink_slab+0x135/0x1f0
[493061.240015] [<ffffffff8109e932>] kswapd+0x632/0x8e0
[493061.240015] [<ffffffff81055ce6>] kthread+0x96/0xa0
[493061.240015] [<ffffffff81378eb4>] kernel_thread_helper+0x4/0x10
[493061.240015]
[493061.240015] to a RECLAIM_FS-irq-unsafe lock:
[493061.240015] (&tty->atomic_write_lock){+.+.+.}
[493061.240015] ... which became RECLAIM_FS-irq-unsafe at:
[493061.240015] ... [<ffffffff810681f6>] mark_held_locks+0x76/0x150
[493061.240015] [<ffffffff81068ab0>] lockdep_trace_alloc+0xb0/0xe0
[493061.240015] [<ffffffff810c2a78>] __kmalloc+0x78/0x160
[493061.240015] [<ffffffff811bf126>] tty_write+0x136/0x270
[493061.240015] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0
[493061.240015] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493061.240015] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493061.240015] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493061.240015]
[493061.240015] other info that might help us debug this:
[493061.240015]
[493061.240015] Possible interrupt unsafe locking scenario:
[493061.240015]
[493061.240015] CPU0 CPU1
[493061.240015] ---- ----
[493061.240015] lock(&tty->atomic_write_lock);
[493061.240015] local_irq_disable();
[493061.240015] lock(&s->s_dquot.dqptr_sem);
[493061.240015] lock(&tty->atomic_write_lock);
[493061.240015] <Interrupt>
[493061.240015] lock(&s->s_dquot.dqptr_sem);
[493061.240015]
[493061.240015] *** DEADLOCK ***
[493061.240015]
[493061.240015] 4 locks held by linuxdcpp/11649:
[493063.862237] #0: (&sb->s_type->i_mutex_key#3){+.+.+.}, at: [<ffffffff81090947>] generic_file_aio_write+0x57/0xe0
[493063.862237] #1: (jbd_handle){+.+.-.}, at: [<ffffffff81152098>] start_this_handle+0x358/0x450
[493063.862237] #2: (&ei->truncate_mutex){+.+...}, at: [<ffffffff811367c5>] ext3_get_blocks_handle+0xe5/0xbf0
[493063.862237] #3: (&s->s_dquot.dqptr_sem){++++-.}, at: [<ffffffff8111888d>] __dquot_alloc_space+0x9d/0x230
[493063.862237]
[493063.862237] the dependencies between RECLAIM_FS-irq-safe lock and the holding lock:
[493063.862237] -> (&s->s_dquot.dqptr_sem){++++-.} ops: 187112656 {
[493063.862237] HARDIRQ-ON-W at:
[493063.862237] [<ffffffff810699c5>] __lock_acquire+0x795/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50
[493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80
[493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510
[493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80
[493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130
[493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560
[493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] HARDIRQ-ON-R at:
[493063.862237] [<ffffffff8106988d>] __lock_acquire+0x65d/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e9f>] down_read+0x2f/0x50
[493063.862237] [<ffffffff8111bac3>] dquot_alloc_inode+0x43/0x140
[493063.862237] [<ffffffff8113354b>] ext3_new_inode+0x89b/0x980
[493063.862237] [<ffffffff8113b3fd>] ext3_create+0x8d/0x110
[493063.862237] [<ffffffff810d26fc>] vfs_create+0xac/0xe0
[493063.862237] [<ffffffff810d4149>] do_last.clone.30+0x469/0x7d0
[493063.862237] [<ffffffff810d5b50>] path_openat+0xd0/0x410
[493063.862237] [<ffffffff810d5fa4>] do_filp_open+0x44/0xa0
[493063.862237] [<ffffffff810c5f9c>] do_sys_open+0xfc/0x1e0
[493063.862237] [<ffffffff810c609b>] sys_open+0x1b/0x20
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] SOFTIRQ-ON-W at:
[493063.862237] [<ffffffff810699ff>] __lock_acquire+0x7cf/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50
[493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80
[493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510
[493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80
[493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130
[493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560
[493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] SOFTIRQ-ON-R at:
[493063.862237] [<ffffffff810699ff>] __lock_acquire+0x7cf/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e9f>] down_read+0x2f/0x50
[493063.862237] [<ffffffff8111bac3>] dquot_alloc_inode+0x43/0x140
[493063.862237] [<ffffffff8113354b>] ext3_new_inode+0x89b/0x980
[493063.862237] [<ffffffff8113b3fd>] ext3_create+0x8d/0x110
[493063.862237] [<ffffffff810d26fc>] vfs_create+0xac/0xe0
[493063.862237] [<ffffffff810d4149>] do_last.clone.30+0x469/0x7d0
[493063.862237] [<ffffffff810d5b50>] path_openat+0xd0/0x410
[493063.862237] [<ffffffff810d5fa4>] do_filp_open+0x44/0xa0
[493063.862237] [<ffffffff810c5f9c>] do_sys_open+0xfc/0x1e0
[493063.862237] [<ffffffff810c609b>] sys_open+0x1b/0x20
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] IN-RECLAIM_FS-W at:
[493063.862237] [<ffffffff81069914>] __lock_acquire+0x6e4/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50
[493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80
[493063.862237] [<ffffffff81119d15>] dquot_drop+0x25/0x30
[493063.862237] [<ffffffff81138278>] ext3_evict_inode+0x158/0x260
[493063.862237] [<ffffffff810e0167>] evict+0xa7/0x1b0
[493063.862237] [<ffffffff810e0807>] dispose_list+0x47/0x60
[493063.862237] [<ffffffff810e1222>] prune_icache_sb+0x192/0x370
[493063.862237] [<ffffffff810c9e03>] prune_super+0x153/0x1b0
[493063.862237] [<ffffffff8109c355>] shrink_slab+0x135/0x1f0
[493063.862237] [<ffffffff8109e932>] kswapd+0x632/0x8e0
[493063.862237] [<ffffffff81055ce6>] kthread+0x96/0xa0
[493063.862237] [<ffffffff81378eb4>] kernel_thread_helper+0x4/0x10
[493063.862237] INITIAL USE at:
[493063.862237] [<ffffffff81069619>] __lock_acquire+0x3e9/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81375e4c>] down_write+0x2c/0x50
[493063.862237] [<ffffffff81119c97>] __dquot_drop+0x27/0x80
[493063.862237] [<ffffffff8111ac23>] vfs_load_quota_inode+0x4c3/0x510
[493063.862237] [<ffffffff8111aed0>] dquot_quota_on+0x70/0x80
[493063.862237] [<ffffffff8113dadb>] ext3_quota_on+0xfb/0x130
[493063.862237] [<ffffffff8111c726>] do_quotactl+0x536/0x560
[493063.862237] [<ffffffff8111c81e>] sys_quotactl+0xce/0x1a0
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] }
[493063.862237] ... key at: [<ffffffff81f48938>] __key.28495+0x0/0x8
[493063.862237] ... acquired at:
[493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0
[493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300
[493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80
[493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200
[493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230
[493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680
[493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0
[493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120
[493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540
[493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200
[493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280
[493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430
[493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0
[493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237]
[493063.862237]
[493063.862237] the dependencies between the lock to be acquired and RECLAIM_FS-irq-unsafe lock:
[493063.862237] -> (&tty->atomic_write_lock){+.+.+.} ops: 270462830 {
[493063.862237] HARDIRQ-ON-W at:
[493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150
[493063.862237] [<ffffffff8106837d>] trace_hardirqs_on_caller+0xad/0x1e0
[493063.862237] [<ffffffff810684bd>] trace_hardirqs_on+0xd/0x10
[493063.862237] [<ffffffff81374cbd>] mutex_trylock+0xfd/0x170
[493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60
[493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270
[493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] SOFTIRQ-ON-W at:
[493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150
[493063.862237] [<ffffffff810683ed>] trace_hardirqs_on_caller+0x11d/0x1e0
[493063.862237] [<ffffffff810684bd>] trace_hardirqs_on+0xd/0x10
[493063.862237] [<ffffffff81374cbd>] mutex_trylock+0xfd/0x170
[493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60
[493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270
[493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] RECLAIM_FS-ON-W at:
[493063.862237] [<ffffffff810681f6>] mark_held_locks+0x76/0x150
[493063.862237] [<ffffffff81068ab0>] lockdep_trace_alloc+0xb0/0xe0
[493063.862237] [<ffffffff810c2a78>] __kmalloc+0x78/0x160
[493063.862237] [<ffffffff811bf126>] tty_write+0x136/0x270
[493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] INITIAL USE at:
[493063.862237] [<ffffffff81069619>] __lock_acquire+0x3e9/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81374c6d>] mutex_trylock+0xad/0x170
[493063.862237] [<ffffffff811befb3>] tty_write_lock+0x23/0x60
[493063.862237] [<ffffffff811bf0c3>] tty_write+0xd3/0x270
[493063.862237] [<ffffffff811bf30d>] redirected_tty_write+0xad/0xb0
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237] }
[493063.862237] ... key at: [<ffffffff81f549b8>] __key.27415+0x0/0x8
[493063.862237] ... acquired at:
[493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0
[493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300
[493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80
[493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200
[493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230
[493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680
[493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0
[493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120
[493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540
[493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200
[493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280
[493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430
[493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0
[493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b
[493063.862237]
[493063.862237]
[493063.862237] stack backtrace:
[493063.862237] Pid: 11649, comm: linuxdcpp Not tainted 3.2.2 #1
[493063.862237] Call Trace:
[493063.862237] [<ffffffff8106774a>] check_usage+0x4aa/0x4b0
[493063.862237] [<ffffffff810050f4>] ? print_context_stack+0x74/0xd0
[493063.862237] [<ffffffff810677b0>] check_irq_usage+0x60/0xf0
[493063.862237] [<ffffffff8106a244>] __lock_acquire+0x1014/0x1eb0
[493063.862237] [<ffffffff8100e6fa>] ? save_stack_trace+0x2a/0x50
[493063.862237] [<ffffffff8106b6ad>] lock_acquire+0x8d/0xb0
[493063.862237] [<ffffffff811bf40a>] ? tty_write_message+0x2a/0x80
[493063.862237] [<ffffffff81374d6b>] mutex_lock_nested+0x3b/0x300
[493063.862237] [<ffffffff811bf40a>] ? tty_write_message+0x2a/0x80
[493063.862237] [<ffffffff811bf40a>] tty_write_message+0x2a/0x80
[493063.862237] [<ffffffff81118377>] flush_warnings+0xe7/0x200
[493063.862237] [<ffffffff8111898a>] __dquot_alloc_space+0x19a/0x230
[493063.862237] [<ffffffff81130f4c>] ext3_new_blocks+0x6c/0x680
[493063.862237] [<ffffffff81374f8f>] ? mutex_lock_nested+0x25f/0x300
[493063.862237] [<ffffffff81374f7b>] ? mutex_lock_nested+0x24b/0x300
[493063.862237] [<ffffffff811369d5>] ext3_get_blocks_handle+0x2f5/0xbf0
[493063.862237] [<ffffffff8106973a>] ? __lock_acquire+0x50a/0x1eb0
[493063.862237] [<ffffffff8106973a>] ? __lock_acquire+0x50a/0x1eb0
[493063.862237] [<ffffffff810f2c2b>] ? create_empty_buffers+0x4b/0xd0
[493063.862237] [<ffffffff8113738f>] ext3_get_block+0xbf/0x120
[493063.862237] [<ffffffff810f44bb>] __block_write_begin+0x1db/0x540
[493063.862237] [<ffffffff811372d0>] ? ext3_get_blocks_handle+0xbf0/0xbf0
[493063.862237] [<ffffffff8113637f>] ext3_write_begin+0xaf/0x200
[493063.862237] [<ffffffff8108ea40>] generic_file_buffered_write+0x110/0x280
[493063.862237] [<ffffffff8103f221>] ? current_fs_time+0x11/0x50
[493063.862237] [<ffffffff810906e1>] __generic_file_aio_write+0x221/0x430
[493063.862237] [<ffffffff81090963>] generic_file_aio_write+0x73/0xe0
[493063.862237] [<ffffffff810c62ca>] do_sync_write+0xda/0x120
[493063.862237] [<ffffffff810a850b>] ? might_fault+0x3b/0x90
[493063.862237] [<ffffffff81167287>] ? security_file_permission+0x27/0xb0
[493063.862237] [<ffffffff810c6b76>] vfs_write+0xc6/0x180
[493063.862237] [<ffffffff810c6e8c>] sys_write+0x4c/0x90
[493063.862237] [<ffffffff81377b7b>] system_call_fastpath+0x16/0x1b

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Casey Schaufler: "Re: [PATCH 0/5] IPC: checkpoint/restore in userspace enhancements"
Previous message: Johannes Berg: "Re: [PATCH v3] printk: add console output tracing"
In reply to: Markus: "Deadlock?"
Next in thread: Jan Kara: "Re: Deadlock?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]