[2.6.21] possible circular locking dependency detected IN FILESYSTEM CODE (ext3)

From: Folkert van Heusden
Date: Fri Apr 27 2007 - 09:39:33 EST


Hi,

Running 2.6.21 on a P4 with HT and 2GB of ram. One filesystem which is
ext3. After 2+ hours of uptime I got the following circular locking
dependency in my dmesg output:

[ 7429.853775] =======================================================
[ 7429.853884] [ INFO: possible circular locking dependency detected ]
[ 7429.853939] 2.6.21 #1
[ 7429.854005] -------------------------------------------------------
[ 7429.854058] moo/5453 is trying to acquire lock:
[ 7429.854112] (tty_mutex){--..}, at: [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.854286]
[ 7429.854287] but task is already holding lock:
[ 7429.854413] (&s->s_dquot.dqptr_sem){----}, at: [<c109e3e4>] dquot_alloc_space+0x50/0x189
[ 7429.854423]
[ 7429.854424] which lock already depends on the new lock.
[ 7429.854426]
[ 7429.854427]
[ 7429.854428] the existing dependency chain (in reverse order) is:
[ 7429.854431]
[ 7429.854432] -> #4 (&s->s_dquot.dqptr_sem){----}:
[ 7429.854437] [<c103983e>] check_prev_add+0x15b/0x281
[ 7429.854445] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.854452] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.854460] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.854468] [<c1033dbf>] down_read+0x2b/0x3d
[ 7429.854477] [<c109e3e4>] dquot_alloc_space+0x50/0x189
[ 7429.854490] [<c10af496>] ext3_new_blocks+0x44b/0x5a2
[ 7429.854497] [<c10b1a28>] ext3_alloc_blocks+0x40/0xdf
[ 7429.854505] [<c10b1b17>] ext3_alloc_branch+0x50/0x21b
[ 7429.854512] [<c10b1ff5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7429.854519] [<c10b231e>] ext3_getblk+0x97/0x228
[ 7429.854530] [<c10b24c9>] ext3_bread+0x1a/0x78
[ 7429.854536] [<c10b760a>] ext3_mkdir+0xf4/0x270
[ 7429.854545] [<c1077cb6>] vfs_mkdir+0xb3/0x161
[ 7429.854553] [<c1077df0>] sys_mkdirat+0x8c/0xc4
[ 7429.854560] [<c1077e48>] sys_mkdir+0x20/0x22
[ 7429.854570] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.854578] [<ffffffff>] 0xffffffff
[ 7429.854600]
[ 7429.854600] -> #3 (&ei->truncate_mutex){--..}:
[ 7429.854608] [<c103983e>] check_prev_add+0x15b/0x281
[ 7429.854616] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.854623] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.854630] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.854637] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.854647] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.854654] [<c10b3b97>] ext3_truncate+0x170/0x468
[ 7429.854661] [<c105dea4>] vmtruncate+0xa6/0x116
[ 7429.854669] [<c108276f>] inode_setattr+0x145/0x16c
[ 7429.854678] [<c10b4ab9>] ext3_setattr+0x150/0x22f
[ 7429.854690] [<c1082ae8>] notify_change+0x352/0x386
[ 7429.854697] [<c106ccf6>] do_truncate+0x52/0x75
[ 7429.854705] [<c10771aa>] may_open+0x1ec/0x231
[ 7429.854713] [<c1077368>] open_namei+0xda/0x59b
[ 7429.854720] [<c106d95e>] do_filp_open+0x2c/0x53
[ 7429.854731] [<c106dc93>] do_sys_open+0x52/0xd8
[ 7429.854738] [<c106dd35>] sys_open+0x1c/0x1e
[ 7429.854745] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.854752] [<ffffffff>] 0xffffffff
[ 7429.854760]
[ 7429.854761] -> #2 (&inode->i_alloc_sem){--..}:
[ 7429.854770] [<c103983e>] check_prev_add+0x15b/0x281
[ 7429.854777] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.854784] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.854792] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.854798] [<c1033e55>] down_write+0x2b/0x45
[ 7429.854808] [<c1082a6f>] notify_change+0x2d9/0x386
[ 7429.854818] [<c106ccf6>] do_truncate+0x52/0x75
[ 7429.854826] [<c10771aa>] may_open+0x1ec/0x231
[ 7429.854834] [<c1077368>] open_namei+0xda/0x59b
[ 7429.854842] [<c106d95e>] do_filp_open+0x2c/0x53
[ 7429.854850] [<c106dc93>] do_sys_open+0x52/0xd8
[ 7429.854860] [<c106dd35>] sys_open+0x1c/0x1e
[ 7429.854867] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.854874] [<ffffffff>] 0xffffffff
[ 7429.854882]
[ 7429.854883] -> #1 (&sysfs_inode_imutex_key){--..}:
[ 7429.854887] [<c103983e>] check_prev_add+0x15b/0x281
[ 7429.854895] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.854902] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.854910] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.854917] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.854924] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.854934] [<c10ac0b7>] remove_dir+0x2c/0xf1
[ 7429.854942] [<c10ac184>] sysfs_remove_subdir+0x8/0xa
[ 7429.854950] [<c10ad58b>] sysfs_remove_group+0x67/0x78
[ 7429.854957] [<c1154f44>] dpm_sysfs_remove+0x12/0x16
[ 7429.854966] [<c11546b3>] device_pm_remove+0x21/0x62
[ 7429.854977] [<c114e700>] device_del+0x89/0x1dd
[ 7429.854984] [<c114e85f>] device_unregister+0xb/0x15
[ 7429.854992] [<c114ea5a>] device_destroy+0x8d/0x9a
[ 7429.854999] [<c113a453>] vcs_remove_sysfs+0x1c/0x39
[ 7429.855007] [<c1140bd9>] con_close+0x5e/0x6b
[ 7429.855017] [<c11329d6>] release_dev+0x4c4/0x6ce
[ 7429.855025] [<c1133059>] tty_release+0x12/0x1c
[ 7429.855033] [<c106f67d>] __fput+0x145/0x16e
[ 7429.855040] [<c106f536>] fput+0x19/0x1b
[ 7429.855046] [<c106ddb6>] filp_close+0x3c/0x75
[ 7429.855056] [<c106de58>] sys_close+0x69/0xb4
[ 7429.855064] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.855070] [<ffffffff>] 0xffffffff
[ 7429.855081]
[ 7429.855082] -> #0 (tty_mutex){--..}:
[ 7429.855085] [<c1039717>] check_prev_add+0x34/0x281
[ 7429.855098] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.855105] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.855112] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.855120] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.855127] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855134] [<c109dd5a>] print_warning+0x8c/0x15d
[ 7429.855144] [<c109e518>] dquot_alloc_space+0x184/0x189
[ 7429.855151] [<c10af496>] ext3_new_blocks+0x44b/0x5a2
[ 7429.855162] [<c10b1a28>] ext3_alloc_blocks+0x40/0xdf
[ 7429.855169] [<c10b1b17>] ext3_alloc_branch+0x50/0x21b
[ 7429.855177] [<c10b1ff5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7429.855184] [<c10b221c>] ext3_get_block+0x78/0xe3
[ 7429.855195] [<c108f265>] __block_prepare_write+0x168/0x408
[ 7429.855202] [<c108fd47>] block_prepare_write+0x28/0x3b
[ 7429.855210] [<c10b26c8>] ext3_prepare_write+0xe3/0x17e
[ 7429.855218] [<c1052a29>] generic_file_buffered_write+0x1b1/0x611
[ 7429.855226] [<c1053137>] __generic_file_aio_write_nolock+0x2ae/0x5ad
[ 7429.855239] [<c105353f>] generic_file_aio_write+0x58/0xc4
[ 7429.855246] [<c10b02d3>] ext3_file_write+0x2d/0xba
[ 7429.855253] [<c106e73f>] do_sync_write+0xc7/0x116
[ 7429.855260] [<c106e8e6>] vfs_write+0x158/0x15d
[ 7429.855266] [<c106e98c>] sys_write+0x3d/0x64
[ 7429.855275] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.855282] [<ffffffff>] 0xffffffff
[ 7429.855290]
[ 7429.855291] other info that might help us debug this:
[ 7429.855293]
[ 7429.855296] 3 locks held by moo/5453:
[ 7429.855297] #0: (&inode->i_mutex){--..}, at: [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855305] #1: (&ei->truncate_mutex){--..}, at: [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855312] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c109e3e4>] dquot_alloc_space+0x50/0x189
[ 7429.855323]
[ 7429.855324] stack backtrace:
[ 7429.855326] [<c1004d53>] show_trace_log_lvl+0x1a/0x30
[ 7429.855331] [<c1004d7b>] show_trace+0x12/0x14
[ 7429.855335] [<c1004e75>] dump_stack+0x16/0x18
[ 7429.855339] [<c103908e>] print_circular_bug_tail+0x6f/0x71
[ 7429.855343] [<c1039717>] check_prev_add+0x34/0x281
[ 7429.855347] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.855350] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.855355] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.855362] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.855366] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855370] [<c109dd5a>] print_warning+0x8c/0x15d
[ 7429.855374] [<c109e518>] dquot_alloc_space+0x184/0x189
[ 7429.855378] [<c10af496>] ext3_new_blocks+0x44b/0x5a2
[ 7429.855382] [<c10b1a28>] ext3_alloc_blocks+0x40/0xdf
[ 7429.855386] [<c10b1b17>] ext3_alloc_branch+0x50/0x21b
[ 7429.855007] [<c1140bd9>] con_close+0x5e/0x6b
[ 7429.855017] [<c11329d6>] release_dev+0x4c4/0x6ce
[ 7429.855025] [<c1133059>] tty_release+0x12/0x1c
[ 7429.855033] [<c106f67d>] __fput+0x145/0x16e
[ 7429.855040] [<c106f536>] fput+0x19/0x1b
[ 7429.855046] [<c106ddb6>] filp_close+0x3c/0x75
[ 7429.855056] [<c106de58>] sys_close+0x69/0xb4
[ 7429.855064] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.855070] [<ffffffff>] 0xffffffff
[ 7429.855081]
[ 7429.855082] -> #0 (tty_mutex){--..}:
[ 7429.855085] [<c1039717>] check_prev_add+0x34/0x281
[ 7429.855098] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.855105] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.855112] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.855120] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.855127] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855134] [<c109dd5a>] print_warning+0x8c/0x15d
[ 7429.855144] [<c109e518>] dquot_alloc_space+0x184/0x189
[ 7429.855151] [<c10af496>] ext3_new_blocks+0x44b/0x5a2
[ 7429.855162] [<c10b1a28>] ext3_alloc_blocks+0x40/0xdf
[ 7429.855169] [<c10b1b17>] ext3_alloc_branch+0x50/0x21b
[ 7429.855177] [<c10b1ff5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7429.855184] [<c10b221c>] ext3_get_block+0x78/0xe3
[ 7429.855195] [<c108f265>] __block_prepare_write+0x168/0x408
[ 7429.855202] [<c108fd47>] block_prepare_write+0x28/0x3b
[ 7429.855210] [<c10b26c8>] ext3_prepare_write+0xe3/0x17e
[ 7429.855218] [<c1052a29>] generic_file_buffered_write+0x1b1/0x611
[ 7429.855226] [<c1053137>] __generic_file_aio_write_nolock+0x2ae/0x5ad
[ 7429.855239] [<c105353f>] generic_file_aio_write+0x58/0xc4
[ 7429.855246] [<c10b02d3>] ext3_file_write+0x2d/0xba
[ 7429.855253] [<c106e73f>] do_sync_write+0xc7/0x116
[ 7429.855260] [<c106e8e6>] vfs_write+0x158/0x15d
[ 7429.855266] [<c106e98c>] sys_write+0x3d/0x64
[ 7429.855275] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.855282] [<ffffffff>] 0xffffffff
[ 7429.855290]
[ 7429.855291] other info that might help us debug this:
[ 7429.855293]
[ 7429.855296] 3 locks held by moo/5453:
[ 7429.855297] #0: (&inode->i_mutex){--..}, at: [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855305] #1: (&ei->truncate_mutex){--..}, at: [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855312] #2: (&s->s_dquot.dqptr_sem){----}, at: [<c109e3e4>] dquot_alloc_space+0x50/0x189
[ 7429.855323]
[ 7429.855324] stack backtrace:
[ 7429.855326] [<c1004d53>] show_trace_log_lvl+0x1a/0x30
[ 7429.855331] [<c1004d7b>] show_trace+0x12/0x14
[ 7429.855335] [<c1004e75>] dump_stack+0x16/0x18
[ 7429.855339] [<c103908e>] print_circular_bug_tail+0x6f/0x71
[ 7429.855343] [<c1039717>] check_prev_add+0x34/0x281
[ 7429.855347] [<c10399ef>] check_prevs_add+0x8b/0xe8
[ 7429.855350] [<c103b3bf>] __lock_acquire+0x692/0xb81
[ 7429.855355] [<c103bd16>] lock_acquire+0x62/0x81
[ 7429.855362] [<c120d400>] __mutex_lock_slowpath+0x75/0x28c
[ 7429.855366] [<c120d37f>] mutex_lock+0x8/0xa
[ 7429.855370] [<c109dd5a>] print_warning+0x8c/0x15d
[ 7429.855374] [<c109e518>] dquot_alloc_space+0x184/0x189
[ 7429.855378] [<c10af496>] ext3_new_blocks+0x44b/0x5a2
[ 7429.855382] [<c10b1a28>] ext3_alloc_blocks+0x40/0xdf
[ 7429.855386] [<c10b1b17>] ext3_alloc_branch+0x50/0x21b
[ 7429.855389] [<c10b1ff5>] ext3_get_blocks_handle+0x1b8/0x367
[ 7429.855393] [<c10b221c>] ext3_get_block+0x78/0xe3
[ 7429.855400] [<c108f265>] __block_prepare_write+0x168/0x408
[ 7429.855405] [<c108fd47>] block_prepare_write+0x28/0x3b
[ 7429.855409] [<c10b26c8>] ext3_prepare_write+0xe3/0x17e
[ 7429.855412] [<c1052a29>] generic_file_buffered_write+0x1b1/0x611
[ 7429.855417] [<c1053137>] __generic_file_aio_write_nolock+0x2ae/0x5ad
[ 7429.855422] [<c105353f>] generic_file_aio_write+0x58/0xc4
[ 7429.855426] [<c10b02d3>] ext3_file_write+0x2d/0xba
[ 7429.855430] [<c106e73f>] do_sync_write+0xc7/0x116
[ 7429.855433] [<c106e8e6>] vfs_write+0x158/0x15d
[ 7429.855440] [<c106e98c>] sys_write+0x3d/0x64
[ 7429.855445] [<c1003f74>] syscall_call+0x7/0xb
[ 7429.855449] =======================


Folkert van Heusden

--
www.biglumber.com <- site where one can exchange PGP key signatures
----------------------------------------------------------------------
Phone: +31-6-41278122, PGP-key: 1F28D8AE, www.vanheusden.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/