Re: 2.6.21 reiserfs -- cicular locking?

From: Takashi Iwai
Date: Fri Apr 27 2007 - 06:09:34 EST

At Thu, 26 Apr 2007 22:44:08 -0700,
Andrew Morton wrote:
>
> On Thu, 26 Apr 2007 16:40:14 -0700 David Brownell <david-b@xxxxxxxxxxx> wrote:
>
> > This might be a Heisenberg, but I figure it's worth posting
> > in case anyone else sees similar oddness. Never seen it
> > before or since. It's as if a gremlin got annoyed with me
> > for switching a filesystem from reiser to ext3. :)
> >
> > - Dave
> >
> >
> > =======================================================
> > [ INFO: possible circular locking dependency detected ]
> > 2.6.21-git #6
> > -------------------------------------------------------
> > vi/4556 is trying to acquire lock:
> > (&REISERFS_SB(s)->xattr_dir_sem){..--}, at: [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> >
> > but task is already holding lock:
> > (&inode->i_mutex){--..}, at: [<ffffffff8026c4fa>] chown_common+0x93/0xb3
> >
> > which lock already depends on the new lock.
> >
> >
> > the existing dependency chain (in reverse order) is:
> >
> > -> #1 (&inode->i_mutex){--..}:
> > [<ffffffff802405f4>] __lock_acquire+0x9f7/0xbaa
> > [<ffffffff802ca1ac>] get_xa_root+0x49/0x107
> > [<ffffffff80240822>] lock_acquire+0x7b/0x9f
> > [<ffffffff802ca1ac>] get_xa_root+0x49/0x107
> > [<ffffffff8023dee0>] save_trace+0x40/0x9e
> > [<ffffffff8041e1db>] __mutex_lock_slowpath+0xd8/0x281
> > [<ffffffff8041ff18>] _spin_unlock_irq+0x24/0x4a
> > [<ffffffff802ca1ac>] get_xa_root+0x49/0x107
> > [<ffffffff802ca2a8>] open_xa_dir+0x1c/0xf8
> > [<ffffffff8041f34b>] __down_read+0x34/0x9d
> > [<ffffffff802cb50a>] reiserfs_delete_xattrs+0x64/0x185
> > [<ffffffff802fd620>] _atomic_dec_and_lock+0x14/0x34
> > [<ffffffff802b252e>] reiserfs_delete_inode+0x38/0xae
> > [<ffffffff8027f6bc>] generic_delete_inode+0x64/0xf5
> > [<ffffffff802b24f6>] reiserfs_delete_inode+0x0/0xae
> > [<ffffffff8027f6d2>] generic_delete_inode+0x7a/0xf5
> > [<ffffffff80276b95>] do_unlinkat+0xd9/0x14f
> > [<ffffffff8023f75f>] trace_hardirqs_on+0x123/0x14d
> > [<ffffffff8041f4d1>] trace_hardirqs_on_thunk+0x35/0x37
> > [<ffffffff8020954e>] system_call+0x7e/0x83
> > [<ffffffffffffffff>] 0xffffffffffffffff
> >
> > -> #0 (&REISERFS_SB(s)->xattr_dir_sem){..--}:
> > [<ffffffff8023ee6f>] print_circular_bug_header+0xcc/0xd3
> > [<ffffffff802404f0>] __lock_acquire+0x8f3/0xbaa
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff80240822>] lock_acquire+0x7b/0x9f
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff8023bb03>] down_read+0x32/0x3b
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff8022e285>] __capable+0x9/0x1d
> > [<ffffffff802b2828>] reiserfs_setattr+0x11e/0x1ec
> > [<ffffffff8022bd6b>] current_fs_time+0x35/0x3a
> > [<ffffffff802803a1>] notify_change+0x122/0x231
> > [<ffffffff8026c505>] chown_common+0x9e/0xb3
> > [<ffffffff8026e7a6>] fget+0x88/0xa7
> > [<ffffffff8026c54a>] sys_fchown+0x30/0x47
> > [<ffffffff8020954e>] system_call+0x7e/0x83
> > [<ffffffffffffffff>] 0xffffffffffffffff
> >
> > other info that might help us debug this:
> >
> > 1 lock held by vi/4556:
> > #0: (&inode->i_mutex){--..}, at: [<ffffffff8026c4fa>] chown_common+0x93/0xb3
> >
> > stack backtrace:
> >
> > Call Trace:
> > [<ffffffff8023eacc>] print_circular_bug_tail+0x69/0x72
> > [<ffffffff8023ee6f>] print_circular_bug_header+0xcc/0xd3
> > [<ffffffff802404f0>] __lock_acquire+0x8f3/0xbaa
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff80240822>] lock_acquire+0x7b/0x9f
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff8023bb03>] down_read+0x32/0x3b
> > [<ffffffff802caf4b>] reiserfs_chown_xattrs+0x5b/0x128
> > [<ffffffff8022e285>] __capable+0x9/0x1d
> > [<ffffffff802b2828>] reiserfs_setattr+0x11e/0x1ec
> > [<ffffffff8022bd6b>] current_fs_time+0x35/0x3a
> > [<ffffffff802803a1>] notify_change+0x122/0x231
> > [<ffffffff8026c505>] chown_common+0x9e/0xb3
> > [<ffffffff8026e7a6>] fget+0x88/0xa7
> > [<ffffffff8026c54a>] sys_fchown+0x30/0x47
> > [<ffffffff8020954e>] system_call+0x7e/0x83
> >
>
> cc added. This was also reported againt -rc7-mm1 (or 2)

I got a similar bug right now at the fresh boot of 2.6.21.


ReiserFS: sda2: found reiserfs format "3.6" with standard journal
ReiserFS: sda2: using ordered data mode
ReiserFS: sda2: journal params: device sda2, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: sda2: checking transaction log (sda2)
ReiserFS: sda2: Using r5 hash to sort names
ReiserFS: sda2: Removing [3613 1354701 0x0 SD]..done
ReiserFS: sda2: There were 1 uncompleted unlinks/truncates. Completed

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.21-work #1
-------------------------------------------------------
mktemp/1459 is trying to acquire lock:
(&REISERFS_I(inode)->xattr_sem){..--}, at: [<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]

but task is already holding lock:
(&inode->i_mutex){--..}, at: [<c016d7dc>] open_namei+0xe2/0x5a2

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&inode->i_mutex){--..}:
[<c0134d2e>] __lock_acquire+0xa27/0xbbb
[<e08a2e9a>] get_xa_root+0x42/0xfc [reiserfs]
[<c0134f29>] lock_acquire+0x67/0x81
[<e08a2e9a>] get_xa_root+0x42/0xfc [reiserfs]
[<c028b69d>] __mutex_lock_slowpath+0xe3/0x241
[<e08a2e9a>] get_xa_root+0x42/0xfc [reiserfs]
[<e08a2e9a>] get_xa_root+0x42/0xfc [reiserfs]
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<e08a2f83>] open_xa_dir+0x16/0xd9 [reiserfs]
[<e08a3feb>] reiserfs_delete_xattrs+0x4b/0x15b [reiserfs]
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<c012e0d6>] down_read+0x3d/0x4e
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<e08a3ff7>] reiserfs_delete_xattrs+0x57/0x15b [reiserfs]
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<e088b171>] reiserfs_delete_inode+0x35/0xa1 [reiserfs]
[<c01c0fce>] _atomic_dec_and_lock+0x2a/0x48
[<e088b13c>] reiserfs_delete_inode+0x0/0xa1 [reiserfs]
[<c017507f>] generic_delete_inode+0x75/0xdd
[<c01747c4>] iput+0x60/0x62
[<e0892878>] finish_unfinished+0x2ee/0x350 [reiserfs]
[<c016bc00>] lookup_one_len+0x21/0x59
[<e08a3d45>] reiserfs_xattr_init+0x8f/0x1f6 [reiserfs]
[<e0893cd4>] reiserfs_fill_super+0x95e/0xab6 [reiserfs]
[<c0129520>] rcu_barrier+0x5a/0x6a
[<c0105cd0>] dump_trace+0x89/0x93
[<c010a075>] save_stack_trace+0x1c/0x37
[<c01327a5>] save_trace+0x40/0x92
[<c0165c02>] sget+0x1f/0x33b
[<c0134e2e>] __lock_acquire+0xb27/0xbbb
[<c0165c02>] sget+0x1f/0x33b
[<c01c483e>] vsnprintf+0x450/0x48c
[<c01c494c>] snprintf+0x1f/0x22
[<c0194687>] disk_name+0x7e/0x88
[<c016677f>] get_sb_bdev+0xe6/0x130
[<e0891ccc>] get_super_block+0x20/0x25 [reiserfs]
[<e0893376>] reiserfs_fill_super+0x0/0xab6 [reiserfs]
[<c016633b>] vfs_kern_mount+0x83/0xf6
[<c01663f0>] do_kern_mount+0x2d/0x3e
[<c0177ee4>] do_mount+0x612/0x685
[<c01543e9>] __handle_mm_fault+0x50c/0x902
[<c01543b2>] __handle_mm_fault+0x4d5/0x902
[<c028c912>] _spin_unlock+0x14/0x1c
[<c01547a8>] __handle_mm_fault+0x8cb/0x902
[<c014d104>] get_page_from_freelist+0x1fd/0x31d
[<c0133eeb>] trace_hardirqs_on+0x126/0x150
[<c012e14d>] up_read+0x14/0x27
[<c014d28c>] __alloc_pages+0x68/0x2aa
[<c0176899>] copy_mount_options+0x26/0x109
[<c0177fce>] sys_mount+0x77/0xae
[<c0104d74>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff

-> #1 (&REISERFS_SB(s)->xattr_dir_sem){..--}:
[<c0134d2e>] __lock_acquire+0xa27/0xbbb
[<e08a3bf2>] reiserfs_listxattr+0x5a/0x11e [reiserfs]
[<c0134f29>] lock_acquire+0x67/0x81
[<e08a3bf2>] reiserfs_listxattr+0x5a/0x11e [reiserfs]
[<c012e0d6>] down_read+0x3d/0x4e
[<e08a3bf2>] reiserfs_listxattr+0x5a/0x11e [reiserfs]
[<e08a3bf2>] reiserfs_listxattr+0x5a/0x11e [reiserfs]
[<c0133eeb>] trace_hardirqs_on+0x126/0x150
[<e08a3b98>] reiserfs_listxattr+0x0/0x11e [reiserfs]
[<c017ab4f>] vfs_listxattr+0x46/0x7c
[<c017abc9>] listxattr+0x44/0x87
[<c017ac73>] sys_llistxattr+0x33/0x44
[<c012e14d>] up_read+0x14/0x27
[<c0104dbc>] restore_nocheck+0x12/0x15
[<c0104d74>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff

-> #0 (&REISERFS_I(inode)->xattr_sem){..--}:
[<c012e6d2>] print_stack_trace+0x4e/0x5c
[<c0134c1a>] __lock_acquire+0x913/0xbbb
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<c017570b>] new_inode+0x24/0x8a
[<c0134f29>] lock_acquire+0x67/0x81
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<c012e0d6>] down_read+0x3d/0x4e
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<e08873b5>] reiserfs_create+0x3d/0x1bc [reiserfs]
[<e08a2e37>] reiserfs_permission+0x0/0x21 [reiserfs]
[<c016abf2>] permission+0xc8/0xdb
[<c016b11f>] vfs_create+0x9c/0x106
[<c016d871>] open_namei+0x177/0x5a2
[<c016348d>] do_filp_open+0x25/0x39
[<c028c912>] _spin_unlock+0x14/0x1c
[<c016325b>] get_unused_fd+0xb3/0xbd
[<c01634e3>] do_sys_open+0x42/0xbe
[<c0163598>] sys_open+0x1c/0x1e
[<c0104d74>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff

other info that might help us debug this:

1 lock held by mktemp/1459:
#0: (&inode->i_mutex){--..}, at: [<c016d7dc>] open_namei+0xe2/0x5a2

stack backtrace:
[<c013343e>] print_circular_bug_tail+0x5f/0x67
[<c0134c1a>] __lock_acquire+0x913/0xbbb
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<c017570b>] new_inode+0x24/0x8a
[<c0134f29>] lock_acquire+0x67/0x81
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<c012e0d6>] down_read+0x3d/0x4e
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<e08a5236>] reiserfs_cache_default_acl+0x2a/0x9c [reiserfs]
[<e08873b5>] reiserfs_create+0x3d/0x1bc [reiserfs]
[<e08a2e37>] reiserfs_permission+0x0/0x21 [reiserfs]
[<c016abf2>] permission+0xc8/0xdb
[<c016b11f>] vfs_create+0x9c/0x106
[<c016d871>] open_namei+0x177/0x5a2
[<c016348d>] do_filp_open+0x25/0x39
[<c028c912>] _spin_unlock+0x14/0x1c
[<c016325b>] get_unused_fd+0xb3/0xbd
[<c01634e3>] do_sys_open+0x42/0xbe
[<c0163598>] sys_open+0x1c/0x1e
[<c0104d74>] syscall_call+0x7/0xb
=======================


Takashi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/