Re: possible circular locking dependency detected

From: Andrew Morton
Date: Fri May 21 2010 - 17:15:00 EST


On Thu, 20 May 2010 12:34:00 -0400 (EDT)
Ciprian Docan <docan@xxxxxxxxxxxxxxxx> wrote:

>
> Hi,
>
> I got the following in the dmesg:
>
> --------------------------------------------------------------------
> [ INFO: possible circular locking dependency detected ]
> 2.6.33-rc8 #4
> -------------------------------------------------------
> fdisk/29231 is trying to acquire lock:
> (&type->s_umount_key#47){++++..}, at: [<ffffffff810fb13c>]
> get_super+0x5c/0xaf
>
> but task is already holding lock:
> (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811f2df0>]
> blkdev_ioctl+0x5c5/0x6b1
>
> which lock already depends on the new lock.
>
>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (&bdev->bd_mutex){+.+.+.}:
> [<ffffffff8106e65b>] __lock_acquire+0xb5d/0xd05
> [<ffffffff8106e8cf>] lock_acquire+0xcc/0xe9
> [<ffffffff81402d09>] __mutex_lock_common+0x4c/0x348
> [<ffffffff814030c9>] mutex_lock_nested+0x3e/0x43
> [<ffffffff8111f4a9>] __blkdev_put+0x34/0x16c
> [<ffffffff8111f5f1>] blkdev_put+0x10/0x12
> [<ffffffff8112063b>] close_bdev_exclusive+0x24/0x2d
> [<ffffffff810fbcaa>] get_sb_bdev+0xef/0x1a1
> [<ffffffffa0114189>] vfat_get_sb+0x18/0x1a [vfat]
> [<ffffffff810fb8bc>] vfs_kern_mount+0xa9/0x168
> [<ffffffff810fb9e3>] do_kern_mount+0x4d/0xed
> [<ffffffff81110f54>] do_mount+0x72f/0x7a6
> [<ffffffff81111053>] sys_mount+0x88/0xc2
> [<ffffffff8100236b>] system_call_fastpath+0x16/0x1b

vfs_kern_mount() holds s_umount. My brain isn't large enough to work
out where that lock was taken, yet it's so obvious that no code
comments were needed. Sigh. Might be down under sget().

vfs_kern_mount() ends up calling into __blkdev_put(), which takes
bd_mutex.

> -> #0 (&type->s_umount_key#47){++++..}:
> [<ffffffff8106e505>] __lock_acquire+0xa07/0xd05
> [<ffffffff8106e8cf>] lock_acquire+0xcc/0xe9
> [<ffffffff81403450>] down_read+0x51/0x84
> [<ffffffff810fb13c>] get_super+0x5c/0xaf
> [<ffffffff8111facd>] fsync_bdev+0x18/0x48
> [<ffffffff811f433c>] invalidate_partition+0x25/0x42
> [<ffffffff8114bda2>] rescan_partitions+0x37/0x3a7
> [<ffffffff811f2dff>] blkdev_ioctl+0x5d4/0x6b1
> [<ffffffff8111eca4>] block_ioctl+0x37/0x3b
> [<ffffffff811060d0>] vfs_ioctl+0x32/0xa6
> [<ffffffff81106650>] do_vfs_ioctl+0x490/0x4d6
> [<ffffffff811066ec>] sys_ioctl+0x56/0x79
> [<ffffffff8100236b>] system_call_fastpath+0x16/0x1b

blkdev_reread_part() takes bd_mutex then does
rescan_partitions
->invalidate_partition
->fsync_bdev
->get_super (takes s_umount for reading)

> other info that might help us debug this:
>
> 1 lock held by fdisk/29231:
> #0: (&bdev->bd_mutex){+.+.+.}, at: [<ffffffff811f2df0>]
> blkdev_ioctl+0x5c5/0x6b1
>
> stack backtrace:
> Pid: 29231, comm: fdisk Not tainted 2.6.33-rc8 #4
> Call Trace:
> [<ffffffff8106d6dc>] print_circular_bug+0xa8/0xb6
> [<ffffffff8106e505>] __lock_acquire+0xa07/0xd05
> [<ffffffff81062009>] ? sched_clock_local+0x1c/0x82
> [<ffffffff8106e8cf>] lock_acquire+0xcc/0xe9
> [<ffffffff810fb13c>] ? get_super+0x5c/0xaf
> [<ffffffff8106b936>] ? lock_release_holdtime+0x2c/0xdb
> [<ffffffff81403450>] down_read+0x51/0x84
> [<ffffffff810fb13c>] ? get_super+0x5c/0xaf
> [<ffffffff810fb13c>] get_super+0x5c/0xaf
> [<ffffffff8111facd>] fsync_bdev+0x18/0x48
> [<ffffffff811f433c>] invalidate_partition+0x25/0x42
> [<ffffffff81402c8e>] ? mutex_trylock+0x12a/0x159
> [<ffffffff8114bda2>] rescan_partitions+0x37/0x3a7
> [<ffffffff8106d0c9>] ? trace_hardirqs_on+0xd/0xf
> [<ffffffff811f2df0>] ? blkdev_ioctl+0x5c5/0x6b1
> [<ffffffff811f2dff>] blkdev_ioctl+0x5d4/0x6b1
> [<ffffffff8106d098>] ? trace_hardirqs_on_caller+0x118/0x13c
> [<ffffffff8111eca4>] block_ioctl+0x37/0x3b
> [<ffffffff811060d0>] vfs_ioctl+0x32/0xa6
> [<ffffffff81106650>] do_vfs_ioctl+0x490/0x4d6
> [<ffffffff811066ec>] sys_ioctl+0x56/0x79
> [<ffffffff8102f9bd>] ? __wake_up+0x22/0x4d
> [<ffffffff8100236b>] system_call_fastpath+0x16/0x1b
> -------------------------------------------------------------------------
>
> Kernel version used: 2.6.33-rc8 #4. I do not remember the exact steps, but
> I was trying to format an USB stick using the fdisk. Please let me know if
> you need additional informations. Thank you.
>

So yup, that's ab/ba deadlockable. I cannot immediately see any change
which might have caused that. Tejun has been mucking with the
partitions code recently but nothing leaps out at me.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/