Re: [PATCH] block: fix possible bd_size_lock deadlock

[Damien Le Moal]

On 2021/03/13 4:37, Jens Axboe wrote:
> On 3/11/21 5:11 AM, yanfei.xu@xxxxxxxxxxxxx wrote:
>> From: Yanfei Xu <yanfei.xu@xxxxxxxxxxxxx>
>>
>> bd_size_lock spinlock could be taken in block softirq, thus we should
>> disable the softirq before taking the lock.
>>
>> WARNING: inconsistent lock state
>> 5.12.0-rc2-syzkaller #0 Not tainted
>> --------------------------------
>> inconsistent {SOFTIRQ-ON-W} -> {IN-SOFTIRQ-R} usage.
>> kworker/u4:0/7 [HC0[0]:SC1[1]:HE0:SE0] takes:
>> 8f87826c (&inode->i_size_seqcount){+.+-}-{0:0}, at:
>> end_bio_bh_io_sync+0x38/0x54 fs/buffer.c:3006
>> {SOFTIRQ-ON-W} state was registered at:
>>   lock_acquire.part.0+0xf0/0x41c kernel/locking/lockdep.c:5510
>>   lock_acquire+0x6c/0x74 kernel/locking/lockdep.c:5483
>>   do_write_seqcount_begin_nested include/linux/seqlock.h:520 [inline]
>>   do_write_seqcount_begin include/linux/seqlock.h:545 [inline]
>>   i_size_write include/linux/fs.h:863 [inline]
>>   set_capacity+0x13c/0x1f8 block/genhd.c:50
>>   brd_alloc+0x130/0x180 drivers/block/brd.c:401
>>   brd_init+0xcc/0x1e0 drivers/block/brd.c:500
>>   do_one_initcall+0x8c/0x59c init/main.c:1226
>>   do_initcall_level init/main.c:1299 [inline]
>>   do_initcalls init/main.c:1315 [inline]
>>   do_basic_setup init/main.c:1335 [inline]
>>   kernel_init_freeable+0x2cc/0x330 init/main.c:1537
>>   kernel_init+0x10/0x120 init/main.c:1424
>>   ret_from_fork+0x14/0x20 arch/arm/kernel/entry-common.S:158
>>   0x0
>> irq event stamp: 2783413
>> hardirqs last  enabled at (2783412): [<802011ec>]
>> __do_softirq+0xf4/0x7ac kernel/softirq.c:329
>> hardirqs last disabled at (2783413): [<8277d260>]
>> __raw_read_lock_irqsave include/linux/rwlock_api_smp.h:157 [inline]
>> hardirqs last disabled at (2783413): [<8277d260>]
>> _raw_read_lock_irqsave+0x84/0x88 kernel/locking/spinlock.c:231
>> softirqs last  enabled at (2783410): [<826b5050>] spin_unlock_bh
>> include/linux/spinlock.h:399 [inline]
>> softirqs last  enabled at (2783410): [<826b5050>]
>> batadv_nc_purge_paths+0x10c/0x148 net/batman-adv/network-coding.c:467
>> softirqs last disabled at (2783411): [<8024ddfc>] do_softirq_own_stack
>> include/asm-generic/softirq_stack.h:10 [inline]
>> softirqs last disabled at (2783411): [<8024ddfc>] do_softirq
>> kernel/softirq.c:248 [inline]
>> softirqs last disabled at (2783411): [<8024ddfc>] do_softirq+0xd8/0xe4
>> kernel/softirq.c:235
>>
>> other info that might help us debug this:
>>  Possible unsafe locking scenario:
>>
>>        CPU0
>>        ----
>>   lock(&inode->i_size_seqcount);
>>   <Interrupt>
>>     lock(&inode->i_size_seqcount);
>>
>>  *** DEADLOCK ***
>>
>> 3 locks held by kworker/u4:0/7:
>>  #0: 88c622a8 ((wq_completion)bat_events){+.+.}-{0:0}, at: set_work_data
>> kernel/workqueue.c:615 [inline]
>>  #0: 88c622a8 ((wq_completion)bat_events){+.+.}-{0:0}, at:
>> set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
>>  #0: 88c622a8 ((wq_completion)bat_events){+.+.}-{0:0}, at:
>> process_one_work+0x214/0x998 kernel/workqueue.c:2246
>>  #1: 85147ef8
>> ((work_completion)(&(&bat_priv->nc.work)->work)){+.+.}-{0:0}, at:
>> set_work_data kernel/workqueue.c:615 [inline]
>>  #1: 85147ef8
>> ((work_completion)(&(&bat_priv->nc.work)->work)){+.+.}-{0:0}, at:
>> set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
>>  #1: 85147ef8
>> ((work_completion)(&(&bat_priv->nc.work)->work)){+.+.}-{0:0}, at:
>> process_one_work+0x214/0x998 kernel/workqueue.c:2246
>>  #2: 8f878010 (&ni->size_lock){...-}-{2:2}, at:
>> ntfs_end_buffer_async_read+0x6c/0x558 fs/ntfs/aops.c:66
> 
> Damien? We have that revert queued up for this for 5.12, but looking
> at that, the state before that was kind of messy too.

Indeed... I was thinking about this and I think I am with Christoph on this:
drivers should not call set_capacity() from command completion context. I think
the best thing to do would be to fix drivers that do that but that may not be RC
material ?

Looking into more details of this case, it is slightly different though.
set_capacity() is here not called from soft IRQ context. It looks like a regular
initialization, but one that seems way too early in the boot process when a
secondary core is being initialized with IRQ not yet enabled... I think. And the
warnings come from i_size_write() calling preempt_disable() rather than
set_capacity() use of spin_lock(&bdev->bd_size_lock).

I wonder how it is possible to have brd being initialized so early.
I am not sure how to fix that. It looks like arm arch code territory.

For now, we could revert the revert as I do not think that Yanfei patch is
enough since completions may be from hard IRQ context too, which is not covered
with the spin_lock_bh() variants (c.f. a similar problem we are facing with that
in scsi completion [1])
I do not have any good idea how to proceed though.

[1]
https://lore.kernel.org/linux-scsi/PH0PR04MB7416C8330459E92D8AA21A889B6F9@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/T/#t

-- 
Damien Le Moal
Western Digital Research