Re: possible deadlock in io_link_timeout_fn

From: Pavel Begunkov
Date: Tue Feb 23 2021 - 08:54:58 EST

Next message: Juri Lelli: "Re: [RT v5.11-rt7] WARNING at include/linux/seqlock.h:271 nft_counter_eval"
Previous message: Leon Romanovsky: "Re: [PATCH v15 2/4] phy: Add media type and speed serdes configuration interfaces"
In reply to: syzbot: "possible deadlock in io_link_timeout_fn"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On 23/02/2021 13:35, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: 31caf8b2 Merge branch 'linus' of git://git.kernel.org/pub/..
> git tree: upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=11c64f12d00000
> kernel config: https://syzkaller.appspot.com/x/.config?x=5a8f3a57fabb4015
> dashboard link: https://syzkaller.appspot.com/bug?extid=9a512c5bdc15635eab70
>
> Unfortunately, I don't have any reproducer for this issue yet.
>
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+9a512c5bdc15635eab70@xxxxxxxxxxxxxxxxxxxxxxxxx

Ok, that IRQ recycling stuff kicks in. I guess we can take
work_clean out of spin, I'll try it out this evening.

Either, Jen's io-wq patches go 5.12 in the end, and it would probably be
solved naturally.

>
> =====================================================
> WARNING: HARDIRQ-safe -> HARDIRQ-unsafe lock order detected
> 5.11.0-syzkaller #0 Not tainted
> -----------------------------------------------------
> syz-executor.0/12185 [HC0[0]:SC0[0]:HE0:SE1] is trying to acquire:
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: spin_lock include/linux/spinlock.h:354 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_req_clean_work fs/io_uring.c:1405 [inline]
> ffff888013db4820 (&fs->lock){+.+.}-{2:2}, at: io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
>
> and this task is already holding:
> ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> which would create a new lock dependency:
> (&ctx->completion_lock){-...}-{2:2} -> (&fs->lock){+.+.}-{2:2}
>
> but this new dependency connects a HARDIRQ-irq-safe lock:
> (&ctx->completion_lock){-...}-{2:2}
>
> ... which became HARDIRQ-irq-safe at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
> __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
> __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
> hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
> local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
> __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
> asm_call_irq_on_stack+0xf/0x20
> __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
> run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
> sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
> asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
> __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
> _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
> spin_unlock_irq include/linux/spinlock.h:404 [inline]
> io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
> __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
> __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
> __tctx_task_work fs/io_uring.c:2204 [inline]
> tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
> task_work_run+0xdd/0x1a0 kernel/task_work.c:140
> tracehook_notify_signal include/linux/tracehook.h:212 [inline]
> handle_signal_work kernel/entry/common.c:145 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> to a HARDIRQ-irq-unsafe lock:
> (&fs->lock){+.+.}-{2:2}
>
> ... which became HARDIRQ-irq-unsafe at:
> ...
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
>
> other info that might help us debug this:
>
> Possible interrupt unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(&fs->lock);
> local_irq_disable();
> lock(&ctx->completion_lock);
> lock(&fs->lock);
> <Interrupt>
> lock(&ctx->completion_lock);
>
> *** DEADLOCK ***
>
> 1 lock held by syz-executor.0/12185:
> #0: ffff88806bea6718 (&ctx->completion_lock){-...}-{2:2}, at: io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
>
> the dependencies between HARDIRQ-irq-safe lock and the holding lock:
> -> (&ctx->completion_lock){-...}-{2:2} {
> IN-HARDIRQ-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_link_timeout_fn+0xbf/0x720 fs/io_uring.c:6495
> __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
> __hrtimer_run_queues+0x609/0xe40 kernel/time/hrtimer.c:1583
> hrtimer_interrupt+0x334/0x940 kernel/time/hrtimer.c:1645
> local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1089 [inline]
> __sysvec_apic_timer_interrupt+0x146/0x540 arch/x86/kernel/apic/apic.c:1106
> asm_call_irq_on_stack+0xf/0x20
> __run_sysvec_on_irqstack arch/x86/include/asm/irq_stack.h:37 [inline]
> run_sysvec_on_irqstack_cond arch/x86/include/asm/irq_stack.h:89 [inline]
> sysvec_apic_timer_interrupt+0xbd/0x100 arch/x86/kernel/apic/apic.c:1100
> asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:635
> __raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
> _raw_spin_unlock_irq+0x25/0x40 kernel/locking/spinlock.c:199
> spin_unlock_irq include/linux/spinlock.h:404 [inline]
> io_queue_linked_timeout+0x194/0x1f0 fs/io_uring.c:6541
> __io_queue_sqe+0x32f/0xdb0 fs/io_uring.c:6607
> __io_req_task_submit+0x18e/0x240 fs/io_uring.c:2344
> __tctx_task_work fs/io_uring.c:2204 [inline]
> tctx_task_work+0x12b/0x330 fs/io_uring.c:2230
> task_work_run+0xdd/0x1a0 kernel/task_work.c:140
> tracehook_notify_signal include/linux/tracehook.h:212 [inline]
> handle_signal_work kernel/entry/common.c:145 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x221/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> INITIAL USE at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
> _raw_spin_lock_irqsave+0x39/0x50 kernel/locking/spinlock.c:159
> io_req_complete_post+0x4e/0x920 fs/io_uring.c:1923
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_queue_sqe+0xa3b/0xfa0 fs/io_uring.c:6620
> io_submit_sqe fs/io_uring.c:6707 [inline]
> io_submit_sqes+0x15f5/0x2b60 fs/io_uring.c:6939
> __do_sys_io_uring_enter+0x1154/0x1f50 fs/io_uring.c:9454
> do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> }
> ... key at: [<ffffffff8fe69a80>] __key.9+0x0/0x40
> ... acquired at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
>
> the dependencies between the lock to be acquired
> and HARDIRQ-irq-unsafe lock:
> -> (&fs->lock){+.+.}-{2:2} {
> HARDIRQ-ON-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> SOFTIRQ-ON-W at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> INITIAL USE at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> set_fs_pwd+0x85/0x2a0 fs/fs_struct.c:39
> init_chdir+0x106/0x14e fs/init.c:54
> devtmpfs_setup drivers/base/devtmpfs.c:415 [inline]
> devtmpfsd+0x76/0x333 drivers/base/devtmpfs.c:430
> kthread+0x3b1/0x4a0 kernel/kthread.c:292
> ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294
> }
> ... key at: [<ffffffff8fe68260>] __key.1+0x0/0x40
> ... acquired at:
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
>
> stack backtrace:
> CPU: 3 PID: 12185 Comm: syz-executor.0 Not tainted 5.11.0-syzkaller #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.14.0-2 04/01/2014
> Call Trace:
> __dump_stack lib/dump_stack.c:79 [inline]
> dump_stack+0xfa/0x151 lib/dump_stack.c:120
> print_bad_irq_dependency kernel/locking/lockdep.c:2460 [inline]
> check_irq_usage.cold+0x50d/0x744 kernel/locking/lockdep.c:2689
> check_prev_add kernel/locking/lockdep.c:2940 [inline]
> check_prevs_add kernel/locking/lockdep.c:3059 [inline]
> validate_chain kernel/locking/lockdep.c:3674 [inline]
> __lock_acquire+0x2b2c/0x54c0 kernel/locking/lockdep.c:4900
> lock_acquire kernel/locking/lockdep.c:5510 [inline]
> lock_acquire+0x1ab/0x730 kernel/locking/lockdep.c:5475
> __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
> _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:151
> spin_lock include/linux/spinlock.h:354 [inline]
> io_req_clean_work fs/io_uring.c:1405 [inline]
> io_dismantle_req+0x90f/0xf90 fs/io_uring.c:2051
> io_req_complete_post+0xf6/0x920 fs/io_uring.c:1933
> __io_req_complete fs/io_uring.c:1963 [inline]
> io_req_complete fs/io_uring.c:1968 [inline]
> io_cancel_defer_files fs/io_uring.c:8945 [inline]
> io_uring_cancel_task_requests+0x67e/0xea0 fs/io_uring.c:9052
> __io_uring_files_cancel+0x110/0x1b0 fs/io_uring.c:9140
> io_uring_files_cancel include/linux/io_uring.h:65 [inline]
> do_exit+0x2fe/0x2ae0 kernel/exit.c:780
> do_group_exit+0x125/0x310 kernel/exit.c:922
> get_signal+0x42c/0x2100 kernel/signal.c:2773
> arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
> handle_signal_work kernel/entry/common.c:147 [inline]
> exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
> exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:208
> __syscall_exit_to_user_mode_work kernel/entry/common.c:290 [inline]
> syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:301
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> RIP: 0033:0x465ef9
> Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007ffb56aa0218 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
> RAX: 0000000000000000 RBX: 000000000056bf68 RCX: 0000000000465ef9
> RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000056bf68
> RBP: 000000000056bf60 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056bf6c
> R13: 00007fff198147ff R14: 00007ffb56aa0300 R15: 0000000000022000
>
>
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@xxxxxxxxxxxxxxxx.
>
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
>

--
Pavel Begunkov

Next message: Juri Lelli: "Re: [RT v5.11-rt7] WARNING at include/linux/seqlock.h:271 nft_counter_eval"
Previous message: Leon Romanovsky: "Re: [PATCH v15 2/4] phy: Add media type and speed serdes configuration interfaces"
In reply to: syzbot: "possible deadlock in io_link_timeout_fn"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]