[Linux Kernel Bug][drivers/media/dvb] possible deadlock in dvb_demux_release

[Chenyuan Yang]

Dear Linux Developers for DVB,

We encountered "possible deadlock in dvb_demux_release" when testing
the dvb driver with Syzkaller and our generated specifications.

The C and syz reproducers and the config for the kernel are attached.

```
======================================================
WARNING: possible circular locking dependency detected
6.6.0-gd2f51b3516da #1 Not tainted
------------------------------------------------------
syz-executor325/10412 is trying to acquire lock:
ffff8880468d8ad8 (&dmxdev->mutex){+.+.}-{3:3}, at:
dvb_dmxdev_filter_free linux/drivers/media/dvb-core/dmxdev.c:833
[inline]
ffff8880468d8ad8 (&dmxdev->mutex){+.+.}-{3:3}, at:
dvb_demux_release+0x8a/0x600
linux/drivers/media/dvb-core/dmxdev.c:1246

but task is already holding lock:
ffffc9000a5aa4c0 (&ctx->mutex){+.+.}-{3:3}, at: _dmxdev_lock+0x40/0x90
linux/drivers/media/dvb-core/dvb_vb2.c:110

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&ctx->mutex){+.+.}-{3:3}:
       __lock_release linux/kernel/locking/lockdep.c:5467 [inline]
       lock_release+0x3c0/0x870 linux/kernel/locking/lockdep.c:5773
       __mutex_unlock_slowpath+0x9e/0x600 linux/kernel/locking/mutex.c:907
       dvb_demux_do_ioctl+0x3ab/0x1630
linux/drivers/media/dvb-core/dmxdev.c:1171
       dvb_usercopy+0xc2/0x280 linux/drivers/media/dvb-core/dvbdev.c:986
       dvb_demux_ioctl+0x31/0x40 linux/drivers/media/dvb-core/dmxdev.c:1185
       vfs_ioctl linux/fs/ioctl.c:51 [inline]
       __do_sys_ioctl linux/fs/ioctl.c:871 [inline]
       __se_sys_ioctl linux/fs/ioctl.c:857 [inline]
       __x64_sys_ioctl+0x1a2/0x210 linux/fs/ioctl.c:857
       do_syscall_x64 linux/arch/x86/entry/common.c:51 [inline]
       do_syscall_64+0x40/0x110 linux/arch/x86/entry/common.c:82
       entry_SYSCALL_64_after_hwframe+0x63/0x6b

-> #0 (&dmxdev->mutex){+.+.}-{3:3}:
       check_prev_add linux/kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add linux/kernel/locking/lockdep.c:3253 [inline]
       validate_chain linux/kernel/locking/lockdep.c:3868 [inline]
       __lock_acquire+0x24a1/0x3b40 linux/kernel/locking/lockdep.c:5136
       lock_acquire linux/kernel/locking/lockdep.c:5753 [inline]
       lock_acquire+0x219/0x650 linux/kernel/locking/lockdep.c:5718
       __mutex_lock_common linux/kernel/locking/mutex.c:603 [inline]
       __mutex_lock+0x14c/0x940 linux/kernel/locking/mutex.c:747
       dvb_dmxdev_filter_free linux/drivers/media/dvb-core/dmxdev.c:833 [inline]
       dvb_demux_release+0x8a/0x600 linux/drivers/media/dvb-core/dmxdev.c:1246
       __fput+0x287/0xbf0 linux/fs/file_table.c:394
       task_work_run+0x16d/0x260 linux/kernel/task_work.c:180
       exit_task_work linux/./include/linux/task_work.h:38 [inline]
       do_exit+0xc38/0x2c00 linux/kernel/exit.c:871
       do_group_exit+0xd9/0x2b0 linux/kernel/exit.c:1021
       get_signal+0x244a/0x2640 linux/kernel/signal.c:2904
       arch_do_signal_or_restart+0x86/0x7e0 linux/arch/x86/kernel/signal.c:309
       exit_to_user_mode_loop linux/kernel/entry/common.c:168 [inline]
       exit_to_user_mode_prepare+0x150/0x250 linux/kernel/entry/common.c:204
       __syscall_exit_to_user_mode_work linux/kernel/entry/common.c:285 [inline]
       syscall_exit_to_user_mode+0x1b/0x50 linux/kernel/entry/common.c:296
       do_syscall_64+0x4d/0x110 linux/arch/x86/entry/common.c:88
       entry_SYSCALL_64_after_hwframe+0x63/0x6b

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&ctx->mutex);
                               lock(&dmxdev->mutex);
                               lock(&ctx->mutex);
  lock(&dmxdev->mutex);

 *** DEADLOCK ***
```

If you have any questions or require more information, please feel
free to contact us.

Reported-by: Chenyuan Yang <chenyuan0y@xxxxxxxxx>

Best,
Chenyuan

Attachment: repro.cprog
Description: Binary data

Attachment: repro.prog
Description: Binary data

Attachment: repro.report
Description: Binary data

Attachment: config
Description: Binary data