Re: possible deadlock in proc_pid_syscall

From: Konstantin Khlebnikov
Date: Sat Nov 16 2019 - 11:22:58 EST

Next message: Sasha Levin: "[PATCH AUTOSEL 4.19 090/237] scsi: hisi_sas: Fix NULL pointer dereference"
Previous message: Sasha Levin: "[PATCH AUTOSEL 4.19 093/237] powerpc/mm/radix: Fix overuse of small pages in splitting logic"
In reply to: syzbot: "possible deadlock in proc_pid_syscall"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On 16/11/2019 17.15, syzbot wrote:

Hello,

syzbot found the following crash on:

HEAD commit:ÂÂÂ eb70e26c Merge tag 'arm64-fixes' of git://git.kernel.org/p..
git tree:ÂÂÂÂÂÂ upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=10a38772e00000
kernel config:Â https://syzkaller.appspot.com/x/.config?x=7654f9089a2e8c85
dashboard link: https://syzkaller.appspot.com/bug?extid=ffc5f92d13ebf6b3e760
compiler:ÂÂÂÂÂÂ clang version 9.0.0 (/home/glider/llvm/clang 80fee25776c2fb61e74c1ecb1a523375c2500b69)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+ffc5f92d13ebf6b3e760@xxxxxxxxxxxxxxxxxxxxxxxxx

======================================================
WARNING: possible circular locking dependency detected
5.4.0-rc7+ #0 Not tainted
------------------------------------------------------
syz-executor.4/16763 is trying to acquire lock:
ffff8880916fc710 (&sig->cred_guard_mutex){+.+.}, at: lock_trace fs/proc/base.c:406 [inline]
ffff8880916fc710 (&sig->cred_guard_mutex){+.+.}, at: proc_pid_syscall+0x66/0x3a0 fs/proc/base.c:635

but task is already holding lock:
ffff8880a02001c0 (&p->lock){+.+.}, at: seq_read+0x6e/0xd70 fs/seq_file.c:161

which lock already depends on the new lock.

It seems reason is similar to this problem
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eb94cd96e05d6c65a07937e66a04ea265c1b767d
but this chain is more complicated.

Exec grabs task->signal->cred_guard_mutex in prepare_bprm_creds() and
then open file -- vfs locks nests inside cred_guard_mutex.

Procfs grabs cred_guard_mutex in lock_trace when reads files like "syscall".
inside "p->lock" (mutex inside seq_file which serialize read and seek).

Seq_file usually are not executable (exec /proc/self/syscall will fail before calling read).
It seems there is no trivial deadlock, but really might be some chain with vfs locks.

the existing dependency chain (in reverse order) is:

-> #3 (&p->lock){+.+.}:
ÂÂÂÂÂÂ lock_acquire+0x158/0x250 kernel/locking/lockdep.c:4487
ÂÂÂÂÂÂ __mutex_lock_common+0x14c/0x2e20 kernel/locking/mutex.c:956
ÂÂÂÂÂÂ __mutex_lock kernel/locking/mutex.c:1103 [inline]
ÂÂÂÂÂÂ mutex_lock_nested+0x1b/0x30 kernel/locking/mutex.c:1118
ÂÂÂÂÂÂ seq_read+0x6e/0xd70 fs/seq_file.c:161
ÂÂÂÂÂÂ proc_reg_read+0x1e2/0x2d0 fs/proc/inode.c:223
ÂÂÂÂÂÂ do_loop_readv_writev fs/read_write.c:714 [inline]
ÂÂÂÂÂÂ do_iter_read+0x4b1/0x5b0 fs/read_write.c:935
ÂÂÂÂÂÂ vfs_readv+0xc2/0x120 fs/read_write.c:997
ÂÂÂÂÂÂ kernel_readv fs/splice.c:359 [inline]
ÂÂÂÂÂÂ default_file_splice_read+0x544/0x8d0 fs/splice.c:414
ÂÂÂÂÂÂ do_splice_to fs/splice.c:877 [inline]
ÂÂÂÂÂÂ splice_direct_to_actor+0x39c/0xac0 fs/splice.c:954
ÂÂÂÂÂÂ do_splice_direct+0x200/0x330 fs/splice.c:1063
ÂÂÂÂÂÂ do_sendfile+0x7e4/0xfd0 fs/read_write.c:1464
ÂÂÂÂÂÂ __do_sys_sendfile64 fs/read_write.c:1525 [inline]
ÂÂÂÂÂÂ __se_sys_sendfile64 fs/read_write.c:1511 [inline]
ÂÂÂÂÂÂ __x64_sys_sendfile64+0x176/0x1b0 fs/read_write.c:1511
ÂÂÂÂÂÂ do_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
ÂÂÂÂÂÂ entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (sb_writers#3){.+.+}:
ÂÂÂÂÂÂ lock_acquire+0x158/0x250 kernel/locking/lockdep.c:4487
ÂÂÂÂÂÂ percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
ÂÂÂÂÂÂ __sb_start_write+0xeb/0x430 fs/super.c:1672
ÂÂÂÂÂÂ sb_start_write include/linux/fs.h:1650 [inline]
ÂÂÂÂÂÂ mnt_want_write+0x4a/0xa0 fs/namespace.c:354
ÂÂÂÂÂÂ ovl_want_write+0x77/0x80 fs/overlayfs/util.c:21
ÂÂÂÂÂÂ ovl_create_object+0xaf/0x2d0 fs/overlayfs/dir.c:596
ÂÂÂÂÂÂ ovl_create+0x29/0x30 fs/overlayfs/dir.c:627
ÂÂÂÂÂÂ lookup_open fs/namei.c:3224 [inline]
ÂÂÂÂÂÂ do_last fs/namei.c:3314 [inline]
ÂÂÂÂÂÂ path_openat+0x2236/0x4420 fs/namei.c:3525
ÂÂÂÂÂÂ do_filp_open+0x192/0x3d0 fs/namei.c:3555
ÂÂÂÂÂÂ do_sys_open+0x29f/0x560 fs/open.c:1097
ÂÂÂÂÂÂ ksys_open include/linux/syscalls.h:1385 [inline]
ÂÂÂÂÂÂ __do_sys_creat fs/open.c:1155 [inline]
ÂÂÂÂÂÂ __se_sys_creat fs/open.c:1153 [inline]
ÂÂÂÂÂÂ __x64_sys_creat+0x65/0x70 fs/open.c:1153
ÂÂÂÂÂÂ do_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
ÂÂÂÂÂÂ entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&ovl_i_mutex_dir_key[depth]){++++}:
ÂÂÂÂÂÂ lock_acquire+0x158/0x250 kernel/locking/lockdep.c:4487
ÂÂÂÂÂÂ down_read+0x39/0x50 kernel/locking/rwsem.c:1495
ÂÂÂÂÂÂ inode_lock_shared include/linux/fs.h:801 [inline]
ÂÂÂÂÂÂ do_last fs/namei.c:3313 [inline]
ÂÂÂÂÂÂ path_openat+0xa9a/0x4420 fs/namei.c:3525
ÂÂÂÂÂÂ do_filp_open+0x192/0x3d0 fs/namei.c:3555
ÂÂÂÂÂÂ do_open_execat+0xff/0x610 fs/exec.c:857
ÂÂÂÂÂÂ __do_execve_file+0x766/0x1cc0 fs/exec.c:1762
ÂÂÂÂÂÂ do_execveat_common fs/exec.c:1868 [inline]
ÂÂÂÂÂÂ do_execve fs/exec.c:1885 [inline]
ÂÂÂÂÂÂ __do_sys_execve fs/exec.c:1961 [inline]
ÂÂÂÂÂÂ __se_sys_execve fs/exec.c:1956 [inline]
ÂÂÂÂÂÂ __x64_sys_execve+0x94/0xb0 fs/exec.c:1956
ÂÂÂÂÂÂ do_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
ÂÂÂÂÂÂ entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&sig->cred_guard_mutex){+.+.}:
ÂÂÂÂÂÂ check_prev_add kernel/locking/lockdep.c:2476 [inline]
ÂÂÂÂÂÂ check_prevs_add kernel/locking/lockdep.c:2581 [inline]
ÂÂÂÂÂÂ validate_chain+0x161e/0x7bc0 kernel/locking/lockdep.c:2971
ÂÂÂÂÂÂ __lock_acquire+0xc75/0x1be0 kernel/locking/lockdep.c:3955
ÂÂÂÂÂÂ lock_acquire+0x158/0x250 kernel/locking/lockdep.c:4487
ÂÂÂÂÂÂ __mutex_lock_common+0x14c/0x2e20 kernel/locking/mutex.c:956
ÂÂÂÂÂÂ __mutex_lock kernel/locking/mutex.c:1103 [inline]
ÂÂÂÂÂÂ mutex_lock_killable_nested+0x1b/0x30 kernel/locking/mutex.c:1133
ÂÂÂÂÂÂ lock_trace fs/proc/base.c:406 [inline]
ÂÂÂÂÂÂ proc_pid_syscall+0x66/0x3a0 fs/proc/base.c:635
ÂÂÂÂÂÂ proc_single_show+0xd8/0x120 fs/proc/base.c:756
ÂÂÂÂÂÂ seq_read+0x4cd/0xd70 fs/seq_file.c:229
ÂÂÂÂÂÂ do_loop_readv_writev fs/read_write.c:714 [inline]
ÂÂÂÂÂÂ do_iter_read+0x4b1/0x5b0 fs/read_write.c:935
ÂÂÂÂÂÂ vfs_readv fs/read_write.c:997 [inline]
ÂÂÂÂÂÂ do_preadv+0x178/0x290 fs/read_write.c:1089
ÂÂÂÂÂÂ __do_sys_preadv fs/read_write.c:1139 [inline]
ÂÂÂÂÂÂ __se_sys_preadv fs/read_write.c:1134 [inline]
ÂÂÂÂÂÂ __x64_sys_preadv+0x9e/0xb0 fs/read_write.c:1134
ÂÂÂÂÂÂ do_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
ÂÂÂÂÂÂ entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
Â &sig->cred_guard_mutex --> sb_writers#3 --> &p->lock

ÂPossible unsafe locking scenario:

ÂÂÂÂÂÂ CPU0ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ CPU1
ÂÂÂÂÂÂ ----ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ----
Â lock(&p->lock);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ lock(sb_writers#3);
ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ lock(&p->lock);
Â lock(&sig->cred_guard_mutex);

Â*** DEADLOCK ***

1 lock held by syz-executor.4/16763:
Â#0: ffff8880a02001c0 (&p->lock){+.+.}, at: seq_read+0x6e/0xd70 fs/seq_file.c:161

stack backtrace:
CPU: 0 PID: 16763 Comm: syz-executor.4 Not tainted 5.4.0-rc7+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
Â__dump_stack lib/dump_stack.c:77 [inline]
Âdump_stack+0x1fb/0x318 lib/dump_stack.c:118
Âprint_circular_bug+0xc2d/0xe60 kernel/locking/lockdep.c:1685
Âcheck_noncircular+0x206/0x3a0 kernel/locking/lockdep.c:1809
Âcheck_prev_add kernel/locking/lockdep.c:2476 [inline]
Âcheck_prevs_add kernel/locking/lockdep.c:2581 [inline]
Âvalidate_chain+0x161e/0x7bc0 kernel/locking/lockdep.c:2971
Â__lock_acquire+0xc75/0x1be0 kernel/locking/lockdep.c:3955
Âlock_acquire+0x158/0x250 kernel/locking/lockdep.c:4487
Â__mutex_lock_common+0x14c/0x2e20 kernel/locking/mutex.c:956
Â__mutex_lock kernel/locking/mutex.c:1103 [inline]
Âmutex_lock_killable_nested+0x1b/0x30 kernel/locking/mutex.c:1133
Âlock_trace fs/proc/base.c:406 [inline]
Âproc_pid_syscall+0x66/0x3a0 fs/proc/base.c:635
Âproc_single_show+0xd8/0x120 fs/proc/base.c:756
Âseq_read+0x4cd/0xd70 fs/seq_file.c:229
Âdo_loop_readv_writev fs/read_write.c:714 [inline]
Âdo_iter_read+0x4b1/0x5b0 fs/read_write.c:935
Âvfs_readv fs/read_write.c:997 [inline]
Âdo_preadv+0x178/0x290 fs/read_write.c:1089
Â__do_sys_preadv fs/read_write.c:1139 [inline]
Â__se_sys_preadv fs/read_write.c:1134 [inline]
Â__x64_sys_preadv+0x9e/0xb0 fs/read_write.c:1134
Âdo_syscall_64+0xf7/0x1c0 arch/x86/entry/common.c:290
Âentry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45a669
Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ff34e552c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000127
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000000000045a669
RDX: 00000000000001e3 RSI: 00000000200013c0 RDI: 0000000000000004
RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007ff34e5536d4
R13: 00000000004c7ef1 R14: 00000000004dde10 R15: 00000000ffffffff

---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@xxxxxxxxxxxxxxxxx

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

Next message: Sasha Levin: "[PATCH AUTOSEL 4.19 090/237] scsi: hisi_sas: Fix NULL pointer dereference"
Previous message: Sasha Levin: "[PATCH AUTOSEL 4.19 093/237] powerpc/mm/radix: Fix overuse of small pages in splitting logic"
In reply to: syzbot: "possible deadlock in proc_pid_syscall"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]