Re: WARNING: lock held when returning to user space in fuse_lock_inode

[Miklos Szeredi]

On Tue, Jul 17, 2018 at 1:36 PM, Dmitry Vyukov <dvyukov@xxxxxxxxxx> wrote:
> On Tue, Jul 17, 2018 at 1:14 PM, Miklos Szeredi <miklos@xxxxxxxxxx> wrote:
>> On Thu, Jul 12, 2018 at 5:49 PM, syzbot
>> <syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
>>> Hello,
>>>
>>> syzbot found the following crash on:
>>>
>>> HEAD commit:    c25c74b7476e Merge tag 'trace-v4.18-rc3-2' of git://git.ke..
>>> git tree:       upstream
>>> console output: https://syzkaller.appspot.com/x/log.txt?x=177bcec2400000
>>> kernel config:  https://syzkaller.appspot.com/x/.config?x=25856fac4e580aa7
>>> dashboard link: https://syzkaller.appspot.com/bug?extid=3f7b29af1baa9d0a55be
>>> compiler:       gcc (GCC) 8.0.1 20180413 (experimental)
>>> syzkaller repro:https://syzkaller.appspot.com/x/repro.syz?x=13aa7678400000
>>> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17492678400000
>>>
>>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>>> Reported-by: syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx
>>>
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>> random: sshd: uninitialized urandom read (32 bytes read)
>>>
>>> ================================================
>>> WARNING: lock held when returning to user space!
>>> 4.18.0-rc4+ #143 Not tainted
>>> ------------------------------------------------
>>> syz-executor012/4539 is leaving the kernel with locks still held!
>>> 1 lock held by syz-executor012/4539:
>>>  #0: (____ptrval____) (&fi->mutex){+.+.}, at: fuse_lock_inode+0xaf/0xe0
>>> fs/fuse/inode.c:363
>>
>> False positive.
>>
>> fi->mutex is definitely not held by the acquiring task when returning
>> to userspace.  Maybe syzkaller is confused by the fact that there are
>> several interdependent tasks involved with fuse:  the one calling into
>> fuse by doing something (looking up ./file0/file0) and the one that
>> reads the fuse device (returning with the LOOKUP request for "file0").
>> The second one will return with that lock held, but it's not the one
>> that acquired it, so there's no bug at all here.
>
> Hi Miklos,
>
> syzkaller is unrelated here. That's what kernel self-detects and
> prints. So either way there is something to fix in kernel here: either
> fuse or lockdep.
>
> +Alistair did some analysis offline, hope you don't mind if I repost
> your description:
> ===
> Just from reading the code, I think I can see how this happens. Fuse
> is wrapping its inode mutex with a check for "parallel_dirops", which
> is set up in process_init_reply(). The FUSE_PARALLEL_DIROPS appears to
> always be set, in fuse_send_init(), but its initial state is to be
> disabled. So if the mutex gets taken, and it'll never be unlocked if
> the initial command is flushed by fuse_readdir()'s use of
> fuse_lock_inode().
> ===

Ah, indeed.  Fix attached.

Thanks,
Miklos
From: Miklos Szeredi <mszeredi@xxxxxxxxxx>
Subject: fuse: fix inital parallel dirops

If parallel dirops are enabled in FUSE_INIT reply, then first operation may
leave fi->mutex held.

Reported-by: syzbot+3f7b29af1baa9d0a55be@xxxxxxxxxxxxxxxxxxxxxxxxx
Signed-off-by: Miklos Szeredi <mszeredi@xxxxxxxxxx>
---
 fs/fuse/dir.c    |   10 ++++++----
 fs/fuse/fuse_i.h |    4 ++--
 fs/fuse/inode.c  |   14 ++++++++++----
 3 files changed, 18 insertions(+), 10 deletions(-)

--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -355,11 +355,12 @@ static struct dentry *fuse_lookup(struct
 	struct inode *inode;
 	struct dentry *newent;
 	bool outarg_valid = true;
+	bool locked;
 
-	fuse_lock_inode(dir);
+	locked = fuse_lock_inode(dir);
 	err = fuse_lookup_name(dir->i_sb, get_node_id(dir), &entry->d_name,
 			       &outarg, &inode);
-	fuse_unlock_inode(dir);
+	fuse_unlock_inode(dir, locked);
 	if (err == -ENOENT) {
 		outarg_valid = false;
 		err = 0;
@@ -1340,6 +1341,7 @@ static int fuse_readdir(struct file *fil
 	struct fuse_conn *fc = get_fuse_conn(inode);
 	struct fuse_req *req;
 	u64 attr_version = 0;
+	bool locked;
 
 	if (is_bad_inode(inode))
 		return -EIO;
@@ -1367,9 +1369,9 @@ static int fuse_readdir(struct file *fil
 		fuse_read_fill(req, file, ctx->pos, PAGE_SIZE,
 			       FUSE_READDIR);
 	}
-	fuse_lock_inode(inode);
+	locked = fuse_lock_inode(inode);
 	fuse_request_send(fc, req);
-	fuse_unlock_inode(inode);
+	fuse_unlock_inode(inode, locked);
 	nbytes = req->out.args[0].size;
 	err = req->out.h.error;
 	fuse_put_request(fc, req);
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -974,8 +974,8 @@ int fuse_do_setattr(struct dentry *dentr
 
 void fuse_set_initialized(struct fuse_conn *fc);
 
-void fuse_unlock_inode(struct inode *inode);
-void fuse_lock_inode(struct inode *inode);
+void fuse_unlock_inode(struct inode *inode, bool locked);
+bool fuse_lock_inode(struct inode *inode);
 
 int fuse_setxattr(struct inode *inode, const char *name, const void *value,
 		  size_t size, int flags);
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -357,15 +357,21 @@ int fuse_reverse_inval_inode(struct supe
 	return 0;
 }
 
-void fuse_lock_inode(struct inode *inode)
+bool fuse_lock_inode(struct inode *inode)
 {
-	if (!get_fuse_conn(inode)->parallel_dirops)
+	bool locked = false;
+
+	if (!get_fuse_conn(inode)->parallel_dirops) {
 		mutex_lock(&get_fuse_inode(inode)->mutex);
+		locked = true;
+	}
+
+	return locked;
 }
 
-void fuse_unlock_inode(struct inode *inode)
+void fuse_unlock_inode(struct inode *inode, bool locked)
 {
-	if (!get_fuse_conn(inode)->parallel_dirops)
+	if (locked)
 		mutex_unlock(&get_fuse_inode(inode)->mutex);
 }