Re: slab-out-of-bounds in iov_iter_revert()

From: Qian Cai
Date: Thu Sep 17 2020 - 10:11:35 EST

Next message: Hou Tao: "[PATCH 2/2] locktorture: call percpu_free_rwsem() to do percpu-rwsem cleanup"
Previous message: Ard Biesheuvel: "Re: [PATCH v2 2/2] ARM: support PHYS_OFFSET minimum aligned at 64KiB boundary"
In reply to: Al Viro: "Re: slab-out-of-bounds in iov_iter_revert()"
Next in thread: Al Viro: "Re: slab-out-of-bounds in iov_iter_revert()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Thu, 2020-09-17 at 03:14 +0100, Al Viro wrote:
> On Thu, Sep 17, 2020 at 03:04:40AM +0100, Al Viro wrote:
> > On Wed, Sep 16, 2020 at 05:09:49PM -0400, Qian Cai wrote:
> > > On Sat, 2020-09-12 at 00:55 +0100, Al Viro wrote:
> > > > On Fri, Sep 11, 2020 at 05:59:04PM -0400, Qian Cai wrote:
> > > > > Super easy to reproduce on today's mainline by just fuzzing for a few
> > > > > minutes
> > > > > on virtiofs (if it ever matters). Any thoughts?
> > > >
> > > > Usually happens when ->direct_IO() fucks up and reports the wrong amount
> > > > of data written/read. We had several bugs like that in the past - see
> > > > e.g. 85128b2be673 (fix nfs O_DIRECT advancing iov_iter too much).
> > > >
> > > > Had there been any recent O_DIRECT-related patches on the filesystems
> > > > involved?
> > >
> > > This is only reproducible using FUSE/virtiofs so far, so I will stare at
> > > fuse_direct_IO() until someone can beat me to it.
> >
> > What happens there is that it tries to play with iov_iter_truncate() in
> > ->direct_IO() without a corresponding iov_iter_reexpand(). Could you
> > check if the following helps?
>
> Gyah... Sorry, that should be
>
> Signed-off-by: Al Viro <viro@xxxxxxxxxxxxxxxxxx>

This is now triggering:

[ 81.942804] WARNING: CPU: 24 PID: 1545 at lib/iov_iter.c:1084 iov_iter_revert+0x245/0x8c0
[ 81.942805] Modules linked in: af_key mpls_router ip_tunnel hidp cmtp kernelcapi bnep rfcomm bluetooth ecdh_generic ecc can_bcm can_raw can pptp gre l2tp_ppp l2tp_netlink l2tp_core ip6_udp_tunnel udp_tunnel pppoe pppox ppp_generic slhc crypto_user ib_core nfnetlink scsi_transport_iscsi atm sctp rfkill nls_utf8 isofs intel_rapl_msr intel_rapl_common sb_edac kvm_intel kvm bochs_drm drm_vram_helper irqbypass drm_ttm_helper ttm rapl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_piix4 joydev pcspkr vfat fat ip_tables xfs libcrc32c sr_mod sd_mod cdrom t10_pi sg ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix virtiofs libata ghash_clmulni_intel fuse e1000 serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod
[ 81.942870] CPU: 24 PID: 1545 Comm: trinity-c0 Not tainted 5.9.0-rc5-iter+ #2
[ 81.942871] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 81.942879] RIP: 0010:iov_iter_revert+0x245/0x8c0
[ 81.942882] Code: 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 ce 05 00 00 49 29 f5 48 89 6b 18 4c 89 6b 08 48 83 c4 30 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b e9 3e ff ff ff 48 b8 00 00 00 00 00 fc ff df 48 8d 7b 18 48
[ 81.942884] RSP: 0018:ffff8886bf637b60 EFLAGS: 00010286
[ 81.942886] RAX: 000000000000028b RBX: ffff8886bf637dc8 RCX: 1ffff110d7ec6faf
[ 81.942888] RDX: dffffc0000000000 RSI: ffffffffbe4a754d RDI: ffff8886bf637d68
[ 81.942889] RBP: ffff8886bf637d68 R08: 0000000000000004 R09: ffffed10d7ec6ee2
[ 81.942890] R10: 0000000000000003 R11: ffffed10d7ec6ee2 R12: 0000000000000000
[ 81.942891] R13: ffff8886eaa62d80 R14: ffff8886bf637dd0 R15: ffff8886bf637d78
[ 81.942893] FS: 00007fc78f78d740(0000) GS:ffff8887d2000000(0000) knlGS:0000000000000000
[ 81.942897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 81.942898] CR2: 0000000000000008 CR3: 00000007cadba005 CR4: 0000000000170ee0
[ 81.942899] Call Trace:
[ 81.942909] generic_file_read_iter+0x23b/0x4b0
[ 81.942918] fuse_file_read_iter+0x280/0x4e0 [fuse]
[ 81.942931] ? fuse_direct_IO+0xd30/0xd30 [fuse]
[ 81.942949] ? _raw_spin_lock_irqsave+0x80/0xe0
[ 81.942957] ? timerqueue_add+0x15e/0x280
[ 81.942960] ? _raw_spin_lock_irqsave+0x80/0xe0
[ 81.942966] new_sync_read+0x3b7/0x620
[ 81.942968] ? __ia32_sys_llseek+0x2e0/0x2e0
[ 81.942971] ==================================================================
[ 81.942984] BUG: KASAN: slab-out-of-bounds in iov_iter_revert+0x693/0x8c0
[ 81.942995] ? hrtimer_start_range_ns+0x495/0x900
[ 81.942997] ? hrtimer_try_to_cancel+0x6d/0x330
[ 81.943002] Read of size 8 at addr ffff88867e6a6ff8 by task trinity-c21/1600
[ 81.943005] ? hrtimer_forward+0x1b0/0x1b0

[ 81.943026] ? __fsnotify_update_child_dentry_flags.part.12+0x290/0x290
[ 81.943030] ? _cond_resched+0x15/0x30
[ 81.943041] ? __inode_security_revalidate+0x9d/0xc0
[ 81.943047] CPU: 12 PID: 1600 Comm: trinity-c21 Not tainted 5.9.0-rc5-iter+ #2
[ 81.943050] Hardware name: Red Hat KVM, BIOS 1.14.0-1.module+el8.3.0+7638+07cf13d2 04/01/2014
[ 81.943053] vfs_read+0x22b/0x460
[ 81.943056] ksys_read+0xe5/0x1b0
[ 81.943058] Call Trace:
[ 81.943061] ? vfs_write+0x5e0/0x5e0
[ 81.943069] dump_stack+0x7c/0xb0
[ 81.943074] do_syscall_64+0x33/0x40
[ 81.943077] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 81.943081] RIP: 0033:0x7fc78f09d78d
[ 81.943084] ? iov_iter_revert+0x693/0x8c0
[ 81.943097] print_address_description.constprop.7+0x1e/0x230
[ 81.943100] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 01 48
[ 81.943117] ? kmsg_dump_rewind_nolock+0xd9/0xd9
[ 81.943119] RSP: 002b:00007ffe01a9e168 EFLAGS: 00000246
[ 81.943128] ? _raw_write_lock_irqsave+0xe0/0xe0
[ 81.943129] ORIG_RAX: 0000000000000000
[ 81.943133] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc78f09d78d
[ 81.943136] RDX: 000000000000028b RSI: 00007fc78d282000 RDI: 00000000000000f2
[ 81.943140] ? iov_iter_revert+0x693/0x8c0
[ 81.943142] RBP: 0000000000000000 R08: 19d504e75f9b0c9b R09: 00000000000000ff
[ 81.943146] ? iov_iter_revert+0x693/0x8c0
[ 81.943148] R10: 00000000d9d9d9d9 R11: 0000000000000246 R12: 0000000000000002
[ 81.943153] kasan_report.cold.9+0x37/0x7c
[ 81.943155] R13: 00007fc78f786058 R14: 00007fc78f78d6c0 R15: 00007fc78f786000
[ 81.943158] ? iov_iter_revert+0x693/0x8c0
[ 81.943161] iov_iter_revert+0x693/0x8c0
[ 81.943163] ---[ end trace ee32e92b96589675 ]---
[ 81.943172] ? dentry_needs_remove_privs.part.30+0x40/0x40
[ 81.943181] ? generic_write_checks+0x1d2/0x3d0
[ 81.943185] generic_file_direct_write+0x2ed/0x430
[ 81.943195] fuse_file_write_iter+0x5d8/0xb10 [fuse]
[ 81.943209] ? fuse_perform_write+0xed0/0xed0 [fuse]
[ 81.943215] ? kasan_unpoison_shadow+0x30/0x40
[ 81.943221] do_iter_readv_writev+0x3a6/0x700
[ 81.943224] ? no_seek_end_llseek_size+0x20/0x20
[ 81.943228] do_iter_write+0x12f/0x5f0
[ 81.943233] ? timerqueue_add+0x15e/0x280
[ 81.943236] vfs_writev+0x172/0x2d0
[ 81.943240] ? vfs_iter_write+0xc0/0xc0
[ 81.943245] ? hrtimer_start_range_ns+0x495/0x900
[ 81.943248] ? hrtimer_run_softirq+0x210/0x210
[ 81.943252] ? _raw_spin_lock_irq+0x7b/0xd0
[ 81.943256] ? _raw_write_unlock_irqrestore+0x50/0x50
[ 81.943269] ? perf_syscall_enter+0x136/0x8a0
[ 81.943276] ? perf_call_bpf_enter.isra.21+0x1a0/0x1a0
[ 81.943283] ? alarm_setitimer+0xa0/0x110
[ 81.943287] do_pwritev+0x151/0x200
[ 81.943291] ? __ia32_sys_writev+0xb0/0xb0
[ 81.943296] do_syscall_64+0x33/0x40
[ 81.943301] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 81.943306] RIP: 0033:0x7fc78f09d78d
[ 81.943312] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d cb 56 2c 00 f7 d8 64 89 01 48
[ 81.943314] RSP: 002b:00007ffe01a9e168 EFLAGS: 00000246 ORIG_RAX: 0000000000000148
[ 81.943318] RAX: ffffffffffffffda RBX: 0000000000000148 RCX: 00007fc78f09d78d
[ 81.943320] RDX: 000000000000004b RSI: 0000000001ae7790 RDI: 00000000000000f2
[ 81.943323] RBP: 0000000000000148 R08: 0000000075f74000 R09: 0000000000000001
[ 81.943325] R10: 00830624a1148006 R11: 0000000000000246 R12: 0000000000000002
[ 81.943328] R13: 00007fc78f6f3058 R14: 00007fc78f78d6c0 R15: 00007fc78f6f3000

[ 81.943333] Allocated by task 0:
[ 81.943334] (stack is not available)

[ 81.943339] The buggy address belongs to the object at ffff88867e6a6000
which belongs to the cache kmalloc-2k of size 2048
[ 81.943342] The buggy address is located 2040 bytes to the right of
2048-byte region [ffff88867e6a6000, ffff88867e6a6800)
[ 81.943344] The buggy address belongs to the page:
[ 81.943350] page:000000007cdca305 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x67e6a0
[ 81.943354] head:000000007cdca305 order:3 compound_mapcount:0 compound_pincount:0
[ 81.943358] flags: 0x17ffffc0010200(slab|head)
[ 81.943365] raw: 0017ffffc0010200 dead000000000100 dead000000000122 ffff888107c4f0c0
[ 81.943370] raw: 0000000000000000 0000000080080008 00000001ffffffff 0000000000000000
[ 81.943372] page dumped because: kasan: bad access detected

[ 81.943374] Memory state around the buggy address:
[ 81.943379] ffff88867e6a6e80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 81.943382] ffff88867e6a6f00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 81.943384] >ffff88867e6a6f80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 81.943386] ^
[ 81.943388] ffff88867e6a7000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 81.943400] ffff88867e6a7080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 81.943402] ==================================================================
[ 81.943403] Disabling lock debugging due to kernel taint

> ---
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index 6611ef3269a8..92de6b9b06b0 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -3095,7 +3095,7 @@ fuse_direct_IO(struct kiocb *iocb, struct iov_iter
> *iter)
> loff_t pos = 0;
> struct inode *inode;
> loff_t i_size;
> - size_t count = iov_iter_count(iter);
> + size_t count = iov_iter_count(iter), shortened;
> loff_t offset = iocb->ki_pos;
> struct fuse_io_priv *io;
>
> @@ -3111,7 +3111,8 @@ fuse_direct_IO(struct kiocb *iocb, struct iov_iter
> *iter)
> if (offset >= i_size)
> return 0;
> iov_iter_truncate(iter, fuse_round_up(ff->fc, i_size - offset));
> - count = iov_iter_count(iter);
> + shortened = count - iov_iter_count(iter);
> + count -= shortened;
> }
>
> io = kmalloc(sizeof(struct fuse_io_priv), GFP_KERNEL);
> @@ -3177,6 +3178,7 @@ fuse_direct_IO(struct kiocb *iocb, struct iov_iter
> *iter)
> else if (ret < 0 && offset + count > i_size)
> fuse_do_truncate(file);
> }
> + iov_iter_reexpand(iter, iov_iter_count(iter) + shortened);
>
> return ret;
> }
>

Next message: Hou Tao: "[PATCH 2/2] locktorture: call percpu_free_rwsem() to do percpu-rwsem cleanup"
Previous message: Ard Biesheuvel: "Re: [PATCH v2 2/2] ARM: support PHYS_OFFSET minimum aligned at 64KiB boundary"
In reply to: Al Viro: "Re: slab-out-of-bounds in iov_iter_revert()"
Next in thread: Al Viro: "Re: slab-out-of-bounds in iov_iter_revert()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]