Re: [syzbot] [mm?] INFO: task hung in exit_mmap

[Liam R. Howlett]

* Sasha Levin <sashal@xxxxxxxxxx> [241013 09:29]:
> On Thu, Oct 10, 2024 at 04:28:18PM +0100, Lorenzo Stoakes wrote:
> > On Thu, Oct 10, 2024 at 08:19:28AM -0700, syzbot wrote:
> > > Hello,
> > > 
> > > syzbot found the following issue on:
> > > 
> > > HEAD commit:    d3d1556696c1 Merge tag 'mm-hotfixes-stable-2024-10-09-15-4..
> > > git tree:       upstream
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=10416fd0580000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=7a3fccdd0bb995
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=39bc767144c55c8db0ea
> > > compiler:       Debian clang version 15.0.6, GNU ld (GNU Binutils for Debian) 2.40
> > > 
> > > Unfortunately, I don't have any reproducer for this issue yet.
> > > 
> > > Downloadable assets:
> > > disk image: https://storage.googleapis.com/syzbot-assets/0600b551e610/disk-d3d15566.raw.xz
> > > vmlinux: https://storage.googleapis.com/syzbot-assets/d59d43ed3976/vmlinux-d3d15566.xz
> > > kernel image: https://storage.googleapis.com/syzbot-assets/e686a3e7e0d6/bzImage-d3d15566.xz
> > > 
> > > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > > Reported-by: syzbot+39bc767144c55c8db0ea@xxxxxxxxxxxxxxxxxxxxxxxxx
> > > 
> > > INFO: task syz.3.917:7739 blocked for more than 146 seconds.
> > >       Not tainted 6.12.0-rc2-syzkaller-00074-gd3d1556696c1 #0
> > > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > task:syz.3.917       state:D stack:23808 pid:7739  tgid:7739  ppid:5232   flags:0x00004000
> > > Call Trace:
> > >  <TASK>
> > >  context_switch kernel/sched/core.c:5322 [inline]
> > >  __schedule+0x1843/0x4ae0 kernel/sched/core.c:6682
> > >  __schedule_loop kernel/sched/core.c:6759 [inline]
> > >  schedule+0x14b/0x320 kernel/sched/core.c:6774
> > >  schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6831
> > >  rwsem_down_write_slowpath+0xeee/0x13b0 kernel/locking/rwsem.c:1176
> > >  __down_write_common kernel/locking/rwsem.c:1304 [inline]
> > >  __down_write kernel/locking/rwsem.c:1313 [inline]
> > >  down_write+0x1d7/0x220 kernel/locking/rwsem.c:1578
> > >  mmap_write_lock include/linux/mmap_lock.h:106 [inline]
> > >  exit_mmap+0x2bd/0xc40 mm/mmap.c:1872
> > 
> > Hmm, task freezing up or system becoming unstable/locked up is reminsecent
> > of the maple tree bug I fixed in [0], which is still in the unstable hotfix
> > branch.
> > 
> > This is likely not going to repro as it's quite heisenbug-ish to trigger
> > and the failures are like this - somewhat disconnected from the cause, so
> > not sure if there is any case to speed this to Linus's tree.
> > 
> > On the other hand it's a pretty serious problem for stability and likely to
> > continue to manifest in nasty ways like this.
> > 
> > Can't be 100% sure this is the cause, but seems likely.
> > 
> > [0]:https://lore.kernel.org/linux-mm/48b349a2a0f7c76e18772712d0997a5e12ab0a3b.1728314403.git.lorenzo.stoakes@xxxxxxxxxx/
> 
> On my Debian build box, running a 6.1 kernel, I've started hitting a
> similar issue:
> 
> Oct 12 17:24:01 debian kernel: INFO: task sed:3557356 blocked for more than 1208 seconds.
> Oct 12 17:24:01 debian kernel:       Not tainted 6.1.0-26-amd64 #1 Debian 6.1.112-1
> Oct 12 17:24:01 debian kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Oct 12 17:24:01 debian kernel: task:sed             state:D stack:0     pid:3557356 ppid:1      flags:0x00000002
> Oct 12 17:24:01 debian kernel: Call Trace:
> Oct 12 17:24:01 debian kernel:  <TASK>
> Oct 12 17:24:01 debian kernel:  __schedule+0x34d/0x9e0
> Oct 12 17:24:01 debian kernel:  schedule+0x5a/0xd0
> Oct 12 17:24:01 debian kernel:  rwsem_down_write_slowpath+0x311/0x6d0
> Oct 12 17:24:01 debian kernel:  exit_mmap+0xf6/0x2f0
> Oct 12 17:24:01 debian kernel:  __mmput+0x3e/0x130
> Oct 12 17:24:01 debian kernel:  do_exit+0x2fc/0xaf0
> Oct 12 17:24:01 debian kernel:  do_group_exit+0x2d/0x80
> Oct 12 17:24:01 debian kernel:  __x64_sys_exit_group+0x14/0x20
> Oct 12 17:24:01 debian kernel:  do_syscall_64+0x55/0xb0
> Oct 12 17:24:01 debian kernel:  ? do_fault+0x1a4/0x410
> Oct 12 17:24:01 debian kernel:  ? __handle_mm_fault+0x660/0xfa0
> Oct 12 17:24:01 debian kernel:  ? exit_to_user_mode_prepare+0x40/0x1e0
> Oct 12 17:24:01 debian kernel:  ? handle_mm_fault+0xdb/0x2d0
> Oct 12 17:24:01 debian kernel:  ? do_user_addr_fault+0x1b0/0x550
> Oct 12 17:24:01 debian kernel:  ? exit_to_user_mode_prepare+0x40/0x1e0
> Oct 12 17:24:01 debian kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
> Oct 12 17:24:01 debian kernel: RIP: 0033:0x7f797d75a349
> Oct 12 17:24:01 debian kernel: RSP: 002b:00007fff37f0d3c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
> Oct 12 17:24:01 debian kernel: RAX: ffffffffffffffda RBX: 00007f797d8549e0 RCX: 00007f797d75a349
> Oct 12 17:24:01 debian kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
> Oct 12 17:24:01 debian kernel: RBP: 0000000000000000 R08: fffffffffffffe98 R09: 00007fff37f0d2df
> Oct 12 17:24:01 debian kernel: R10: 00007fff37f0d240 R11: 0000000000000246 R12: 00007f797d8549e0
> Oct 12 17:24:01 debian kernel: R13: 00007f797d85a2e0 R14: 0000000000000002 R15: 00007f797d85a2c8
> Oct 12 17:24:01 debian kernel:  </TASK>
> 
> It reproduces fairly easily during a kernel build...
> 
> It doesn't sound like the same issue you're pointing out, right Lorenzo?

It could be.  I suspect there has been a change recently that has
made the bug possible - although, I've not put effort into finding out
yet if that is true.  If the bug existed for a long time (probably since
I fixed the live locking issue in 6.4 that was backported), then you
could be hitting it.

It is a single line fix.  If it happens frequently enough, you could try
it - this fix will go through the backporting route once it lands.

Although, I am not sure it has much to do with the maple tree as I don't
think anyone should have the mm to take the mmap write lock.  If we were
stuck in the maple tree somehow, the mm wouldn't reach the exit_mmap()
path - unless I missed something?

If you can dump the running tasks when you hit it, we could get a clue
from the (probably numerous) backtraces?

Thanks,
Liam