Re: [BUG] RCU stall in blk_mq_timeout_work (potentially a regression in 6.19.7 or 6.19.8)

[Benjamin Eikel]

Hi Thorsten and others,

Am Dienstag, 31. März 2026, 10:32:30 Mitteleuropäische Sommerzeit schrieb Thorsten Leemhuis:
> [CCing the regressions list and Jens, the Block layer maintainer, who
> might have an idea what's wrong here and/or how to best debug it]
> 
> On 3/27/26 20:28, Benjamin Eikel wrote:
> > Dear Linux kernel developers,
> > 
> > I experience repeated RCU stalls in blk_mq_timeout_work causing system
> > freezes. The NVMe drive (more info at the end) shows no errors or
> > controller resets in the kernel log: the hardware appears healthy. No I/O
> > timeout messages precede the stalls.
> > 
> > The stall cascades: khugepaged blocks on __lru_add_drain_all waiting for a
> > workqueue flush that cannot complete, and additional kworkers block on a
> > mutex held by the kblockd rescuer thread. The system becomes noticeably
> > unresponsive and then I notice it in the logs.
> > 
> > In case the logs are too condensed, please tell me and I can provide them
> > fully. If you have pointers on how this could be reproduced and a commit
> > range,> 
> >  I could try to bisect the problem.
> 
> A bisection with mainline would help a lot. Maybe doing it between
> 6.19.6 and 6.19.8 (which would be quicker) might work out, too (but
> normally it's best to focus on mainline).

I have not yet done this because I do not have a reliable way to reproduce it.

> 
> And this is not my area of expertise, but I noticed that some RCU fixes
> went into -rc6, so might be wise to try that first.

I encountered this issue with -rc6 (see log below, same config). I suspect that
this only happens after the notebook has previously been suspended.

Kind regards
Benjamin

== Trace from Linux version 7.0.0-rc6 (1–2 April) ==

Apr 01 20:36:59 thinkpad-benjamin kernel: Linux version 7.0.0-rc6-70-rc6 (benjamin@thinkpad-benjamin) (gcc (Debian 15.2.0-16) 15.2.0, GNU ld (GNU Binutils for Debian) 2.46) #24 SMP PREEMPT_DYNAMIC Tue Mar 31 14:36:43 CEST 2026
Apr 01 20:36:59 thinkpad-benjamin kernel: Command line: BOOT_IMAGE=/vmlinuz-7.0.0-rc6-70-rc6 root=/dev/mapper/thinkpad--benjamin--vg-root ro quiet
[…]
Apr 01 21:06:54 thinkpad-benjamin kernel: PM: suspend entry (s2idle)
[…]
Apr 02 08:44:55 thinkpad-benjamin kernel: PM: suspend exit
[…]
Apr 02 08:50:59 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 120 seconds.
Apr 02 08:50:59 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:50:59 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:50:59 thinkpad-benjamin kernel: task:khugepaged      state:D stack:0     pid:138   tgid:138   ppid:2      task_flags:0x200040 flags:0x00080000
Apr 02 08:50:59 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:50:59 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:50:59 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:50:59 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:50:59 thinkpad-benjamin kernel:  schedule_timeout+0xbd/0x100
Apr 02 08:50:59 thinkpad-benjamin kernel:  wait_for_completion+0x8a/0x140
Apr 02 08:50:59 thinkpad-benjamin kernel:  __flush_work+0x276/0x3b0
Apr 02 08:50:59 thinkpad-benjamin kernel:  ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel:  __lru_add_drain_all+0x198/0x2b0
Apr 02 08:50:59 thinkpad-benjamin kernel:  khugepaged+0x65/0x950
Apr 02 08:50:59 thinkpad-benjamin kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel:  ? __pfx_khugepaged+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:50:59 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:50:59 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:50:59 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 241 seconds.
Apr 02 08:53:00 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:53:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:53:00 thinkpad-benjamin kernel: task:khugepaged      state:D stack:0     pid:138   tgid:138   ppid:2      task_flags:0x200040 flags:0x00080000
Apr 02 08:53:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:53:00 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:53:00 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:53:00 thinkpad-benjamin kernel:  schedule_timeout+0xbd/0x100
Apr 02 08:53:00 thinkpad-benjamin kernel:  wait_for_completion+0x8a/0x140
Apr 02 08:53:00 thinkpad-benjamin kernel:  __flush_work+0x276/0x3b0
Apr 02 08:53:00 thinkpad-benjamin kernel:  ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel:  __lru_add_drain_all+0x198/0x2b0
Apr 02 08:53:00 thinkpad-benjamin kernel:  khugepaged+0x65/0x950
Apr 02 08:53:00 thinkpad-benjamin kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel:  ? __pfx_khugepaged+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:53:00 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:53:00 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:53:00 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 362 seconds.
Apr 02 08:55:00 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:55:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:55:00 thinkpad-benjamin kernel: task:khugepaged      state:D stack:0     pid:138   tgid:138   ppid:2      task_flags:0x200040 flags:0x00080000
Apr 02 08:55:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:55:00 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:55:00 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:55:00 thinkpad-benjamin kernel:  schedule_timeout+0xbd/0x100
Apr 02 08:55:00 thinkpad-benjamin kernel:  wait_for_completion+0x8a/0x140
Apr 02 08:55:00 thinkpad-benjamin kernel:  __flush_work+0x276/0x3b0
Apr 02 08:55:00 thinkpad-benjamin kernel:  ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel:  __lru_add_drain_all+0x198/0x2b0
Apr 02 08:55:00 thinkpad-benjamin kernel:  khugepaged+0x65/0x950
Apr 02 08:55:00 thinkpad-benjamin kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel:  ? __pfx_khugepaged+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:55:00 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:55:00 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:55:00 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 483 seconds.
Apr 02 08:57:01 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:57:01 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:57:01 thinkpad-benjamin kernel: task:khugepaged      state:D stack:0     pid:138   tgid:138   ppid:2      task_flags:0x200040 flags:0x00080000
Apr 02 08:57:01 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:57:01 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:57:01 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:57:01 thinkpad-benjamin kernel:  schedule_timeout+0xbd/0x100
Apr 02 08:57:01 thinkpad-benjamin kernel:  wait_for_completion+0x8a/0x140
Apr 02 08:57:01 thinkpad-benjamin kernel:  __flush_work+0x276/0x3b0
Apr 02 08:57:01 thinkpad-benjamin kernel:  ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel:  __lru_add_drain_all+0x198/0x2b0
Apr 02 08:57:01 thinkpad-benjamin kernel:  khugepaged+0x65/0x950
Apr 02 08:57:01 thinkpad-benjamin kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel:  ? __pfx_khugepaged+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:57:01 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:57:01 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:57:01 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:14  state:D stack:0     pid:126   tgid:126   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel:  idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel:  process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R  running task     stack:0     pid:139   tgid:139   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel:  set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 604 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:khugepaged      state:D stack:0     pid:138   tgid:138   ppid:2      task_flags:0x200040 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_timeout+0xbd/0x100
Apr 02 08:59:02 thinkpad-benjamin kernel:  wait_for_completion+0x8a/0x140
Apr 02 08:59:02 thinkpad-benjamin kernel:  __flush_work+0x276/0x3b0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  __lru_add_drain_all+0x198/0x2b0
Apr 02 08:59:02 thinkpad-benjamin kernel:  khugepaged+0x65/0x950
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_khugepaged+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:16  state:D stack:0     pid:5369  tgid:5369  ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel:  idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel:  process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R  running task     stack:0     pid:139   tgid:139   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:12  state:D stack:0     pid:9088  tgid:9088  ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel:  idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel:  process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R  running task     stack:0     pid:139   tgid:139   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel:  set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:22  state:D stack:0     pid:10249 tgid:10249 ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel:  idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel:  process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R  running task     stack:0     pid:139   tgid:139   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel:  set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel:       Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:31  state:D stack:0     pid:10258 tgid:10258 ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel:  idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel:  process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel:  worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R  running task     stack:0     pid:139   tgid:139   ppid:2      task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel:  <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel:  ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel:  </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[…]
Apr 02 09:05:10 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 2583 jiffies old.
Apr 02 09:05:40 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 10136 jiffies old.
[…]
Apr 02 09:07:11 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 32832 jiffies old.
[…]
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Sync
Apr 02 09:08:09 thinkpad-benjamin kernel: Emergency Sync complete
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Remount R/O