[v6.12] WARNING: at kernel/sched/deadline.c:1995 enqueue_dl_entity (task blocked for more than 28262 seconds)
From: Ilya Maximets
Date: Wed Dec 04 2024 - 08:04:40 EST
Hi. It seems like I'm hitting some bug in the scheduler.
I'm running some tests with Open vSwitch on v6.12 kernel and some time
5 to 8 hours down the line I'm getting task blocked splats and I also
have a WARNING triggered in the scheduler code right before that:
Dec 3 22:19:55 kernel: WARNING: CPU: 27 PID: 3391271 at kernel/sched/deadline.c:1995 enqueue_dl_entity
I have a lot of processes (kernel threads and userpsace threads) stuck
in DN, Ds, D+ and D states. It feels like IO tasks are being scheduled,
but scheduler never picks them up or they are not being scheduled at all
for whatever reason, and threads waiting on these tasks are stuck.
Dec 3 22:22:45 kernel: INFO: task khugepaged:330 blocked for more than 122 seconds.
Dec 3 22:22:45 kernel: INFO: task ovs-monitor-ips:3479822 blocked for more than 122 seconds.
Dec 3 22:22:45 kernel: INFO: task mv:3483072 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: INFO: task khugepaged:330 blocked for more than 245 seconds.
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3479822 blocked for more than 245 seconds.
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3480383 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3481787 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3482631 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: INFO: task mv:3483072 blocked for more than 245 seconds.
Dec 3 22:26:51 kernel: INFO: task khugepaged:330 blocked for more than 368 seconds.
...
Dec 4 06:11:45 kernel: INFO: task khugepaged:330 blocked for more than 28262 seconds.
I have two separate instances where this behavior is reproduced. One is mostly
around file systems, the other was more severe as multiple kernel threads got
stuck in netlink code. The traces do not have much in common, except that most
of blocked tasks are in scheduling. The system is also idle, nothing is really
running. Some of these tasks are holding resources that make other tasks to
block on those resources as well.
I seem to be able to reproduce the issue, but it takes 5-8 hours to do so.
Best regards, Ilya Maximets.
Below are logs from two instances. The first one is from v6.12 + one small
unrelated patch for network namespaces. The second one is from pure v6.12,
but it's not decoded as I lost the vmlinux from that run, the system was also
completely unresponsive when the issue was hit.
=====================
THE FIST DECODED LOG:
=====================
Dec 3 22:19:55 kernel: WARNING: CPU: 27 PID: 3391271 at kernel/sched/deadline.c:1995 enqueue_dl_entity (kernel/sched/deadline.c:1995 (discriminator 1))
Dec 3 22:19:55 kernel: Modules linked in: vport_vxlan vxlan vport_gre ip_gre ip_tunnel gre vport_geneve geneve ip6_udp_tunnel udp_tunnel openvswitch nf_conncount nf_nat tls esp4 veth nfnetlink_cttimeout nfnetlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr intel_rapl_common rfkill intel_uncore_frequency_common skx_edac_common nfit libnvdimm kvm_intel kvm rapl vfat fat iTCO_wdt iTCO_vendor_support virtio_gpu virtio_dma_buf i2c_i801 drm_shmem_helper lpc_ich pcspkr i2c_smbus virtio_balloon drm_kms_helper joydev drm xfs libcrc32c ahci crct10dif_pclmul libahci crc32_pclmul virtio_net crc32c_intel libata ghash_clmulni_intel net_failover virtio_blk virtio_console failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip6_udp_tunnel]
Dec 3 22:19:55 kernel: CPU: 27 UID: 0 PID: 3391271 Comm: kworker/27:1 Kdump: loaded Not tainted 6.12.0+ #77
Dec 3 22:19:55 kernel: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.1-1.el9 04/01/2014
Dec 3 22:19:55 kernel: Workqueue: 0x0 (mm_percpu_wq)
Dec 3 22:19:55 kernel: RIP: 0010:enqueue_dl_entity (kernel/sched/deadline.c:1995 (discriminator 1))
Dec 3 22:19:55 kernel: Code: d2 0f 89 14 fd ff ff e9 0d fb ff ff 45 85 ed 0f
84 65 fd ff ff 5b 44 89 e6 48 89 ef 5d 41 5c 41
5d 41 5e 41 5f e9 76 c4 ff ff <0f> 0b e9 bd f9 ff
ff 0f 0b e9 1f fb ff ff 8b 83 b0 0a 00 00 48 8b
All code
========
0: d2 0f rorb %cl,(%rdi)
2: 89 14 fd ff ff e9 0d mov %edx,0xde9ffff(,%rdi,8)
9: fb sti
a: ff (bad)
b: ff 45 85 incl -0x7b(%rbp)
e: ed in (%dx),%eax
f: 0f 84 65 fd ff ff je 0xfffffffffffffd7a
15: 5b pop %rbx
16: 44 89 e6 mov %r12d,%esi
19: 48 89 ef mov %rbp,%rdi
1c: 5d pop %rbp
1d: 41 5c pop %r12
1f: 41 5d pop %r13
21: 41 5e pop %r14
23: 41 5f pop %r15
25: e9 76 c4 ff ff jmpq 0xffffffffffffc4a0
2a:* 0f 0b ud2 <-- trapping instruction
2c: e9 bd f9 ff ff jmpq 0xfffffffffffff9ee
31: 0f 0b ud2
33: e9 1f fb ff ff jmpq 0xfffffffffffffb57
38: 8b 83 b0 0a 00 00 mov 0xab0(%rbx),%eax
3e: 48 rex.W
3f: 8b .byte 0x8b
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2
2: e9 bd f9 ff ff jmpq 0xfffffffffffff9c4
7: 0f 0b ud2
9: e9 1f fb ff ff jmpq 0xfffffffffffffb2d
e: 8b 83 b0 0a 00 00 mov 0xab0(%rbx),%eax
14: 48 rex.W
15: 8b .byte 0x8b
Dec 3 22:19:55 kernel: RSP: 0018:ffffacba8d87fbb8 EFLAGS: 00010086
Dec 3 22:19:55 kernel: RAX: 0000000000000001 RBX: ffff8f5e3f7b65e8 RCX: 0000000000000000
Dec 3 22:19:55 kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8f5e3f7b65e8
Dec 3 22:19:55 kernel: RBP: ffff8f5e3f7b65e8 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:19:55 kernel: R10: ffff8f5e3f7b5d00 R11: ffff8f4f132ed610 R12: 0000000000000001
Dec 3 22:19:55 kernel: R13: 0000000000000001 R14: 00000000002dc6c0 R15: 0000000000000000
Dec 3 22:19:55 kernel: FS: 0000000000000000(0000) GS:ffff8f5e3f780000(0000) knlGS:0000000000000000
Dec 3 22:19:55 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 3 22:19:55 kernel: CR2: 00007faf752d93e0 CR3: 000000011879a001 CR4: 0000000000772ef0
Dec 3 22:19:55 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 3 22:19:55 kernel: DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400
Dec 3 22:19:55 kernel: PKRU: 55555554
Dec 3 22:19:55 kernel: Call Trace:
Dec 3 22:19:55 kernel: <TASK>
Dec 3 22:19:55 kernel: ? __warn (kernel/panic.c:748)
Dec 3 22:19:55 kernel: ? enqueue_dl_entity (kernel/sched/deadline.c:1995 (discriminator 1))
Dec 3 22:19:55 kernel: ? report_bug (lib/bug.c:201 lib/bug.c:219)
Dec 3 22:19:55 kernel: ? handle_bug (arch/x86/kernel/traps.c:285)
Dec 3 22:19:55 kernel: ? exc_invalid_op (arch/x86/kernel/traps.c:309 (discriminator 1))
Dec 3 22:19:55 kernel: ? asm_exc_invalid_op (./arch/x86/include/asm/idtentry.h:621)
Dec 3 22:19:55 kernel: ? enqueue_dl_entity (kernel/sched/deadline.c:1995 (discriminator 1))
Dec 3 22:19:55 kernel: dl_server_start (kernel/sched/deadline.c:1651)
Dec 3 22:19:55 kernel: enqueue_task_fair (kernel/sched/sched.h:2745 kernel/sched/fair.c:7048)
Dec 3 22:19:55 kernel: enqueue_task (kernel/sched/core.c:2020)
Dec 3 22:19:55 kernel: activate_task (kernel/sched/core.c:2069)
Dec 3 22:19:55 kernel: sched_balance_rq (kernel/sched/fair.c:9642 kernel/sched/fair.c:9676 kernel/sched/fair.c:11753)
Dec 3 22:19:55 kernel: sched_balance_newidle (kernel/sched/fair.c:12799)
Dec 3 22:19:55 kernel: pick_next_task_fair (kernel/sched/fair.c:8950)
Dec 3 22:19:55 kernel: __pick_next_task (kernel/sched/core.c:5972)
Dec 3 22:19:55 kernel: __schedule (kernel/sched/core.c:6647)
Dec 3 22:19:55 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:19:55 kernel: worker_thread (kernel/workqueue.c:3344)
Dec 3 22:19:55 kernel: ? __pfx_worker_thread (kernel/workqueue.c:3337)
Dec 3 22:19:55 kernel: kthread (kernel/kthread.c:389)
Dec 3 22:19:55 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:19:55 kernel: ret_from_fork (arch/x86/kernel/process.c:147)
Dec 3 22:19:55 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:19:55 kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
Dec 3 22:19:55 kernel: </TASK>
Dec 3 22:19:55 kernel: ---[ end trace 0000000000000000 ]---
Dec 3 22:19:55 kernel: ovs-p-13: entered promiscuous mode
Dec 3 22:22:45 kernel: INFO: task khugepaged:330 blocked for more than 122 seconds.
Dec 3 22:22:45 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:22:45 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:22:45 kernel: task:khugepaged state:D stack:0 pid:330 tgid:330 ppid:2 flags:0x00004000
Dec 3 22:22:45 kernel: Call Trace:
Dec 3 22:22:45 kernel: <TASK>
Dec 3 22:22:45 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:22:45 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:22:45 kernel: schedule_timeout (kernel/time/timer.c:2592)
Dec 3 22:22:45 kernel: ? kvm_sched_clock_read (arch/x86/kernel/kvmclock.c:91)
Dec 3 22:22:45 kernel: ? sched_clock (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/tsc.c:285)
Dec 3 22:22:45 kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Dec 3 22:22:45 kernel: ? __smp_call_single_queue (kernel/smp.c:115 kernel/smp.c:411)
Dec 3 22:22:45 kernel: __wait_for_common (kernel/sched/completion.c:95 kernel/sched/completion.c:116)
Dec 3 22:22:45 kernel: ? __pfx_schedule_timeout (kernel/time/timer.c:2577)
Dec 3 22:22:45 kernel: __flush_work (kernel/workqueue.c:4222)
Dec 3 22:22:45 kernel: ? __pfx_wq_barrier_func (kernel/workqueue.c:3718)
Dec 3 22:22:45 kernel: __lru_add_drain_all (mm/swap.c:873 (discriminator 3))
Dec 3 22:22:45 kernel: khugepaged (mm/khugepaged.c:2499 mm/khugepaged.c:2571)
Dec 3 22:22:45 kernel: ? __pfx_khugepaged (mm/khugepaged.c:2564)
Dec 3 22:22:45 kernel: kthread (kernel/kthread.c:389)
Dec 3 22:22:45 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:22:45 kernel: ret_from_fork (arch/x86/kernel/process.c:147)
Dec 3 22:22:45 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:22:45 kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
Dec 3 22:22:45 kernel: </TASK>
Dec 3 22:22:45 kernel: INFO: task ovs-monitor-ips:3479822 blocked for more than 122 seconds.
Dec 3 22:22:45 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:22:45 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:22:45 kernel: task:ovs-monitor-ips state:D stack:0 pid:3479822 tgid:3479822 ppid:1 flags:0x00000002
Dec 3 22:22:45 kernel: Call Trace:
Dec 3 22:22:45 kernel: <TASK>
Dec 3 22:22:45 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:22:45 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:22:45 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:22:45 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:22:45 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:22:45 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:22:45 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:22:45 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:22:45 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:22:45 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:22:45 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:22:45 kernel: ? poll_freewait (fs/select.c:140 (discriminator 3))
Dec 3 22:22:45 kernel: ? do_select (fs/select.c:612)
Dec 3 22:22:45 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:22:45 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:22:45 kernel: ? down_read (./arch/x86/include/asm/preempt.h:79 kernel/locking/rwsem.c:1246 kernel/locking/rwsem.c:1261 kernel/locking/rwsem.c:1526)
Dec 3 22:22:45 kernel: ? unmap_mapping_range (mm/memory.c:3873)
Dec 3 22:22:45 kernel: truncate_pagecache (mm/truncate.c:728)
Dec 3 22:22:45 kernel: xfs_setattr_size+0x139/0x410 xfs
Dec 3 22:22:45 kernel: xfs_vn_setattr+0x78/0x140 xfs
Dec 3 22:22:45 kernel: notify_change (fs/attr.c:503)
Dec 3 22:22:45 kernel: ? do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:22:45 kernel: do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:22:45 kernel: do_open (fs/namei.c:3395 fs/namei.c:3778)
Dec 3 22:22:45 kernel: path_openat (fs/namei.c:3933)
Dec 3 22:22:45 kernel: ? memcg_list_lru_alloc (mm/list_lru.c:475 mm/list_lru.c:489)
Dec 3 22:22:45 kernel: do_filp_open (fs/namei.c:3960)
Dec 3 22:22:45 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:22:45 kernel: ? __skb_try_recv_datagram (net/core/datagram.c:267)
Dec 3 22:22:45 kernel: ? kmem_cache_alloc_noprof (mm/slub.c:4115 mm/slub.c:4141)
Dec 3 22:22:45 kernel: do_sys_openat2 (fs/open.c:1415)
Dec 3 22:22:45 kernel: __x64_sys_openat (fs/open.c:1441)
Dec 3 22:22:45 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:22:45 kernel: ? unix_stream_recvmsg (net/unix/af_unix.c:2997)
Dec 3 22:22:45 kernel: ? __pfx_unix_stream_read_actor (net/unix/af_unix.c:2957)
Dec 3 22:22:45 kernel: ? sock_recvmsg (net/socket.c:1051 net/socket.c:1073)
Dec 3 22:22:45 kernel: ? __sys_recvfrom (net/socket.c:2265)
Dec 3 22:22:45 kernel: ? __pte_offset_map (./include/linux/pgtable.h:324 ./include/linux/pgtable.h:594 mm/pgtable-generic.c:289)
Dec 3 22:22:45 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:22:45 kernel: ? do_wp_page (./include/linux/vmstat.h:75 mm/memory.c:3263 mm/memory.c:3731)
Dec 3 22:22:45 kernel: ? __handle_mm_fault (mm/memory.c:5909)
Dec 3 22:22:45 kernel: ? syscall_exit_work (./include/linux/audit.h:357 kernel/entry/common.c:166)
Dec 3 22:22:45 kernel: ? __count_memcg_events (mm/memcontrol.c:573 mm/memcontrol.c:836)
Dec 3 22:22:45 kernel: ? handle_mm_fault (mm/memory.c:5951 mm/memory.c:6103)
Dec 3 22:22:45 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:22:45 kernel: RIP: 0033:0x7f8537cfd70b
Dec 3 22:22:45 kernel: RSP: 002b:00007fff841fec70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Dec 3 22:22:45 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f8537cfd70b
Dec 3 22:22:45 kernel: RDX: 0000000000080241 RSI: 00007f853707d290 RDI: 00000000ffffff9c
Dec 3 22:22:45 kernel: RBP: 00007f853707d290 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:22:45 kernel: R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000080241
Dec 3 22:22:45 kernel: R13: 00007f85371b4ea0 R14: 0000000000080241 R15: 0000000000000000
Dec 3 22:22:45 kernel: </TASK>
Dec 3 22:22:45 kernel: INFO: task mv:3483072 blocked for more than 122 seconds.
Dec 3 22:22:45 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:22:45 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:22:45 kernel: task:mv state:D stack:0 pid:3483072 tgid:3483072 ppid:3479428 flags:0x00000002
Dec 3 22:22:45 kernel: Call Trace:
Dec 3 22:22:45 kernel: <TASK>
Dec 3 22:22:45 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:22:45 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:22:45 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:22:45 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:22:45 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:22:45 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:22:45 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:22:45 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:22:45 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:22:45 kernel: ? xfs_iunlock+0x108/0x200 xfs
Dec 3 22:22:45 kernel: ? xfs_rename+0x368/0x990 xfs
Dec 3 22:22:45 kernel: ? fsnotify_move (./include/linux/fsnotify.h:72 ./include/linux/fsnotify.h:64 ./include/linux/fsnotify.h:238)
Dec 3 22:22:45 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:22:45 kernel: ? locked_inode_to_wb_and_lock_list (fs/fs-writeback.c:355)
Dec 3 22:22:45 kernel: evict (fs/inode.c:728)
Dec 3 22:22:45 kernel: ? fsnotify_destroy_marks (fs/notify/mark.c:923)
Dec 3 22:22:45 kernel: ? _atomic_dec_and_lock (./arch/x86/include/asm/atomic.h:67 ./include/linux/atomic/atomic-arch-fallback.h:2278 ./include/linux/atomic/atomic-instrumented.h:1384 lib/dec_and_lock.c:29)
Dec 3 22:22:45 kernel: __dentry_kill (fs/dcache.c:618)
Dec 3 22:22:45 kernel: dput (fs/dcache.c:857 fs/dcache.c:845)
Dec 3 22:22:45 kernel: do_renameat2 (fs/namei.c:5174)
Dec 3 22:22:45 kernel: __x64_sys_rename (fs/namei.c:5215)
Dec 3 22:22:45 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:22:45 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:22:45 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:22:45 kernel: RIP: 0033:0x7f3185a5aadb
Dec 3 22:22:45 kernel: RSP: 002b:00007ffe258c4548 EFLAGS: 00000246 ORIG_RAX: 0000000000000052
Dec 3 22:22:45 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3185a5aadb
Dec 3 22:22:45 kernel: RDX: 0000000000000025 RSI: 00007ffe258c5cc3 RDI: 00007ffe258c5cb7
Dec 3 22:22:45 kernel: RBP: 00007ffe258c48f0 R08: 00007ffe258c4670 R09: 00007ffe258c4ac0
Dec 3 22:22:45 kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000000011
Dec 3 22:22:45 kernel: R13: 0000000000000000 R14: 00007ffe258c5cc3 R15: 00007ffe258c4ac0
Dec 3 22:22:45 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task khugepaged:330 blocked for more than 245 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:khugepaged state:D stack:0 pid:330 tgid:330 ppid:2 flags:0x00004000
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: schedule_timeout (kernel/time/timer.c:2592)
Dec 3 22:24:48 kernel: ? kvm_sched_clock_read (arch/x86/kernel/kvmclock.c:91)
Dec 3 22:24:48 kernel: ? sched_clock (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/tsc.c:285)
Dec 3 22:24:48 kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Dec 3 22:24:48 kernel: ? __smp_call_single_queue (kernel/smp.c:115 kernel/smp.c:411)
Dec 3 22:24:48 kernel: __wait_for_common (kernel/sched/completion.c:95 kernel/sched/completion.c:116)
Dec 3 22:24:48 kernel: ? __pfx_schedule_timeout (kernel/time/timer.c:2577)
Dec 3 22:24:48 kernel: __flush_work (kernel/workqueue.c:4222)
Dec 3 22:24:48 kernel: ? __pfx_wq_barrier_func (kernel/workqueue.c:3718)
Dec 3 22:24:48 kernel: __lru_add_drain_all (mm/swap.c:873 (discriminator 3))
Dec 3 22:24:48 kernel: khugepaged (mm/khugepaged.c:2499 mm/khugepaged.c:2571)
Dec 3 22:24:48 kernel: ? __pfx_khugepaged (mm/khugepaged.c:2564)
Dec 3 22:24:48 kernel: kthread (kernel/kthread.c:389)
Dec 3 22:24:48 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:24:48 kernel: ret_from_fork (arch/x86/kernel/process.c:147)
Dec 3 22:24:48 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:24:48 kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3479822 blocked for more than 245 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:ovs-monitor-ips state:D stack:0 pid:3479822 tgid:3479822 ppid:1 flags:0x00000002
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:24:48 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:24:48 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:24:48 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:24:48 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:24:48 kernel: ? poll_freewait (fs/select.c:140 (discriminator 3))
Dec 3 22:24:48 kernel: ? do_select (fs/select.c:612)
Dec 3 22:24:48 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:24:48 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:24:48 kernel: ? down_read (./arch/x86/include/asm/preempt.h:79 kernel/locking/rwsem.c:1246 kernel/locking/rwsem.c:1261 kernel/locking/rwsem.c:1526)
Dec 3 22:24:48 kernel: ? unmap_mapping_range (mm/memory.c:3873)
Dec 3 22:24:48 kernel: truncate_pagecache (mm/truncate.c:728)
Dec 3 22:24:48 kernel: xfs_setattr_size+0x139/0x410 xfs
Dec 3 22:24:48 kernel: xfs_vn_setattr+0x78/0x140 xfs
Dec 3 22:24:48 kernel: notify_change (fs/attr.c:503)
Dec 3 22:24:48 kernel: ? do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_open (fs/namei.c:3395 fs/namei.c:3778)
Dec 3 22:24:48 kernel: path_openat (fs/namei.c:3933)
Dec 3 22:24:48 kernel: ? memcg_list_lru_alloc (mm/list_lru.c:475 mm/list_lru.c:489)
Dec 3 22:24:48 kernel: do_filp_open (fs/namei.c:3960)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:24:48 kernel: ? __skb_try_recv_datagram (net/core/datagram.c:267)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_noprof (mm/slub.c:4115 mm/slub.c:4141)
Dec 3 22:24:48 kernel: do_sys_openat2 (fs/open.c:1415)
Dec 3 22:24:48 kernel: __x64_sys_openat (fs/open.c:1441)
Dec 3 22:24:48 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? unix_stream_recvmsg (net/unix/af_unix.c:2997)
Dec 3 22:24:48 kernel: ? __pfx_unix_stream_read_actor (net/unix/af_unix.c:2957)
Dec 3 22:24:48 kernel: ? sock_recvmsg (net/socket.c:1051 net/socket.c:1073)
Dec 3 22:24:48 kernel: ? __sys_recvfrom (net/socket.c:2265)
Dec 3 22:24:48 kernel: ? __pte_offset_map (./include/linux/pgtable.h:324 ./include/linux/pgtable.h:594 mm/pgtable-generic.c:289)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:24:48 kernel: ? do_wp_page (./include/linux/vmstat.h:75 mm/memory.c:3263 mm/memory.c:3731)
Dec 3 22:24:48 kernel: ? __handle_mm_fault (mm/memory.c:5909)
Dec 3 22:24:48 kernel: ? syscall_exit_work (./include/linux/audit.h:357 kernel/entry/common.c:166)
Dec 3 22:24:48 kernel: ? __count_memcg_events (mm/memcontrol.c:573 mm/memcontrol.c:836)
Dec 3 22:24:48 kernel: ? handle_mm_fault (mm/memory.c:5951 mm/memory.c:6103)
Dec 3 22:24:48 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: RIP: 0033:0x7f8537cfd70b
Dec 3 22:24:48 kernel: RSP: 002b:00007fff841fec70 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Dec 3 22:24:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f8537cfd70b
Dec 3 22:24:48 kernel: RDX: 0000000000080241 RSI: 00007f853707d290 RDI: 00000000ffffff9c
Dec 3 22:24:48 kernel: RBP: 00007f853707d290 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:24:48 kernel: R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000080241
Dec 3 22:24:48 kernel: R13: 00007f85371b4ea0 R14: 0000000000080241 R15: 0000000000000000
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3480383 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:ovs-monitor-ips state:D stack:0 pid:3480383 tgid:3480383 ppid:1 flags:0x00000002
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:24:48 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:24:48 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:24:48 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:24:48 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:24:48 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:24:48 kernel: ? arch_stack_walk (arch/x86/kernel/stacktrace.c:24)
Dec 3 22:24:48 kernel: ? __is_insn_slot_addr (kernel/kprobes.c:299)
Dec 3 22:24:48 kernel: ? is_bpf_text_address (kernel/bpf/core.c:768)
Dec 3 22:24:48 kernel: ? kernel_text_address (kernel/extable.c:97 kernel/extable.c:94)
Dec 3 22:24:48 kernel: ? __kernel_text_address (kernel/extable.c:79)
Dec 3 22:24:48 kernel: ? unwind_get_return_address (arch/x86/kernel/unwind_orc.c:369 arch/x86/kernel/unwind_orc.c:364)
Dec 3 22:24:48 kernel: ? __pfx_stack_trace_consume_entry (kernel/stacktrace.c:83)
Dec 3 22:24:48 kernel: ? arch_stack_walk (arch/x86/kernel/stacktrace.c:26)
Dec 3 22:24:48 kernel: ? kvm_sched_clock_read (arch/x86/kernel/kvmclock.c:91)
Dec 3 22:24:48 kernel: ? local_clock_noinstr (kernel/sched/clock.c:301)
Dec 3 22:24:48 kernel: ? local_clock (./arch/x86/include/asm/preempt.h:94 kernel/sched/clock.c:316)
Dec 3 22:24:48 kernel: ? metadata_update_state (mm/kfence/core.c:313)
Dec 3 22:24:48 kernel: ? inode_init_once (fs/inode.c:405 fs/inode.c:431)
Dec 3 22:24:48 kernel: ? kfence_guarded_alloc (mm/kfence/core.c:502)
Dec 3 22:24:48 kernel: ? sock_alloc_inode (net/socket.c:307)
Dec 3 22:24:48 kernel: ? __kfence_alloc (mm/kfence/core.c:1136)
Dec 3 22:24:48 kernel: ? __kfence_alloc (mm/kfence/core.c:209 mm/kfence/core.c:1130)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_lru_noprof (mm/slub.c:4119 mm/slub.c:4153)
Dec 3 22:24:48 kernel: ? sock_alloc_inode (net/socket.c:307)
Dec 3 22:24:48 kernel: ? down_read (./arch/x86/include/asm/preempt.h:79 kernel/locking/rwsem.c:1246 kernel/locking/rwsem.c:1261 kernel/locking/rwsem.c:1526)
Dec 3 22:24:48 kernel: ? unmap_mapping_range (mm/memory.c:3873)
Dec 3 22:24:48 kernel: truncate_pagecache (mm/truncate.c:728)
Dec 3 22:24:48 kernel: xfs_setattr_size+0x139/0x410 xfs
Dec 3 22:24:48 kernel: xfs_vn_setattr+0x78/0x140 xfs
Dec 3 22:24:48 kernel: notify_change (fs/attr.c:503)
Dec 3 22:24:48 kernel: ? do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_open (fs/namei.c:3395 fs/namei.c:3778)
Dec 3 22:24:48 kernel: path_openat (fs/namei.c:3933)
Dec 3 22:24:48 kernel: do_filp_open (fs/namei.c:3960)
Dec 3 22:24:48 kernel: ? __pfx_unix_stream_read_actor (net/unix/af_unix.c:2957)
Dec 3 22:24:48 kernel: ? sock_recvmsg (net/socket.c:1051 net/socket.c:1073)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_noprof (mm/slub.c:4115 mm/slub.c:4141)
Dec 3 22:24:48 kernel: do_sys_openat2 (fs/open.c:1415)
Dec 3 22:24:48 kernel: __x64_sys_openat (fs/open.c:1441)
Dec 3 22:24:48 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? syscall_exit_work (./include/linux/audit.h:357 kernel/entry/common.c:166)
Dec 3 22:24:48 kernel: ? syscall_exit_to_user_mode (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:92 ./include/linux/entry-common.h:231 kernel/entry/common.c:206 kernel/entry/common.c:218)
Dec 3 22:24:48 kernel: ? do_syscall_64 (arch/x86/entry/common.c:102)
Dec 3 22:24:48 kernel: ? rseq_ip_fixup (kernel/rseq.c:257 kernel/rseq.c:291)
Dec 3 22:24:48 kernel: ? ktime_get_ts64 (kernel/time/timekeeping.c:195 (discriminator 3) kernel/time/timekeeping.c:395 (discriminator 3) kernel/time/timekeeping.c:403 (discriminator 3) kernel/time/timekeeping.c:983 (discriminator 3))
Dec 3 22:24:48 kernel: ? switch_fpu_return (arch/x86/kernel/fpu/context.h:49 arch/x86/kernel/fpu/context.h:76 arch/x86/kernel/fpu/core.c:787)
Dec 3 22:24:48 kernel: ? syscall_exit_to_user_mode (./arch/x86/include/asm/entry-common.h:58 ./arch/x86/include/asm/entry-common.h:65 ./include/linux/entry-common.h:330 kernel/entry/common.c:207 kernel/entry/common.c:218)
Dec 3 22:24:48 kernel: ? __pte_offset_map (./include/linux/pgtable.h:324 ./include/linux/pgtable.h:594 mm/pgtable-generic.c:289)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:24:48 kernel: ? do_wp_page (./include/linux/vmstat.h:75 mm/memory.c:3263 mm/memory.c:3731)
Dec 3 22:24:48 kernel: ? __handle_mm_fault (mm/memory.c:5909)
Dec 3 22:24:48 kernel: ? __count_memcg_events (mm/memcontrol.c:573 mm/memcontrol.c:836)
Dec 3 22:24:48 kernel: ? handle_mm_fault (mm/memory.c:5951 mm/memory.c:6103)
Dec 3 22:24:48 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: RIP: 0033:0x7f85c7afd70b
Dec 3 22:24:48 kernel: RSP: 002b:00007ffe53f6c870 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Dec 3 22:24:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f85c7afd70b
Dec 3 22:24:48 kernel: RDX: 0000000000080241 RSI: 00007f85c6d74290 RDI: 00000000ffffff9c
Dec 3 22:24:48 kernel: RBP: 00007f85c6d74290 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:24:48 kernel: R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000080241
Dec 3 22:24:48 kernel: R13: 00007f85c6e01ea0 R14: 0000000000080241 R15: 0000000000000000
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3481787 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:ovs-monitor-ips state:D stack:0 pid:3481787 tgid:3481787 ppid:1 flags:0x00000002
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:24:48 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:24:48 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:24:48 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:24:48 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:24:48 kernel: ? sock_alloc_inode (net/socket.c:307)
Dec 3 22:24:48 kernel: ? __kfence_alloc (mm/kfence/core.c:1136)
Dec 3 22:24:48 kernel: ? __kfence_alloc (mm/kfence/core.c:209 mm/kfence/core.c:1130)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_lru_noprof (mm/slub.c:4119 mm/slub.c:4153)
Dec 3 22:24:48 kernel: ? sock_alloc_inode (net/socket.c:307)
Dec 3 22:24:48 kernel: ? alloc_inode (fs/inode.c:265)
Dec 3 22:24:48 kernel: ? sock_alloc (net/socket.c:634)
Dec 3 22:24:48 kernel: ? do_accept (net/socket.c:1929)
Dec 3 22:24:48 kernel: ? __sys_accept4 (net/socket.c:1992 net/socket.c:2022)
Dec 3 22:24:48 kernel: ? __x64_sys_accept4 (net/socket.c:2033 net/socket.c:2030 net/socket.c:2030)
Dec 3 22:24:48 kernel: ? do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: ? __pfx_pollwake (fs/select.c:209)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? xfrm_state_mtu (net/xfrm/xfrm_state.c:2842 net/xfrm/xfrm_state.c:2824)
Dec 3 22:24:48 kernel: ? memcg_list_lru_alloc (mm/list_lru.c:475 mm/list_lru.c:489)
Dec 3 22:24:48 kernel: ? down_read (./arch/x86/include/asm/preempt.h:79 kernel/locking/rwsem.c:1246 kernel/locking/rwsem.c:1261 kernel/locking/rwsem.c:1526)
Dec 3 22:24:48 kernel: ? unmap_mapping_range (mm/memory.c:3873)
Dec 3 22:24:48 kernel: truncate_pagecache (mm/truncate.c:728)
Dec 3 22:24:48 kernel: xfs_setattr_size+0x139/0x410 xfs
Dec 3 22:24:48 kernel: xfs_vn_setattr+0x78/0x140 xfs
Dec 3 22:24:48 kernel: notify_change (fs/attr.c:503)
Dec 3 22:24:48 kernel: ? do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_open (fs/namei.c:3395 fs/namei.c:3778)
Dec 3 22:24:48 kernel: path_openat (fs/namei.c:3933)
Dec 3 22:24:48 kernel: do_filp_open (fs/namei.c:3960)
Dec 3 22:24:48 kernel: ? do_wp_page (./include/linux/vmstat.h:75 mm/memory.c:3263 mm/memory.c:3731)
Dec 3 22:24:48 kernel: ? __handle_mm_fault (mm/memory.c:5909)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_noprof (mm/slub.c:4115 mm/slub.c:4141)
Dec 3 22:24:48 kernel: do_sys_openat2 (fs/open.c:1415)
Dec 3 22:24:48 kernel: __x64_sys_openat (fs/open.c:1441)
Dec 3 22:24:48 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: RIP: 0033:0x7f79468fd70b
Dec 3 22:24:48 kernel: RSP: 002b:00007ffd27a6c700 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Dec 3 22:24:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f79468fd70b
Dec 3 22:24:48 kernel: RDX: 0000000000080241 RSI: 00007f7945b64290 RDI: 00000000ffffff9c
Dec 3 22:24:48 kernel: RBP: 00007f7945b64290 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:24:48 kernel: R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000080241
Dec 3 22:24:48 kernel: R13: 00007f7945bf2ea0 R14: 0000000000080241 R15: 0000000000000000
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task ovs-monitor-ips:3482631 blocked for more than 122 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:ovs-monitor-ips state:D stack:0 pid:3482631 tgid:3482631 ppid:1 flags:0x00000002
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:24:48 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:24:48 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:24:48 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:24:48 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:24:48 kernel: ? finish_task_switch.isra.0 (./arch/x86/include/asm/irqflags.h:42 ./arch/x86/include/asm/irqflags.h:97 kernel/sched/sched.h:1518 kernel/sched/core.c:5082 kernel/sched/core.c:5200)
Dec 3 22:24:48 kernel: ? __schedule (kernel/sched/core.c:6699)
Dec 3 22:24:48 kernel: ? xfrm_state_mtu (net/xfrm/xfrm_state.c:2842 net/xfrm/xfrm_state.c:2824)
Dec 3 22:24:48 kernel: ? schedule_hrtimeout_range_clock (kernel/time/hrtimer.c:1332 kernel/time/hrtimer.c:1449 kernel/time/hrtimer.c:2283)
Dec 3 22:24:48 kernel: ? remove_wait_queue (./include/linux/list.h:215 ./include/linux/list.h:229 ./include/linux/wait.h:207 kernel/sched/wait.c:55)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:24:48 kernel: ? poll_freewait (fs/select.c:140 (discriminator 3))
Dec 3 22:24:48 kernel: ? do_select (fs/select.c:612)
Dec 3 22:24:48 kernel: ? down_read (./arch/x86/include/asm/preempt.h:79 kernel/locking/rwsem.c:1246 kernel/locking/rwsem.c:1261 kernel/locking/rwsem.c:1526)
Dec 3 22:24:48 kernel: ? unmap_mapping_range (mm/memory.c:3873)
Dec 3 22:24:48 kernel: truncate_pagecache (mm/truncate.c:728)
Dec 3 22:24:48 kernel: xfs_setattr_size+0x139/0x410 xfs
Dec 3 22:24:48 kernel: xfs_vn_setattr+0x78/0x140 xfs
Dec 3 22:24:48 kernel: notify_change (fs/attr.c:503)
Dec 3 22:24:48 kernel: ? do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_truncate (./include/linux/fs.h:820 fs/open.c:66)
Dec 3 22:24:48 kernel: do_open (fs/namei.c:3395 fs/namei.c:3778)
Dec 3 22:24:48 kernel: path_openat (fs/namei.c:3933)
Dec 3 22:24:48 kernel: do_filp_open (fs/namei.c:3960)
Dec 3 22:24:48 kernel: ? unix_stream_recvmsg (net/unix/af_unix.c:2997)
Dec 3 22:24:48 kernel: ? __pfx_unix_stream_read_actor (net/unix/af_unix.c:2957)
Dec 3 22:24:48 kernel: ? kmem_cache_alloc_noprof (mm/slub.c:4115 mm/slub.c:4141)
Dec 3 22:24:48 kernel: do_sys_openat2 (fs/open.c:1415)
Dec 3 22:24:48 kernel: __x64_sys_openat (fs/open.c:1441)
Dec 3 22:24:48 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? syscall_exit_work (./include/linux/audit.h:357 kernel/entry/common.c:166)
Dec 3 22:24:48 kernel: ? syscall_exit_to_user_mode (./arch/x86/include/asm/irqflags.h:37 ./arch/x86/include/asm/irqflags.h:92 ./include/linux/entry-common.h:231 kernel/entry/common.c:206 kernel/entry/common.c:218)
Dec 3 22:24:48 kernel: ? do_syscall_64 (arch/x86/entry/common.c:102)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:150 kernel/locking/spinlock.c:194)
Dec 3 22:24:48 kernel: ? __skb_try_recv_datagram (net/core/datagram.c:267)
Dec 3 22:24:48 kernel: ? __skb_recv_datagram (net/core/datagram.c:296)
Dec 3 22:24:48 kernel: ? __memcg_slab_free_hook (mm/memcontrol.c:3004 (discriminator 2))
Dec 3 22:24:48 kernel: ? __pte_offset_map (./include/linux/pgtable.h:324 ./include/linux/pgtable.h:594 mm/pgtable-generic.c:289)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:24:48 kernel: ? do_wp_page (./include/linux/vmstat.h:75 mm/memory.c:3263 mm/memory.c:3731)
Dec 3 22:24:48 kernel: ? __handle_mm_fault (mm/memory.c:5909)
Dec 3 22:24:48 kernel: ? __count_memcg_events (mm/memcontrol.c:573 mm/memcontrol.c:836)
Dec 3 22:24:48 kernel: ? handle_mm_fault (mm/memory.c:5951 mm/memory.c:6103)
Dec 3 22:24:48 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: RIP: 0033:0x7f10818fd70b
Dec 3 22:24:48 kernel: RSP: 002b:00007fff83e83f80 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
Dec 3 22:24:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f10818fd70b
Dec 3 22:24:48 kernel: RDX: 0000000000080241 RSI: 00007f1080d5a8a0 RDI: 00000000ffffff9c
Dec 3 22:24:48 kernel: RBP: 00007f1080d5a8a0 R08: 0000000000000000 R09: 0000000000000000
Dec 3 22:24:48 kernel: R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000080241
Dec 3 22:24:48 kernel: R13: 00007f1080d50bb0 R14: 0000000000080241 R15: 0000000000000000
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:24:48 kernel: INFO: task mv:3483072 blocked for more than 245 seconds.
Dec 3 22:24:48 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:24:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:24:48 kernel: task:mv state:D stack:0 pid:3483072 tgid:3483072 ppid:3479428 flags:0x00000002
Dec 3 22:24:48 kernel: Call Trace:
Dec 3 22:24:48 kernel: <TASK>
Dec 3 22:24:48 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:24:48 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:24:48 kernel: io_schedule (kernel/sched/core.c:7552 kernel/sched/core.c:7578)
Dec 3 22:24:48 kernel: folio_wait_bit_common (mm/filemap.c:1301)
Dec 3 22:24:48 kernel: ? xas_load (./include/linux/xarray.h:175 ./include/linux/xarray.h:1264 lib/xarray.c:240)
Dec 3 22:24:48 kernel: ? __pfx_wake_page_function (mm/filemap.c:1117)
Dec 3 22:24:48 kernel: folio_wait_writeback (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:555 mm/page-writeback.c:3187)
Dec 3 22:24:48 kernel: truncate_inode_partial_folio (./arch/x86/include/asm/bitops.h:206 ./arch/x86/include/asm/bitops.h:238 ./include/asm-generic/bitops/instrumented-non-atomic.h:142 ./include/linux/page-flags.h:822 ./include/linux/page-flags.h:843 ./include/linux/mm.h:1115 ./include/linux/mm.h:2137 mm/truncate.c:209)
Dec 3 22:24:48 kernel: truncate_inode_pages_range (mm/truncate.c:354)
Dec 3 22:24:48 kernel: ? xfs_iunlock+0x108/0x200 xfs
Dec 3 22:24:48 kernel: ? xfs_rename+0x368/0x990 xfs
Dec 3 22:24:48 kernel: ? fsnotify_move (./include/linux/fsnotify.h:72 ./include/linux/fsnotify.h:64 ./include/linux/fsnotify.h:238)
Dec 3 22:24:48 kernel: ? _raw_spin_unlock (./arch/x86/include/asm/paravirt.h:589 ./arch/x86/include/asm/qspinlock.h:57 ./include/linux/spinlock.h:204 ./include/linux/spinlock_api_smp.h:142 kernel/locking/spinlock.c:186)
Dec 3 22:24:48 kernel: ? locked_inode_to_wb_and_lock_list (fs/fs-writeback.c:355)
Dec 3 22:24:48 kernel: evict (fs/inode.c:728)
Dec 3 22:24:48 kernel: ? fsnotify_destroy_marks (fs/notify/mark.c:923)
Dec 3 22:24:48 kernel: ? _atomic_dec_and_lock (./arch/x86/include/asm/atomic.h:67 ./include/linux/atomic/atomic-arch-fallback.h:2278 ./include/linux/atomic/atomic-instrumented.h:1384 lib/dec_and_lock.c:29)
Dec 3 22:24:48 kernel: __dentry_kill (fs/dcache.c:618)
Dec 3 22:24:48 kernel: dput (fs/dcache.c:857 fs/dcache.c:845)
Dec 3 22:24:48 kernel: do_renameat2 (fs/namei.c:5174)
Dec 3 22:24:48 kernel: __x64_sys_rename (fs/namei.c:5215)
Dec 3 22:24:48 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 3 22:24:48 kernel: ? do_user_addr_fault (./include/linux/mm.h:730 arch/x86/mm/fault.c:1340)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 3 22:24:48 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 3 22:24:48 kernel: RIP: 0033:0x7f3185a5aadb
Dec 3 22:24:48 kernel: RSP: 002b:00007ffe258c4548 EFLAGS: 00000246 ORIG_RAX: 0000000000000052
Dec 3 22:24:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f3185a5aadb
Dec 3 22:24:48 kernel: RDX: 0000000000000025 RSI: 00007ffe258c5cc3 RDI: 00007ffe258c5cb7
Dec 3 22:24:48 kernel: RBP: 00007ffe258c48f0 R08: 00007ffe258c4670 R09: 00007ffe258c4ac0
Dec 3 22:24:48 kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000000011
Dec 3 22:24:48 kernel: R13: 0000000000000000 R14: 00007ffe258c5cc3 R15: 00007ffe258c4ac0
Dec 3 22:24:48 kernel: </TASK>
Dec 3 22:26:51 kernel: INFO: task khugepaged:330 blocked for more than 368 seconds.
Dec 3 22:26:51 kernel: Tainted: G W 6.12.0+ #77
Dec 3 22:26:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 22:26:51 kernel: task:khugepaged state:D stack:0 pid:330 tgid:330 ppid:2 flags:0x00004000
Dec 3 22:26:51 kernel: Call Trace:
Dec 3 22:26:51 kernel: <TASK>
Dec 3 22:26:51 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 3 22:26:51 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 3 22:26:51 kernel: schedule_timeout (kernel/time/timer.c:2592)
Dec 3 22:26:51 kernel: ? kvm_sched_clock_read (arch/x86/kernel/kvmclock.c:91)
Dec 3 22:26:51 kernel: ? sched_clock (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/tsc.c:285)
Dec 3 22:26:51 kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Dec 3 22:26:51 kernel: ? __smp_call_single_queue (kernel/smp.c:115 kernel/smp.c:411)
Dec 3 22:26:51 kernel: __wait_for_common (kernel/sched/completion.c:95 kernel/sched/completion.c:116)
Dec 3 22:26:51 kernel: ? __pfx_schedule_timeout (kernel/time/timer.c:2577)
Dec 3 22:26:51 kernel: __flush_work (kernel/workqueue.c:4222)
Dec 3 22:26:51 kernel: ? __pfx_wq_barrier_func (kernel/workqueue.c:3718)
Dec 3 22:26:51 kernel: __lru_add_drain_all (mm/swap.c:873 (discriminator 3))
Dec 3 22:26:51 kernel: khugepaged (mm/khugepaged.c:2499 mm/khugepaged.c:2571)
Dec 3 22:26:51 kernel: ? __pfx_khugepaged (mm/khugepaged.c:2564)
Dec 3 22:26:51 kernel: kthread (kernel/kthread.c:389)
Dec 3 22:26:51 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:26:51 kernel: ret_from_fork (arch/x86/kernel/process.c:147)
Dec 3 22:26:51 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 3 22:26:51 kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
Dec 3 22:26:51 kernel: </TASK>
Dec 3 22:26:51 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Still blocked after many hours:
Dec 4 06:11:45 kernel: INFO: task khugepaged:330 blocked for more than 28262 seconds.
Dec 4 06:11:45 kernel: Tainted: G W 6.12.0+ #77
Dec 4 06:11:45 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 4 06:11:45 kernel: task:khugepaged state:D stack:0 pid:330 tgid:330 ppid:2 flags:0x00004000
Dec 4 06:11:45 kernel: Call Trace:
Dec 4 06:11:45 kernel: <TASK>
Dec 4 06:11:45 kernel: __schedule (kernel/sched/core.c:5328 kernel/sched/core.c:6693)
Dec 4 06:11:45 kernel: schedule (./arch/x86/include/asm/preempt.h:84 kernel/sched/core.c:6771 kernel/sched/core.c:6785)
Dec 4 06:11:45 kernel: schedule_timeout (kernel/time/timer.c:2592)
Dec 4 06:11:45 kernel: ? kvm_sched_clock_read (arch/x86/kernel/kvmclock.c:91)
Dec 4 06:11:45 kernel: ? sched_clock (./arch/x86/include/asm/preempt.h:94 arch/x86/kernel/tsc.c:285)
Dec 4 06:11:45 kernel: ? sched_clock_cpu (kernel/sched/clock.c:394)
Dec 4 06:11:45 kernel: ? __smp_call_single_queue (kernel/smp.c:115 kernel/smp.c:411)
Dec 4 06:11:45 kernel: __wait_for_common (kernel/sched/completion.c:95 kernel/sched/completion.c:116)
Dec 4 06:11:45 kernel: ? __pfx_schedule_timeout (kernel/time/timer.c:2577)
Dec 4 06:11:45 kernel: __flush_work (kernel/workqueue.c:4222)
Dec 4 06:11:45 kernel: ? __pfx_wq_barrier_func (kernel/workqueue.c:3718)
Dec 4 06:11:45 kernel: __lru_add_drain_all (mm/swap.c:873 (discriminator 3))
Dec 4 06:11:45 kernel: khugepaged (mm/khugepaged.c:2499 mm/khugepaged.c:2571)
Dec 4 06:11:45 kernel: ? __pfx_khugepaged (mm/khugepaged.c:2564)
Dec 4 06:11:45 kernel: kthread (kernel/kthread.c:389)
Dec 4 06:11:45 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 4 06:11:45 kernel: ret_from_fork (arch/x86/kernel/process.c:147)
Dec 4 06:11:45 kernel: ? __pfx_kthread (kernel/kthread.c:342)
Dec 4 06:11:45 kernel: ret_from_fork_asm (arch/x86/entry/entry_64.S:257)
Dec 4 06:11:45 kernel: </TASK>
Dec 4 06:11:45 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
The system is actually idle:
Dec 4 06:27:07 kernel: sysrq: Show backtrace of all active CPUs
Dec 4 06:27:07 kernel: NMI backtrace for cpu 30
Dec 4 06:27:07 kernel: CPU: 30 UID: 0 PID: 10810 Comm: bash Kdump: loaded Tainted: G W 6.12.0+ #77
Dec 4 06:27:07 kernel: Tainted: [W]=WARN
Dec 4 06:27:07 kernel: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.1-1.el9 04/01/2014
Dec 4 06:27:07 kernel: Call Trace:
Dec 4 06:27:07 kernel: <TASK>
Dec 4 06:27:07 kernel: dump_stack_lvl (lib/dump_stack.c:123)
Dec 4 06:27:07 kernel: nmi_cpu_backtrace (lib/nmi_backtrace.c:113)
Dec 4 06:27:07 kernel: ? __pfx_nmi_raise_cpu_backtrace (arch/x86/kernel/apic/hw_nmi.c:35)
Dec 4 06:27:07 kernel: nmi_trigger_cpumask_backtrace (lib/nmi_backtrace.c:62)
Dec 4 06:27:07 kernel: __handle_sysrq (drivers/tty/sysrq.c:613)
Dec 4 06:27:07 kernel: write_sysrq_trigger (drivers/tty/sysrq.c:1184)
Dec 4 06:27:07 kernel: proc_reg_write (fs/proc/inode.c:330 fs/proc/inode.c:342)
Dec 4 06:27:07 kernel: vfs_write (fs/read_write.c:681)
Dec 4 06:27:07 kernel: ? do_fcntl (fs/fcntl.c:463)
Dec 4 06:27:07 kernel: ksys_write (fs/read_write.c:736)
Dec 4 06:27:07 kernel: do_syscall_64 (arch/x86/entry/common.c:52 arch/x86/entry/common.c:83)
Dec 4 06:27:07 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 4 06:27:07 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 4 06:27:07 kernel: ? clear_bhb_loop (arch/x86/entry/entry_64.S:1539)
Dec 4 06:27:07 kernel: entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130)
Dec 4 06:27:07 kernel: RIP: 0033:0x7fb77cefda57
Dec 4 06:27:07 kernel: </TASK>
Dec 4 06:27:07 kernel: Sending NMI from CPU 30 to CPUs 0-29,31-39:
Dec 4 06:27:07 kernel: NMI backtrace for cpu 12 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 6 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 16 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 4 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 20 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 10 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 33 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 18 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 1 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 23 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 27 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 3 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 11 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 32 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 7 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 0 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 13 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 5 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 26 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 28 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 24 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 37 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 39 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 17 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 22 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 31 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 8 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 21 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 29 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 14 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 2 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 9 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 35 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 25 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 15 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 36 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 19 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 34 skipped: idling at default_idle
Dec 4 06:27:07 kernel: NMI backtrace for cpu 38 skipped: idling at default_idle
=====================
SECOND UNDECODED LOG:
=====================
Dec 3 01:58:46 kernel: ------------[ cut here ]------------
Dec 3 01:58:46 kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [kworker/11:1:866154]
Dec 3 01:58:46 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 01:58:46 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000000 softirq=2109356/2109356 fqs=0
Dec 3 01:58:46 kernel: rcu: #011(detected by 18, t=60004 jiffies, g=11126601, q=393720 ncpus=40)
Dec 3 01:58:46 kernel: Sending NMI from CPU 18 to CPUs 17:
Dec 3 01:58:46 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 70001 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 01:58:46 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 01:58:46 kernel: rcu: rcu_preempt kthread starved for 70004 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 01:58:46 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 01:58:46 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 01:58:46 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 01:58:46 kernel: Call Trace:
Dec 3 01:58:46 kernel: <TASK>
Dec 3 01:58:46 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 01:58:46 kernel: ? __schedule+0xfe/0x620
Dec 3 01:58:46 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 01:58:46 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 01:58:46 kernel: ? schedule+0x23/0xa0
Dec 3 01:58:46 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 01:58:46 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 01:58:46 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 01:58:46 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 01:58:46 kernel: ? kthread+0xcc/0x100
Dec 3 01:58:46 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 01:58:46 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 01:58:46 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 01:58:46 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 01:58:46 kernel: </TASK>
Dec 3 02:00:48 kernel: INFO: task kworker/u165:0:746685 blocked for more than 122 seconds.
Dec 3 02:00:48 kernel: Not tainted 6.12.0 #64
Dec 3 02:00:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:00:48 kernel: task:kworker/u165:0 state:D stack:0 pid:746685 tgid:746685 ppid:2 flags:0x00004000
Dec 3 02:00:48 kernel: Workqueue: events_unbound linkwatch_event
Dec 3 02:00:48 kernel: Call Trace:
Dec 3 02:00:48 kernel: <TASK>
Dec 3 02:00:48 kernel: __schedule+0x23f/0x620
Dec 3 02:00:48 kernel: schedule+0x23/0xa0
Dec 3 02:00:48 kernel: schedule_preempt_disabled+0x11/0x20
Dec 3 02:00:48 kernel: __mutex_lock.constprop.0+0x31d/0x650
Dec 3 02:00:48 kernel: ? __schedule+0x247/0x620
Dec 3 02:00:48 kernel: linkwatch_event+0xa/0x30
Dec 3 02:00:48 kernel: process_one_work+0x179/0x390
Dec 3 02:00:48 kernel: worker_thread+0x239/0x340
Dec 3 02:00:48 kernel: ? __pfx_worker_thread+0x10/0x10
Dec 3 02:00:48 kernel: kthread+0xcc/0x100
Dec 3 02:00:48 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:00:48 kernel: ret_from_fork+0x2d/0x50
Dec 3 02:00:48 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:00:48 kernel: ret_from_fork_asm+0x1a/0x30
Dec 3 02:00:48 kernel: </TASK>
Dec 3 02:00:48 kernel: INFO: task kworker/5:2:900494 blocked for more than 122 seconds.
Dec 3 02:00:48 kernel: Not tainted 6.12.0 #64
Dec 3 02:00:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:00:48 kernel: task:kworker/5:2 state:D stack:0 pid:900494 tgid:900494 ppid:2 flags:0x00004000
Dec 3 02:00:48 kernel: Workqueue: events xfrm_state_gc_task
Dec 3 02:00:48 kernel: Call Trace:
Dec 3 02:00:48 kernel: <TASK>
Dec 3 02:00:48 kernel: __schedule+0x23f/0x620
Dec 3 02:00:48 kernel: schedule+0x23/0xa0
Dec 3 02:00:48 kernel: schedule_timeout+0x14a/0x160
Dec 3 02:00:48 kernel: ? __hrtimer_start_range_ns+0x20b/0x2e0
Dec 3 02:00:48 kernel: ? kvm_clock_get_cycles+0x14/0x30
Dec 3 02:00:48 kernel: ? ktime_get+0x34/0xc0
Dec 3 02:00:48 kernel: ? timerqueue_del+0x2a/0x50
Dec 3 02:00:48 kernel: __wait_for_common+0x8f/0x1d0
Dec 3 02:00:48 kernel: ? __pfx_schedule_timeout+0x10/0x10
Dec 3 02:00:48 kernel: wait_for_completion_state+0x1d/0x40
Dec 3 02:00:48 kernel: __wait_rcu_gp+0x126/0x130
Dec 3 02:00:48 kernel: synchronize_rcu_normal.part.0+0x3a/0x60
Dec 3 02:00:48 kernel: ? __pfx_call_rcu_hurry+0x10/0x10
Dec 3 02:00:48 kernel: ? __pfx_wakeme_after_rcu+0x10/0x10
Dec 3 02:00:48 kernel: xfrm_state_gc_task+0x56/0xa0
Dec 3 02:00:48 kernel: process_one_work+0x179/0x390
Dec 3 02:00:48 kernel: worker_thread+0x239/0x340
Dec 3 02:00:48 kernel: ? __pfx_worker_thread+0x10/0x10
Dec 3 02:00:48 kernel: kthread+0xcc/0x100
Dec 3 02:00:48 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:00:48 kernel: ret_from_fork+0x2d/0x50
Dec 3 02:00:48 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:00:48 kernel: ret_from_fork_asm+0x1a/0x30
Dec 3 02:00:48 kernel: </TASK>
Dec 3 02:00:48 kernel: INFO: task systemd-udevd:995278 blocked for more than 122 seconds.
Dec 3 02:00:48 kernel: Not tainted 6.12.0 #64
Dec 3 02:00:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:00:48 kernel: task:systemd-udevd state:D stack:0 pid:995278 tgid:995278 ppid:1080 flags:0x00000002
Dec 3 02:00:48 kernel: Call Trace:
Dec 3 02:00:48 kernel: <TASK>
Dec 3 02:00:48 kernel: __schedule+0x23f/0x620
Dec 3 02:00:48 kernel: schedule+0x23/0xa0
Dec 3 02:00:48 kernel: netlink_table_grab.part.0+0x82/0xe0
Dec 3 02:00:48 kernel: ? __pfx_default_wake_function+0x10/0x10
Dec 3 02:00:48 kernel: netlink_release+0x36c/0x520
Dec 3 02:00:48 kernel: ? __pfx_netlink_hash+0x10/0x10
Dec 3 02:00:48 kernel: ? __pfx_netlink_compare+0x10/0x10
Dec 3 02:00:48 kernel: __sock_release+0x3a/0xc0
Dec 3 02:00:48 kernel: sock_close+0x11/0x20
Dec 3 02:00:48 kernel: __fput+0xdb/0x2a0
Dec 3 02:00:48 kernel: task_work_run+0x55/0x90
Dec 3 02:00:48 kernel: do_exit+0x279/0x4b0
Dec 3 02:00:48 kernel: do_group_exit+0x2c/0x80
Dec 3 02:00:48 kernel: __x64_sys_exit_group+0x14/0x20
Dec 3 02:00:48 kernel: x64_sys_call+0x1836/0x1840
Dec 3 02:00:48 kernel: do_syscall_64+0x79/0x150
Dec 3 02:00:48 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:00:48 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:00:48 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:00:48 kernel: RIP: 0033:0x7f6cfa8d921d
Dec 3 02:00:48 kernel: RSP: 002b:00007ffd7e081b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Dec 3 02:00:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6cfa8d921d
Dec 3 02:00:48 kernel: RDX: 00000000000000e7 RSI: fffffffffffffe88 RDI: 0000000000000000
Dec 3 02:00:48 kernel: RBP: 00007ffd7e081c00 R08: 00005615d9e8dbb0 R09: 0000000000000004
Dec 3 02:00:48 kernel: R10: 0000000000000018 R11: 0000000000000246 R12: 00007ffd7e081bb0
Dec 3 02:00:48 kernel: R13: 00005615d9d0e8b0 R14: 00005615d9e8dbb0 R15: 0000000000000000
Dec 3 02:00:48 kernel: </TASK>
Dec 3 02:00:48 kernel: INFO: task systemd-udevd:995279 blocked for more than 122 seconds.
Dec 3 02:00:48 kernel: Not tainted 6.12.0 #64
Dec 3 02:00:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:00:48 kernel: task:systemd-udevd state:D stack:0 pid:995279 tgid:995279 ppid:1080 flags:0x00004002
Dec 3 02:00:48 kernel: Call Trace:
Dec 3 02:00:48 kernel: <TASK>
Dec 3 02:00:48 kernel: __schedule+0x23f/0x620
Dec 3 02:00:48 kernel: schedule+0x23/0xa0
Dec 3 02:00:48 kernel: netlink_table_grab.part.0+0x82/0xe0
Dec 3 02:00:48 kernel: ? __pfx_default_wake_function+0x10/0x10
Dec 3 02:00:48 kernel: netlink_release+0x36c/0x520
Dec 3 02:00:48 kernel: ? __pfx_netlink_hash+0x10/0x10
Dec 3 02:00:48 kernel: ? __pfx_netlink_compare+0x10/0x10
Dec 3 02:00:48 kernel: __sock_release+0x3a/0xc0
Dec 3 02:00:48 kernel: sock_close+0x11/0x20
Dec 3 02:00:48 kernel: __fput+0xdb/0x2a0
Dec 3 02:00:48 kernel: task_work_run+0x55/0x90
Dec 3 02:00:48 kernel: do_exit+0x279/0x4b0
Dec 3 02:00:48 kernel: do_group_exit+0x2c/0x80
Dec 3 02:00:48 kernel: __x64_sys_exit_group+0x14/0x20
Dec 3 02:00:48 kernel: x64_sys_call+0x1836/0x1840
Dec 3 02:00:48 kernel: do_syscall_64+0x79/0x150
Dec 3 02:00:48 kernel: ? get_page_from_freelist+0x333/0x630
Dec 3 02:00:48 kernel: ? __alloc_pages_noprof+0x186/0x350
Dec 3 02:00:48 kernel: ? __mod_memcg_lruvec_state+0x95/0x150
Dec 3 02:00:48 kernel: ? __lruvec_stat_mod_folio+0x80/0xd0
Dec 3 02:00:48 kernel: ? __folio_mod_stat+0x2a/0x80
Dec 3 02:00:48 kernel: ? _raw_spin_unlock+0xa/0x30
Dec 3 02:00:48 kernel: ? wp_page_copy+0x4e0/0x710
Dec 3 02:00:48 kernel: ? __pte_offset_map+0x17/0x160
Dec 3 02:00:48 kernel: ? _raw_spin_unlock+0xa/0x30
Dec 3 02:00:48 kernel: ? do_wp_page+0x666/0x760
Dec 3 02:00:48 kernel: ? __handle_mm_fault+0x326/0x730
Dec 3 02:00:48 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:00:48 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:00:48 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:00:48 kernel: RIP: 0033:0x7f6cfa8d921d
Dec 3 02:00:48 kernel: RSP: 002b:00007ffd7e081b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Dec 3 02:00:48 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6cfa8d921d
Dec 3 02:00:48 kernel: RDX: 00000000000000e7 RSI: fffffffffffffe88 RDI: 0000000000000000
Dec 3 02:00:48 kernel: RBP: 00007ffd7e081c00 R08: 00005615d9e932b0 R09: 0000000000000004
Dec 3 02:00:48 kernel: R10: 0000000000000018 R11: 0000000000000246 R12: 00007ffd7e081bb0
Dec 3 02:00:48 kernel: R13: 00005615d9d0e8b0 R14: 00005615d9e932b0 R15: 0000000000000000
Dec 3 02:00:48 kernel: </TASK>
Dec 3 02:00:48 kernel: INFO: task ip:998743 blocked for more than 122 seconds.
Dec 3 02:00:48 kernel: Not tainted 6.12.0 #64
Dec 3 02:00:48 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:00:48 kernel: task:ip state:D stack:0 pid:998743 tgid:998743 ppid:998736 flags:0x00000002
Dec 3 02:00:48 kernel: Call Trace:
Dec 3 02:00:48 kernel: <TASK>
Dec 3 02:00:48 kernel: __schedule+0x23f/0x620
Dec 3 02:00:48 kernel: schedule+0x23/0xa0
Dec 3 02:00:48 kernel: schedule_preempt_disabled+0x11/0x20
Dec 3 02:00:48 kernel: __mutex_lock.constprop.0+0x31d/0x650
Dec 3 02:00:48 kernel: rtnetlink_rcv_msg+0x111/0x410
Dec 3 02:00:48 kernel: ? avc_has_perm_noaudit+0x67/0xf0
Dec 3 02:00:48 kernel: ? __pfx_rtnetlink_rcv_msg+0x10/0x10
Dec 3 02:00:48 kernel: netlink_rcv_skb+0x54/0x100
Dec 3 02:00:48 kernel: netlink_unicast+0x243/0x370
Dec 3 02:00:48 kernel: netlink_sendmsg+0x1f6/0x430
Dec 3 02:00:48 kernel: __sys_sendto+0x1f3/0x200
Dec 3 02:00:48 kernel: ? do_read_fault+0x10a/0x1e0
Dec 3 02:00:48 kernel: ? do_fault+0x21f/0x380
Dec 3 02:00:48 kernel: ? pte_offset_map_nolock+0x2b/0xb0
Dec 3 02:00:48 kernel: __x64_sys_sendto+0x20/0x30
Dec 3 02:00:48 kernel: do_syscall_64+0x79/0x150
Dec 3 02:00:48 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:00:48 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:00:48 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:00:48 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:00:48 kernel: RIP: 0033:0x7f287830f860
Dec 3 02:00:48 kernel: RSP: 002b:00007ffdd4f2c518 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 3 02:00:48 kernel: RAX: ffffffffffffffda RBX: 00007ffdd4f2cc88 RCX: 00007f287830f860
Dec 3 02:00:48 kernel: RDX: 0000000000000020 RSI: 00007ffdd4f2c520 RDI: 0000000000000003
Dec 3 02:00:48 kernel: RBP: 00007ffdd4f2d7e4 R08: 0000000000000000 R09: 0000000000000000
Dec 3 02:00:48 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
Dec 3 02:00:48 kernel: R13: 000055969d419040 R14: 00007ffdd4f2cc78 R15: 0000000000000004
Dec 3 02:00:48 kernel: </TASK>
Dec 3 02:01:56 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:01:56 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:01:56 kernel: rcu: #011(detected by 8, t=250009 jiffies, g=11126601, q=1577177 ncpus=40)
Dec 3 02:01:56 kernel: Sending NMI from CPU 8 to CPUs 17:
Dec 3 02:01:56 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 260006 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:01:56 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:01:56 kernel: rcu: rcu_preempt kthread starved for 260009 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:01:56 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:01:56 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:01:56 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:01:56 kernel: Call Trace:
Dec 3 02:01:56 kernel: <TASK>
Dec 3 02:01:56 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:01:56 kernel: ? __schedule+0xfe/0x620
Dec 3 02:01:56 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:01:56 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:01:56 kernel: ? schedule+0x23/0xa0
Dec 3 02:01:56 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:01:56 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:01:56 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:01:56 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:01:56 kernel: ? kthread+0xcc/0x100
Dec 3 02:01:56 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:01:56 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:01:56 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:01:56 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:01:56 kernel: </TASK>
Dec 3 02:02:51 kernel: INFO: task kworker/u165:0:746685 blocked for more than 245 seconds.
Dec 3 02:02:51 kernel: Not tainted 6.12.0 #64
Dec 3 02:02:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:02:51 kernel: task:kworker/u165:0 state:D stack:0 pid:746685 tgid:746685 ppid:2 flags:0x00004000
Dec 3 02:02:51 kernel: Workqueue: events_unbound linkwatch_event
Dec 3 02:02:51 kernel: Call Trace:
Dec 3 02:02:51 kernel: <TASK>
Dec 3 02:02:51 kernel: __schedule+0x23f/0x620
Dec 3 02:02:51 kernel: schedule+0x23/0xa0
Dec 3 02:02:51 kernel: schedule_preempt_disabled+0x11/0x20
Dec 3 02:02:51 kernel: __mutex_lock.constprop.0+0x31d/0x650
Dec 3 02:02:51 kernel: ? __schedule+0x247/0x620
Dec 3 02:02:51 kernel: linkwatch_event+0xa/0x30
Dec 3 02:02:51 kernel: process_one_work+0x179/0x390
Dec 3 02:02:51 kernel: worker_thread+0x239/0x340
Dec 3 02:02:51 kernel: ? __pfx_worker_thread+0x10/0x10
Dec 3 02:02:51 kernel: kthread+0xcc/0x100
Dec 3 02:02:51 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:02:51 kernel: ret_from_fork+0x2d/0x50
Dec 3 02:02:51 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:02:51 kernel: ret_from_fork_asm+0x1a/0x30
Dec 3 02:02:51 kernel: </TASK>
Dec 3 02:02:51 kernel: INFO: task kworker/5:2:900494 blocked for more than 245 seconds.
Dec 3 02:02:51 kernel: Not tainted 6.12.0 #64
Dec 3 02:02:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:02:51 kernel: task:kworker/5:2 state:D stack:0 pid:900494 tgid:900494 ppid:2 flags:0x00004000
Dec 3 02:02:51 kernel: Workqueue: events xfrm_state_gc_task
Dec 3 02:02:51 kernel: Call Trace:
Dec 3 02:02:51 kernel: <TASK>
Dec 3 02:02:51 kernel: __schedule+0x23f/0x620
Dec 3 02:02:51 kernel: schedule+0x23/0xa0
Dec 3 02:02:51 kernel: schedule_timeout+0x14a/0x160
Dec 3 02:02:51 kernel: ? __hrtimer_start_range_ns+0x20b/0x2e0
Dec 3 02:02:51 kernel: ? kvm_clock_get_cycles+0x14/0x30
Dec 3 02:02:51 kernel: ? ktime_get+0x34/0xc0
Dec 3 02:02:51 kernel: ? timerqueue_del+0x2a/0x50
Dec 3 02:02:51 kernel: __wait_for_common+0x8f/0x1d0
Dec 3 02:02:51 kernel: ? __pfx_schedule_timeout+0x10/0x10
Dec 3 02:02:51 kernel: wait_for_completion_state+0x1d/0x40
Dec 3 02:02:51 kernel: __wait_rcu_gp+0x126/0x130
Dec 3 02:02:51 kernel: synchronize_rcu_normal.part.0+0x3a/0x60
Dec 3 02:02:51 kernel: ? __pfx_call_rcu_hurry+0x10/0x10
Dec 3 02:02:51 kernel: ? __pfx_wakeme_after_rcu+0x10/0x10
Dec 3 02:02:51 kernel: xfrm_state_gc_task+0x56/0xa0
Dec 3 02:02:51 kernel: process_one_work+0x179/0x390
Dec 3 02:02:51 kernel: worker_thread+0x239/0x340
Dec 3 02:02:51 kernel: ? __pfx_worker_thread+0x10/0x10
Dec 3 02:02:51 kernel: kthread+0xcc/0x100
Dec 3 02:02:51 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:02:51 kernel: ret_from_fork+0x2d/0x50
Dec 3 02:02:51 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:02:51 kernel: ret_from_fork_asm+0x1a/0x30
Dec 3 02:02:51 kernel: </TASK>
Dec 3 02:02:51 kernel: INFO: task systemd-udevd:995278 blocked for more than 245 seconds.
Dec 3 02:02:51 kernel: Not tainted 6.12.0 #64
Dec 3 02:02:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:02:51 kernel: task:systemd-udevd state:D stack:0 pid:995278 tgid:995278 ppid:1080 flags:0x00000002
Dec 3 02:02:51 kernel: Call Trace:
Dec 3 02:02:51 kernel: <TASK>
Dec 3 02:02:51 kernel: __schedule+0x23f/0x620
Dec 3 02:02:51 kernel: schedule+0x23/0xa0
Dec 3 02:02:51 kernel: netlink_table_grab.part.0+0x82/0xe0
Dec 3 02:02:51 kernel: ? __pfx_default_wake_function+0x10/0x10
Dec 3 02:02:51 kernel: netlink_release+0x36c/0x520
Dec 3 02:02:51 kernel: ? __pfx_netlink_hash+0x10/0x10
Dec 3 02:02:51 kernel: ? __pfx_netlink_compare+0x10/0x10
Dec 3 02:02:51 kernel: __sock_release+0x3a/0xc0
Dec 3 02:02:51 kernel: sock_close+0x11/0x20
Dec 3 02:02:51 kernel: __fput+0xdb/0x2a0
Dec 3 02:02:51 kernel: task_work_run+0x55/0x90
Dec 3 02:02:51 kernel: do_exit+0x279/0x4b0
Dec 3 02:02:51 kernel: do_group_exit+0x2c/0x80
Dec 3 02:02:51 kernel: __x64_sys_exit_group+0x14/0x20
Dec 3 02:02:51 kernel: x64_sys_call+0x1836/0x1840
Dec 3 02:02:51 kernel: do_syscall_64+0x79/0x150
Dec 3 02:02:51 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:02:51 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:02:51 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:02:51 kernel: RIP: 0033:0x7f6cfa8d921d
Dec 3 02:02:51 kernel: RSP: 002b:00007ffd7e081b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Dec 3 02:02:51 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6cfa8d921d
Dec 3 02:02:51 kernel: RDX: 00000000000000e7 RSI: fffffffffffffe88 RDI: 0000000000000000
Dec 3 02:02:51 kernel: RBP: 00007ffd7e081c00 R08: 00005615d9e8dbb0 R09: 0000000000000004
Dec 3 02:02:51 kernel: R10: 0000000000000018 R11: 0000000000000246 R12: 00007ffd7e081bb0
Dec 3 02:02:51 kernel: R13: 00005615d9d0e8b0 R14: 00005615d9e8dbb0 R15: 0000000000000000
Dec 3 02:02:51 kernel: </TASK>
Dec 3 02:02:51 kernel: INFO: task systemd-udevd:995279 blocked for more than 245 seconds.
Dec 3 02:02:51 kernel: Not tainted 6.12.0 #64
Dec 3 02:02:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:02:51 kernel: task:systemd-udevd state:D stack:0 pid:995279 tgid:995279 ppid:1080 flags:0x00004002
Dec 3 02:02:51 kernel: Call Trace:
Dec 3 02:02:51 kernel: <TASK>
Dec 3 02:02:51 kernel: __schedule+0x23f/0x620
Dec 3 02:02:51 kernel: schedule+0x23/0xa0
Dec 3 02:02:51 kernel: netlink_table_grab.part.0+0x82/0xe0
Dec 3 02:02:51 kernel: ? __pfx_default_wake_function+0x10/0x10
Dec 3 02:02:51 kernel: netlink_release+0x36c/0x520
Dec 3 02:02:51 kernel: ? __pfx_netlink_hash+0x10/0x10
Dec 3 02:02:51 kernel: ? __pfx_netlink_compare+0x10/0x10
Dec 3 02:02:51 kernel: __sock_release+0x3a/0xc0
Dec 3 02:02:51 kernel: sock_close+0x11/0x20
Dec 3 02:02:51 kernel: __fput+0xdb/0x2a0
Dec 3 02:02:51 kernel: task_work_run+0x55/0x90
Dec 3 02:02:51 kernel: do_exit+0x279/0x4b0
Dec 3 02:02:51 kernel: do_group_exit+0x2c/0x80
Dec 3 02:02:51 kernel: __x64_sys_exit_group+0x14/0x20
Dec 3 02:02:51 kernel: x64_sys_call+0x1836/0x1840
Dec 3 02:02:51 kernel: do_syscall_64+0x79/0x150
Dec 3 02:02:51 kernel: ? get_page_from_freelist+0x333/0x630
Dec 3 02:02:51 kernel: ? __alloc_pages_noprof+0x186/0x350
Dec 3 02:02:51 kernel: ? __mod_memcg_lruvec_state+0x95/0x150
Dec 3 02:02:51 kernel: ? __lruvec_stat_mod_folio+0x80/0xd0
Dec 3 02:02:51 kernel: ? __folio_mod_stat+0x2a/0x80
Dec 3 02:02:51 kernel: ? _raw_spin_unlock+0xa/0x30
Dec 3 02:02:51 kernel: ? wp_page_copy+0x4e0/0x710
Dec 3 02:02:51 kernel: ? __pte_offset_map+0x17/0x160
Dec 3 02:02:51 kernel: ? _raw_spin_unlock+0xa/0x30
Dec 3 02:02:51 kernel: ? do_wp_page+0x666/0x760
Dec 3 02:02:51 kernel: ? __handle_mm_fault+0x326/0x730
Dec 3 02:02:51 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:02:51 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:02:51 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:02:51 kernel: RIP: 0033:0x7f6cfa8d921d
Dec 3 02:02:51 kernel: RSP: 002b:00007ffd7e081b58 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
Dec 3 02:02:51 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6cfa8d921d
Dec 3 02:02:51 kernel: RDX: 00000000000000e7 RSI: fffffffffffffe88 RDI: 0000000000000000
Dec 3 02:02:51 kernel: RBP: 00007ffd7e081c00 R08: 00005615d9e932b0 R09: 0000000000000004
Dec 3 02:02:51 kernel: R10: 0000000000000018 R11: 0000000000000246 R12: 00007ffd7e081bb0
Dec 3 02:02:51 kernel: R13: 00005615d9d0e8b0 R14: 00005615d9e932b0 R15: 0000000000000000
Dec 3 02:02:51 kernel: </TASK>
Dec 3 02:02:51 kernel: INFO: task ip:998743 blocked for more than 245 seconds.
Dec 3 02:02:51 kernel: Not tainted 6.12.0 #64
Dec 3 02:02:51 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 3 02:02:51 kernel: task:ip state:D stack:0 pid:998743 tgid:998743 ppid:998736 flags:0x00000002
Dec 3 02:02:51 kernel: Call Trace:
Dec 3 02:02:51 kernel: <TASK>
Dec 3 02:02:51 kernel: __schedule+0x23f/0x620
Dec 3 02:02:51 kernel: schedule+0x23/0xa0
Dec 3 02:02:51 kernel: schedule_preempt_disabled+0x11/0x20
Dec 3 02:02:51 kernel: __mutex_lock.constprop.0+0x31d/0x650
Dec 3 02:02:51 kernel: rtnetlink_rcv_msg+0x111/0x410
Dec 3 02:02:51 kernel: ? avc_has_perm_noaudit+0x67/0xf0
Dec 3 02:02:51 kernel: ? __pfx_rtnetlink_rcv_msg+0x10/0x10
Dec 3 02:02:51 kernel: netlink_rcv_skb+0x54/0x100
Dec 3 02:02:51 kernel: netlink_unicast+0x243/0x370
Dec 3 02:02:51 kernel: netlink_sendmsg+0x1f6/0x430
Dec 3 02:02:51 kernel: __sys_sendto+0x1f3/0x200
Dec 3 02:02:51 kernel: ? do_read_fault+0x10a/0x1e0
Dec 3 02:02:51 kernel: ? do_fault+0x21f/0x380
Dec 3 02:02:51 kernel: ? pte_offset_map_nolock+0x2b/0xb0
Dec 3 02:02:51 kernel: __x64_sys_sendto+0x20/0x30
Dec 3 02:02:51 kernel: do_syscall_64+0x79/0x150
Dec 3 02:02:51 kernel: ? __count_memcg_events+0x4f/0xe0
Dec 3 02:02:51 kernel: ? handle_mm_fault+0x18e/0x270
Dec 3 02:02:51 kernel: ? do_user_addr_fault+0x34c/0x680
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: ? clear_bhb_loop+0x45/0xa0
Dec 3 02:02:51 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Dec 3 02:02:51 kernel: RIP: 0033:0x7f287830f860
Dec 3 02:02:51 kernel: RSP: 002b:00007ffdd4f2c518 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
Dec 3 02:02:51 kernel: RAX: ffffffffffffffda RBX: 00007ffdd4f2cc88 RCX: 00007f287830f860
Dec 3 02:02:51 kernel: RDX: 0000000000000020 RSI: 00007ffdd4f2c520 RDI: 0000000000000003
Dec 3 02:02:51 kernel: RBP: 00007ffdd4f2d7e4 R08: 0000000000000000 R09: 0000000000000000
Dec 3 02:02:51 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000002
Dec 3 02:02:51 kernel: R13: 000055969d419040 R14: 00007ffdd4f2cc78 R15: 0000000000000004
Dec 3 02:02:51 kernel: </TASK>
Dec 3 02:02:51 kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
Dec 3 02:05:06 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:05:06 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:05:06 kernel: rcu: #011(detected by 5, t=440015 jiffies, g=11126601, q=2760447 ncpus=40)
Dec 3 02:05:06 kernel: Sending NMI from CPU 5 to CPUs 17:
Dec 3 02:05:06 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 450012 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:05:06 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:05:06 kernel: rcu: rcu_preempt kthread starved for 450015 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:05:06 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:05:06 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:05:06 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:05:06 kernel: Call Trace:
Dec 3 02:05:06 kernel: <TASK>
Dec 3 02:05:06 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:05:06 kernel: ? __schedule+0xfe/0x620
Dec 3 02:05:06 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:05:06 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:05:06 kernel: ? schedule+0x23/0xa0
Dec 3 02:05:06 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:05:06 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:05:06 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:05:06 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:05:06 kernel: ? kthread+0xcc/0x100
Dec 3 02:05:06 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:05:06 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:05:06 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:05:06 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:05:06 kernel: </TASK>
Dec 3 02:08:16 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:08:16 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:08:16 kernel: rcu: #011(detected by 15, t=630022 jiffies, g=11126601, q=3942985 ncpus=40)
Dec 3 02:08:16 kernel: Sending NMI from CPU 15 to CPUs 17:
Dec 3 02:08:16 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 640019 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:08:16 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:08:16 kernel: rcu: rcu_preempt kthread starved for 640022 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:08:16 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:08:16 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:08:16 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:08:16 kernel: Call Trace:
Dec 3 02:08:16 kernel: <TASK>
Dec 3 02:08:16 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:08:16 kernel: ? __schedule+0xfe/0x620
Dec 3 02:08:16 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:08:16 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:08:16 kernel: ? schedule+0x23/0xa0
Dec 3 02:08:16 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:08:16 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:08:16 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:08:16 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:08:16 kernel: ? kthread+0xcc/0x100
Dec 3 02:08:16 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:08:16 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:08:16 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:08:16 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:08:16 kernel: </TASK>
Dec 3 02:11:26 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:11:26 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:11:26 kernel: rcu: #011(detected by 13, t=820027 jiffies, g=11126601, q=5124103 ncpus=40)
Dec 3 02:11:26 kernel: Sending NMI from CPU 13 to CPUs 17:
Dec 3 02:11:26 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 830024 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:11:26 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:11:26 kernel: rcu: rcu_preempt kthread starved for 830027 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:11:26 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:11:26 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:11:26 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:11:26 kernel: Call Trace:
Dec 3 02:11:26 kernel: <TASK>
Dec 3 02:11:26 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:11:26 kernel: ? __schedule+0xfe/0x620
Dec 3 02:11:26 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:11:26 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:11:26 kernel: ? schedule+0x23/0xa0
Dec 3 02:11:26 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:11:26 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:11:26 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:11:26 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:11:26 kernel: ? kthread+0xcc/0x100
Dec 3 02:11:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:11:26 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:11:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:11:26 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:11:26 kernel: </TASK>
Dec 3 02:14:36 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:14:36 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:14:36 kernel: rcu: #011(detected by 29, t=1010032 jiffies, g=11126601, q=6331978 ncpus=40)
Dec 3 02:14:36 kernel: Sending NMI from CPU 29 to CPUs 17:
Dec 3 02:14:36 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1020029 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:14:36 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:14:36 kernel: rcu: rcu_preempt kthread starved for 1020032 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:14:36 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:14:36 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:14:36 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:14:36 kernel: Call Trace:
Dec 3 02:14:36 kernel: <TASK>
Dec 3 02:14:36 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:14:36 kernel: ? __schedule+0xfe/0x620
Dec 3 02:14:36 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:14:36 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:14:36 kernel: ? schedule+0x23/0xa0
Dec 3 02:14:36 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:14:36 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:14:36 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:14:36 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:14:36 kernel: ? kthread+0xcc/0x100
Dec 3 02:14:36 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:14:36 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:14:36 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:14:36 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:14:36 kernel: </TASK>
Dec 3 02:17:46 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:17:46 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:17:46 kernel: rcu: #011(detected by 8, t=1200037 jiffies, g=11126601, q=7535150 ncpus=40)
Dec 3 02:17:46 kernel: Sending NMI from CPU 8 to CPUs 17:
Dec 3 02:17:46 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1210034 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:17:46 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:17:46 kernel: rcu: rcu_preempt kthread starved for 1210037 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:17:46 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:17:46 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:17:46 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:17:46 kernel: Call Trace:
Dec 3 02:17:46 kernel: <TASK>
Dec 3 02:17:46 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:17:46 kernel: ? __schedule+0xfe/0x620
Dec 3 02:17:46 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:17:46 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:17:46 kernel: ? schedule+0x23/0xa0
Dec 3 02:17:46 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:17:46 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:17:46 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:17:46 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:17:46 kernel: ? kthread+0xcc/0x100
Dec 3 02:17:46 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:17:46 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:17:46 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:17:46 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:17:46 kernel: </TASK>
Dec 3 02:20:56 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:20:56 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:20:56 kernel: rcu: #011(detected by 8, t=1390043 jiffies, g=11126601, q=8740806 ncpus=40)
Dec 3 02:20:56 kernel: Sending NMI from CPU 8 to CPUs 17:
Dec 3 02:20:56 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1400040 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:20:56 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:20:56 kernel: rcu: rcu_preempt kthread starved for 1400043 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:20:56 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:20:56 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:20:56 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:20:56 kernel: Call Trace:
Dec 3 02:20:56 kernel: <TASK>
Dec 3 02:20:56 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:20:56 kernel: ? __schedule+0xfe/0x620
Dec 3 02:20:56 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:20:56 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:20:56 kernel: ? schedule+0x23/0xa0
Dec 3 02:20:56 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:20:56 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:20:56 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:20:56 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:20:56 kernel: ? kthread+0xcc/0x100
Dec 3 02:20:56 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:20:56 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:20:56 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:20:56 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:20:56 kernel: </TASK>
Dec 3 02:24:06 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:24:06 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:24:06 kernel: rcu: #011(detected by 23, t=1580048 jiffies, g=11126601, q=9924647 ncpus=40)
Dec 3 02:24:06 kernel: Sending NMI from CPU 23 to CPUs 17:
Dec 3 02:24:06 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1590045 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:24:06 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:24:06 kernel: rcu: rcu_preempt kthread starved for 1590048 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:24:06 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:24:06 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:24:06 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:24:06 kernel: Call Trace:
Dec 3 02:24:06 kernel: <TASK>
Dec 3 02:24:06 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:24:06 kernel: ? __schedule+0xfe/0x620
Dec 3 02:24:06 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:24:06 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:24:06 kernel: ? schedule+0x23/0xa0
Dec 3 02:24:06 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:24:06 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:24:06 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:24:06 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:24:06 kernel: ? kthread+0xcc/0x100
Dec 3 02:24:06 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:24:06 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:24:06 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:24:06 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:24:06 kernel: </TASK>
Dec 3 02:27:16 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:27:16 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:27:16 kernel: rcu: #011(detected by 18, t=1770053 jiffies, g=11126601, q=11108597 ncpus=40)
Dec 3 02:27:16 kernel: Sending NMI from CPU 18 to CPUs 17:
Dec 3 02:27:16 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1780050 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:27:16 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:27:16 kernel: rcu: rcu_preempt kthread starved for 1780053 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:27:16 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:27:16 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:27:16 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:27:16 kernel: Call Trace:
Dec 3 02:27:16 kernel: <TASK>
Dec 3 02:27:16 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:27:16 kernel: ? __schedule+0xfe/0x620
Dec 3 02:27:16 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:27:16 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:27:16 kernel: ? schedule+0x23/0xa0
Dec 3 02:27:16 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:27:16 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:27:16 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:27:16 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:27:16 kernel: ? kthread+0xcc/0x100
Dec 3 02:27:16 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:27:16 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:27:16 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:27:16 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:27:16 kernel: </TASK>
Dec 3 02:30:26 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 3 02:30:26 kernel: rcu: #01117-...!: (0 ticks this GP) idle=b124/1/0x4000000000000002 softirq=2109356/2109356 fqs=0
Dec 3 02:30:26 kernel: rcu: #011(detected by 1, t=1960059 jiffies, g=11126601, q=12298028 ncpus=40)
Dec 3 02:30:26 kernel: Sending NMI from CPU 1 to CPUs 17:
Dec 3 02:30:26 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 1970056 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 3 02:30:26 kernel: rcu: #011Possible timer handling issue on cpu=17 timer-softirq=273727
Dec 3 02:30:26 kernel: rcu: rcu_preempt kthread starved for 1970059 jiffies! g11126601 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=17
Dec 3 02:30:26 kernel: rcu: #011Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 3 02:30:26 kernel: rcu: RCU grace-period kthread stack dump:
Dec 3 02:30:26 kernel: task:rcu_preempt state:R stack:0 pid:18 tgid:18 ppid:2 flags:0x00004008
Dec 3 02:30:26 kernel: Call Trace:
Dec 3 02:30:26 kernel: <TASK>
Dec 3 02:30:26 kernel: ? __pick_next_task+0x3e/0x1a0
Dec 3 02:30:26 kernel: ? __schedule+0xfe/0x620
Dec 3 02:30:26 kernel: ? _raw_spin_unlock_irqrestore+0xa/0x30
Dec 3 02:30:26 kernel: ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 3 02:30:26 kernel: ? schedule+0x23/0xa0
Dec 3 02:30:26 kernel: ? schedule_timeout+0x8b/0x160
Dec 3 02:30:26 kernel: ? __pfx_process_timeout+0x10/0x10
Dec 3 02:30:26 kernel: ? rcu_gp_fqs_loop+0x10b/0x500
Dec 3 02:30:26 kernel: ? rcu_gp_kthread+0x13f/0x1d0
Dec 3 02:30:26 kernel: ? kthread+0xcc/0x100
Dec 3 02:30:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:30:26 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:30:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:30:26 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:30:26 kernel: </TASK>
Dec 3 02:30:26 kernel: ? kthread+0xcc/0x100
Dec 3 02:30:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:30:26 kernel: ? ret_from_fork+0x2d/0x50
Dec 3 02:30:26 kernel: ? __pfx_kthread+0x10/0x10
Dec 3 02:30:26 kernel: ? ret_from_fork_asm+0x1a/0x30
Dec 3 02:30:26 kernel: </TASK>