Re: [PATCH -v2] cgroup: fix deadlock caused by cgroup_mutex and cpu_hotplug_lock

[chenridong]

On 2024/7/26 21:04, Michal Koutný wrote:
> Hello.
>
> On Thu, Jul 25, 2024 at 09:48:36AM GMT, chenridong <chenridong@xxxxxxxxxx> wrote:
> > > > This issue can be reproduced by the following methods:
> > > > 1. A large number of cpuset cgroups are deleted.
> > > > 2. Set cpu on and off repeatly.
> > > > 3. Set watchdog_thresh repeatly.
>
> BTW I assume this is some stress testing, not a regular use scenario of
> yours, right?
Yes, I have offered the scripts in Link(V1).

> > > > The reason for this issue is cgroup_mutex and cpu_hotplug_lock are
> > > > acquired in different tasks, which may lead to deadlock.
> > > > It can lead to a deadlock through the following steps:
> > > > 1. A large number of cgroups are deleted, which will put a large
> > > >      number of cgroup_bpf_release works into system_wq. The max_active
> > > >      of system_wq is WQ_DFL_ACTIVE(256). When cgroup_bpf_release can not
> > > >      get cgroup_metux, it may cram system_wq, and it will block work
> > > >      enqueued later.
>
> Who'd be the holder of cgroup_mutex preventing cgroup_bpf_release from
> progress? (That's not clear to me from your diagram.)
This is a cumulative process. The stress testing deletes a large member 
of cgroups, and cgroup_bpf_release is asynchronous, competing with 
cgroup release works. You know, cgroup_mutex is used in many places. 
Finally, the number of `cgroup_bpf_release` instances in system_wq 
accumulates up to 256, and it leads to this issue.


> ...
> > > Given idle worker created independent of WQ_DFL_ACTIVE before handling
> > > work item, no deadlock could rise in your scenario above.
> >
> > Hello Hillf, did you mean to say this issue couldn't happen?
>
> Ridong, can you reproduce this with CONFIG_PROVE_LOCKING (or do you have
> lockdep message from it aready)? It'd be helpful to get insight into
> the suspected dependencies.
>
> Thanks,
> Michal


There is a part of logs, I hope that is enough.
Thanks.

[  955.972520]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  955.976820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  955.979235] task:kworker/0:0     state:D stack:14824 pid:8     tgid:8 
    ppid:2      flags:00
[  955.982879] Workqueue: events cgroup_bpf_release
[  955.985403] Call Trace:
[  955.986851]  <TASK>
[  955.988108]  __schedule+0x393/0x1650
[  955.990202]  ? find_held_lock+0x2b/0x80
[  955.991481]  ? find_held_lock+0x2b/0x80
[  955.992734]  schedule+0x3e/0x130
[  955.993661]  schedule_preempt_disabled+0x15/0x30
[  955.995196]  __mutex_lock+0x6a7/0xce0
[  955.996443]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  955.997280]  ? cgroup_bpf_release+0x5c/0x360
[  955.999220]  ? cgroup_bpf_release+0x5c/0x360
[  956.000452]  ? __pfx_delay_tsc+0x10/0x10
[  956.001390]  cgroup_bpf_release+0x5c/0x360
[  956.002865]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.004836]  process_one_work+0x1f0/0x610
[  956.005974]  worker_thread+0x183/0x340
[  956.007115]  ? __pfx_worker_thread+0x10/0x10
[  956.008276]  kthread+0xd6/0x110
[  956.009291]  ? __pfx_kthread+0x10/0x10
[  956.010474]  ret_from_fork+0x34/0x50
[  956.011581]  ? __pfx_kthread+0x10/0x10
[  956.012741]  ret_from_fork_asm+0x1a/0x30
[  956.014069]  </TASK>
[  956.014875] INFO: task kworker/0:1:9 blocked for more than 327 seconds.
[  956.016724]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.018531] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.020718] task:kworker/0:1     state:D stack:14648 pid:9     tgid:9 
    ppid:2      flags:00
[  956.021664] Workqueue: events cgroup_bpf_release
[  956.024475] Call Trace:
[  956.025225]  <TASK>
[  956.025913]  __schedule+0x393/0x1650
[  956.026947]  ? find_held_lock+0x2b/0x80
[  956.027862]  ? find_held_lock+0x2b/0x80
[  956.029165]  schedule+0x3e/0x130
[  956.030012]  schedule_preempt_disabled+0x15/0x30
[  956.031474]  __mutex_lock+0x6a7/0xce0
[  956.032729]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.034174]  ? cgroup_bpf_release+0x5c/0x360
[  956.035561]  ? cgroup_bpf_release+0x5c/0x360
[  956.036839]  ? __pfx_delay_tsc+0x10/0x10
[  956.038046]  cgroup_bpf_release+0x5c/0x360
[  956.039261]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.041119]  process_one_work+0x1f0/0x610
[  956.042348]  worker_thread+0x183/0x340
[  956.043531]  ? __pfx_worker_thread+0x10/0x10
[  956.044993]  kthread+0xd6/0x110
[  956.045171]  ? __pfx_kthread+0x10/0x10
[  956.045306]  ret_from_fork+0x34/0x50
[  956.045428]  ? __pfx_kthread+0x10/0x10
[  956.045552]  ret_from_fork_asm+0x1a/0x30
[  956.046025]  </TASK>
[  956.046162] INFO: task kworker/2:0:30 blocked for more than 327 seconds.
[  956.046400]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.046566] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.047068] task:kworker/2:0     state:D stack:14152 pid:30 
tgid:30    ppid:2      flags:00
[  956.047377] Workqueue: events cgroup_bpf_release
[  956.047539] Call Trace:
[  956.047809]  <TASK>
[  956.047927]  __schedule+0x393/0x1650
[  956.048068]  ? find_held_lock+0x2b/0x80
[  956.048224]  ? find_held_lock+0x2b/0x80
[  956.048452]  schedule+0x3e/0x130
[  956.048561]  schedule_preempt_disabled+0x15/0x30
[  956.048881]  __mutex_lock+0x6a7/0xce0
[  956.049039]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.049203]  ? cgroup_bpf_release+0x5c/0x360
[  956.049441]  ? cgroup_bpf_release+0x5c/0x360
[  956.049584]  ? __pfx_delay_tsc+0x10/0x10
[  956.049887]  cgroup_bpf_release+0x5c/0x360
[  956.050038]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.050247]  process_one_work+0x1f0/0x610
[  956.050395]  worker_thread+0x183/0x340
[  956.050570]  ? __pfx_worker_thread+0x10/0x10
[  956.050925]  kthread+0xd6/0x110
[  956.051049]  ? __pfx_kthread+0x10/0x10
[  956.051177]  ret_from_fork+0x34/0x50
[  956.051294]  ? __pfx_kthread+0x10/0x10
[  956.051414]  ret_from_fork_asm+0x1a/0x30
[  956.051604]  </TASK>
[  956.051804] INFO: task kworker/3:0:35 blocked for more than 327 seconds.
[  956.052038]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.052222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.052424] task:kworker/3:0     state:D stack:14968 pid:35 
tgid:35    ppid:2      flags:00
[  956.052964] Workqueue: events cgroup_bpf_release
[  956.053199] Call Trace:
[  956.053304]  <TASK>
[  956.053397]  __schedule+0x393/0x1650
[  956.053536]  ? find_held_lock+0x2b/0x80
[  956.053801]  ? find_held_lock+0x2b/0x80
[  956.054017]  schedule+0x3e/0x130
[  956.054166]  schedule_preempt_disabled+0x15/0x30
[  956.054312]  __mutex_lock+0x6a7/0xce0
[  956.054454]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.054645]  ? cgroup_bpf_release+0x5c/0x360
[  956.055025]  ? cgroup_bpf_release+0x5c/0x360
[  956.055200]  ? __pfx_delay_tsc+0x10/0x10
[  956.055347]  cgroup_bpf_release+0x5c/0x360
[  956.055479]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.055846]  process_one_work+0x1f0/0x610
[  956.056026]  worker_thread+0x183/0x340
[  956.056211]  ? __pfx_worker_thread+0x10/0x10
[  956.056354]  kthread+0xd6/0x110
[  956.056511]  ? __pfx_kthread+0x10/0x10
[  956.056636]  ret_from_fork+0x34/0x50
[  956.056949]  ? __pfx_kthread+0x10/0x10
[  956.057091]  ret_from_fork_asm+0x1a/0x30
[  956.057315]  </TASK>
[  956.057430] INFO: task kworker/2:1:48 blocked for more than 327 seconds.
[  956.057602]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.058009] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.058291] task:kworker/2:1     state:D stack:15016 pid:48 
tgid:48    ppid:2      flags:00
[  956.058555] Workqueue: events cgroup_bpf_release
[  956.058727] Call Trace:
[  956.058984]  <TASK>
[  956.059082]  __schedule+0x393/0x1650
[  956.059260]  ? find_held_lock+0x2b/0x80
[  956.059382]  ? find_held_lock+0x2b/0x80
[  956.059524]  schedule+0x3e/0x130
[  956.059636]  schedule_preempt_disabled+0x15/0x30
[  956.059965]  __mutex_lock+0x6a7/0xce0
[  956.060096]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.060344]  ? cgroup_bpf_release+0x5c/0x360
[  956.060529]  ? cgroup_bpf_release+0x5c/0x360
[  956.060828]  ? __pfx_delay_tsc+0x10/0x10
[  956.060980]  cgroup_bpf_release+0x5c/0x360
[  956.061114]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.061412]  process_one_work+0x1f0/0x610
[  956.061614]  worker_thread+0x183/0x340
[  956.062090]  ? __pfx_worker_thread+0x10/0x10
[  956.062278]  kthread+0xd6/0x110
[  956.062395]  ? __pfx_kthread+0x10/0x10
[  956.062544]  ret_from_fork+0x34/0x50
[  956.062807]  ? __pfx_kthread+0x10/0x10
[  956.062955]  ret_from_fork_asm+0x1a/0x30
[  956.063167]  </TASK>
[  956.063276] INFO: task kworker/3:1:49 blocked for more than 327 seconds.
[  956.063466]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.063646] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.064045] task:kworker/3:1     state:D stack:13592 pid:49 
tgid:49    ppid:2      flags:00
[  956.064339] Workqueue: events cgroup_bpf_release
[  956.064488] Call Trace:
[  956.064571]  <TASK>
[  956.064864]  __schedule+0x393/0x1650
[  956.065082]  ? find_held_lock+0x2b/0x80
[  956.065222]  ? find_held_lock+0x2b/0x80
[  956.065395]  schedule+0x3e/0x130
[  956.065521]  schedule_preempt_disabled+0x15/0x30
[  956.065945]  __mutex_lock+0x6a7/0xce0
[  956.066078]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.066240]  ? cgroup_bpf_release+0x5c/0x360
[  956.066396]  ? cgroup_bpf_release+0x5c/0x360
[  956.066542]  ? __pfx_delay_tsc+0x10/0x10
[  956.066777]  cgroup_bpf_release+0x5c/0x360
[  956.067008]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.067225]  process_one_work+0x1f0/0x610
[  956.067411]  worker_thread+0x183/0x340
[  956.067554]  ? __pfx_worker_thread+0x10/0x10
[  956.067934]  kthread+0xd6/0x110
[  956.068092]  ? __pfx_kthread+0x10/0x10
[  956.068234]  ret_from_fork+0x34/0x50
[  956.068426]  ? __pfx_kthread+0x10/0x10
[  956.068554]  ret_from_fork_asm+0x1a/0x30
[  956.068962]  </TASK>
[  956.069129] INFO: task kworker/3:2:69 blocked for more than 327 seconds.
[  956.069319]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.069484] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.069842] task:kworker/3:2     state:D stack:14576 pid:69 
tgid:69    ppid:2      flags:00
[  956.070163] Workqueue: events cgroup_bpf_release
[  956.070312] Call Trace:
[  956.070419]  <TASK>
[  956.070510]  __schedule+0x393/0x1650
[  956.070948]  ? find_held_lock+0x2b/0x80
[  956.071104]  ? find_held_lock+0x2b/0x80
[  956.071280]  schedule+0x3e/0x130
[  956.071415]  schedule_preempt_disabled+0x15/0x30
[  956.071574]  __mutex_lock+0x6a7/0xce0
[  956.071888]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.072075]  ? cgroup_bpf_release+0x5c/0x360
[  956.072233]  ? cgroup_bpf_release+0x5c/0x360
[  956.072484]  ? __pfx_delay_tsc+0x10/0x10
[  956.072623]  cgroup_bpf_release+0x5c/0x360
[  956.073091]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.073395]  process_one_work+0x1f0/0x610
[  956.073582]  worker_thread+0x183/0x340
[  956.073981]  ? __pfx_worker_thread+0x10/0x10
[  956.074125]  kthread+0xd6/0x110
[  956.074308]  ? __pfx_kthread+0x10/0x10
[  956.074441]  ret_from_fork+0x34/0x50
[  956.074555]  ? __pfx_kthread+0x10/0x10
[  956.074834]  ret_from_fork_asm+0x1a/0x30
[  956.075067]  </TASK>
[  956.075407] INFO: task kworker/2:2:78 blocked for more than 327 seconds.
[  956.075593]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.076081] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.076375] task:kworker/2:2     state:D stack:14992 pid:78 
tgid:78    ppid:2      flags:00
[  956.076814] Workqueue: events cgroup_bpf_release
[  956.076993] Call Trace:
[  956.077081]  <TASK>
[  956.077174]  __schedule+0x393/0x1650
[  956.077369]  ? find_held_lock+0x2b/0x80
[  956.077524]  ? find_held_lock+0x2b/0x80
[  956.077796]  schedule+0x3e/0x130
[  956.077935]  schedule_preempt_disabled+0x15/0x30
[  956.078112]  __mutex_lock+0x6a7/0xce0
[  956.078261]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.078474]  ? cgroup_bpf_release+0x5c/0x360
[  956.078631]  ? cgroup_bpf_release+0x5c/0x360
[  956.079004]  ? __pfx_delay_tsc+0x10/0x10
[  956.079149]  cgroup_bpf_release+0x5c/0x360
[  956.079358]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.079604]  process_one_work+0x1f0/0x610
[  956.080047]  worker_thread+0x183/0x340
[  956.080186]  ? __pfx_worker_thread+0x10/0x10
[  956.080369]  kthread+0xd6/0x110
[  956.080480]  ? __pfx_kthread+0x10/0x10
[  956.080608]  ret_from_fork+0x34/0x50
[  956.080954]  ? __pfx_kthread+0x10/0x10
[  956.081123]  ret_from_fork_asm+0x1a/0x30
[  956.081286]  </TASK>
[  956.081386] INFO: task kworker/0:2:110 blocked for more than 327 seconds.
[  956.081621]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.082080] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.082313] task:kworker/0:2     state:D stack:14856 pid:110 
tgid:110   ppid:2      flags:00
[  956.082643] Workqueue: events cgroup_bpf_release
[  956.082995] Call Trace:
[  956.083091]  <TASK>
[  956.083226]  __schedule+0x393/0x1650
[  956.083366]  ? find_held_lock+0x2b/0x80
[  956.083531]  ? find_held_lock+0x2b/0x80
[  956.083977]  schedule+0x3e/0x130
[  956.084161]  schedule_preempt_disabled+0x15/0x30
[  956.084314]  __mutex_lock+0x6a7/0xce0
[  956.084450]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.084613]  ? cgroup_bpf_release+0x5c/0x360
[  956.085053]  ? cgroup_bpf_release+0x5c/0x360
[  956.085200]  ? __pfx_delay_tsc+0x10/0x10
[  956.085340]  cgroup_bpf_release+0x5c/0x360
[  956.085508]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.085919]  process_one_work+0x1f0/0x610
[  956.086098]  worker_thread+0x183/0x340
[  956.086235]  ? __pfx_worker_thread+0x10/0x10
[  956.086367]  kthread+0xd6/0x110
[  956.086478]  ? __pfx_kthread+0x10/0x10
[  956.086610]  ret_from_fork+0x34/0x50
[  956.087015]  ? __pfx_kthread+0x10/0x10
[  956.087156]  ret_from_fork_asm+0x1a/0x30
[  956.087368]  </TASK>
[  956.087456] INFO: task kworker/0:3:111 blocked for more than 327 seconds.
[  956.087631]       Not tainted 6.10.0-rc6-00163-g661e504db04c-dirty #135
[  956.088141] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[  956.088397] task:kworker/0:3     state:D stack:14312 pid:111 
tgid:111   ppid:2      flags:00
[  956.088828] Workqueue: events cgroup_bpf_release
[  956.089092] Call Trace:
[  956.089200]  <TASK>
[  956.089308]  __schedule+0x393/0x1650
[  956.089480]  ? find_held_lock+0x2b/0x80
[  956.089624]  ? find_held_lock+0x2b/0x80
[  956.090053]  schedule+0x3e/0x130
[  956.090205]  schedule_preempt_disabled+0x15/0x30
[  956.090383]  __mutex_lock+0x6a7/0xce0
[  956.090503]  ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[  956.090889]  ? cgroup_bpf_release+0x5c/0x360
[  956.091069]  ? cgroup_bpf_release+0x5c/0x360
[  956.091211]  ? __pfx_delay_tsc+0x10/0x10
[  956.091368]  cgroup_bpf_release+0x5c/0x360
[  956.091498]  ? trace_event_raw_event_workqueue_execute_start+0x52/0x90
[  956.092000]  process_one_work+0x1f0/0x610
[  956.092216]  worker_thread+0x183/0x340
[  956.092417]  ? __pfx_worker_thread+0x10/0x10
[  956.092558]  kthread+0xd6/0x110
[  956.092883]  ? __pfx_kthread+0x10/0x10
[  956.093102]  ret_from_fork+0x34/0x50
[  956.093229]  ? __pfx_kthread+0x10/0x10
[  956.093363]  ret_from_fork_asm+0x1a/0x30
[  956.093547]  </TASK>
[  956.093642] Future hung task reports are suppressed, see sysctl 
kernel.hung_task_warnings
[  956.139943]
[  956.139943] Showing all locks held in the system:
[  956.140446] 4 locks held by systemd/1:
[  956.140645]  #0: ffff8881021503f8 (sb_writers#7){....}-{0:0}, at: 
do_rmdir+0xde/0x1b0
[  956.141933]  #1: ffff8881025c1350 
(&type->i_mutex_dir_key#7/1){....}-{3:3}, at: do_rmdir+0x100
[  956.142365]  #2: ffff888105dd17d0 
(&type->i_mutex_dir_key#8){....}-{3:3}, at: vfs_rmdir+0x5b/0
[  956.142887]  #3: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_kn_lock_live+0x4e/0x0
[  956.143460] 3 locks held by kworker/0:0/8:
[  956.143825]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.144198]  #1: ffffc9000004be60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.144560]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.145087] 3 locks held by kworker/0:1/9:
[  956.145256]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.145547]  #1: ffffc90000053e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.146180]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.146556] 3 locks held by kworker/2:0/30:
[  956.146926]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.147253]  #1: ffffc90000117e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.147632]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.148195] 3 locks held by kworker/3:0/35:
[  956.148331]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.148826]  #1: ffffc90000143e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.149171]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.149487] 1 lock held by khungtaskd/40:
[  956.149624]  #0: ffffffff82b58840 (rcu_read_lock){....}-{1:2}, at: 
debug_show_all_locks+0x36/0
[  956.150097] 3 locks held by kworker/2:1/48:
[  956.150252]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.150541]  #1: ffffc900001afe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.151173]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.151457] 3 locks held by kworker/3:1/49:
[  956.151619]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.152091]  #1: ffffc900001b7e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.152475]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.153026] 3 locks held by kworker/3:2/69:
[  956.153156]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.153471]  #1: ffffc90000dafe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.154094]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.154430] 3 locks held by kworker/2:2/78:
[  956.154589]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.155061]  #1: ffffc90000d5fe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.156230]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.156638] 3 locks held by kworker/0:2/110:
[  956.157016]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.157357]  #1: ffffc90000e43e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.157983]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.158303] 3 locks held by kworker/0:3/111:
[  956.158425]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.159041]  #1: ffffc90000e53e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.159390]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.159832] 3 locks held by kworker/3:3/113:
[  956.160005]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.160357]  #1: ffffc90000e6fe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.160912]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.161235] 3 locks held by kworker/0:4/122:
[  956.161388]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.161816]  #1: ffffc90000ebbe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.162164]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.162486] 3 locks held by kworker/0:5/124:
[  956.162605]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.163015]  #1: ffffc90000ec7e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.163406]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.163890] 3 locks held by kworker/3:4/126:
[  956.164045]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.164322]  #1: ffffc90000ed7e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.164906]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.165331] 4 locks held by systemd-udevd/127:
[  956.165463]  #0: ffff888103e376f8 (&p->lock){....}-{3:3}, at: 
seq_read_iter+0x59/0x4c0
[  956.166017]  #1: ffff888111081488 (&of->mutex){....}-{3:3}, at: 
kernfs_seq_start+0x27/0x110
[  956.166409]  #2: ffff8881008774d8 (kn->active#11){....}-{0:0}, at: 
kernfs_seq_start+0x2f/0x110
[  956.166953]  #3: ffff888237cae410 (&dev->mutex){....}-{3:3}, at: 
uevent_show+0x99/0x120
[  956.167248] 3 locks held by kworker/3:5/128:
[  956.167403]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.167965]  #1: ffffc90000e9be60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.168292]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.168611] 3 locks held by kworker/0:6/132:
[  956.168952]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.169238]  #1: ffffc90000e77e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.169578]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.170047] 3 locks held by kworker/2:3/173:
[  956.170179]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.170485]  #1: ffffc90000e0fe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.171090]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.171455] 3 locks held by kworker/2:4/182:
[  956.171603]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.172039]  #1: ffffc90000f57e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.172365]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.172801] 1 lock held by in:imklog/204:
[  956.173012] 1 lock held by sshd/346:
[  956.173153] 4 locks held by systemd-udevd/388:
[  956.173299]  #0: ffff888103e378c8 (&p->lock){....}-{3:3}, at: 
seq_read_iter+0x59/0x4c0
[  956.173633]  #1: ffff888111080088 (&of->mutex){....}-{3:3}, at: 
kernfs_seq_start+0x27/0x110
[  956.174083]  #2: ffff8881008774d8 (kn->active#11){....}-{0:0}, at: 
kernfs_seq_start+0x2f/0x110
[  956.174410]  #3: ffff888237cae410 (&dev->mutex){....}-{3:3}, at: 
uevent_show+0x99/0x120
[  956.174882] 7 locks held by cpu_up_down.sh/389:
[  956.175110]  #0: ffff8881021413f8 (sb_writers#5){....}-{0:0}, at: 
ksys_write+0x69/0xf0
[  956.175440]  #1: ffff888111080288 (&of->mutex#2){....}-{3:3}, at: 
kernfs_fop_write_iter+0xf7/0
[  956.176030]  #2: ffff8881008776e8 (kn->active#54){....}-{0:0}, at: 
kernfs_fop_write_iter+0xff0
[  956.176355]  #3: ffffffff82c12ce8 (device_hotplug_lock){....}-{3:3}, 
at: online_store+0x42/0x0
[  956.176623]  #4: ffff888237cae410 (&dev->mutex){....}-{3:3}, at: 
device_online+0x24/0x90
[  956.177126]  #5: ffffffff82a67a48 (cpu_add_remove_lock){....}-{3:3}, 
at: cpu_up+0x31/0xb0
[  956.177436]  #6: ffffffff82a679b0 (cpu_hotplug_lock){....}-{0:0}, at: 
_cpu_up+0x32/0x1e0
[  956.178182] 3 locks held by watchdog.sh/391:
[  956.178362]  #0: ffff8881021403f8 (sb_writers#4){....}-{0:0}, at: 
ksys_write+0x69/0xf0
[  956.179092]  #1: ffffffff82b6ea48 (watchdog_mutex){....}-{3:3}, at: 
proc_watchdog_thresh+0x2f0
[  956.179377]  #2: ffffffff82a679b0 (cpu_hotplug_lock){....}-{0:0}, at: 
__lockup_detector_recon0
[  956.179996] 3 locks held by kworker/2:5/4630:
[  956.180188]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.180503]  #1: ffffc90007cebe60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.181158]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.181517] 3 locks held by kworker/3:6/4635:
[  956.181640]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.182118]  #1: ffffc90007803e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.182542]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.183216] 3 locks held by kworker/0:7/4637:
[  956.183348]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.183827]  #1: ffffc90007ba3e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.184228]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.184523] 3 locks held by kworker/2:6/4640:
[  956.184812]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.185162]  #1: ffffc90007bf3e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.185518]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.186047] 3 locks held by kworker/3:7/4641:
[  956.186211]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.186508]  #1: ffffc90007d33e60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.187106]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.187443] 3 locks held by kworker/0:8/4642:
[  956.187565]  #0: ffff888100064548 
((wq_completion)events){....}-{0:0}, at: process_one_work+00
[  956.188177]  #1: ffffc90007d3be60 
((work_completion)(&cgrp->bpf.release_work)){....}-{0:0}, a0
[  956.188589]  #2: ffffffff82b67d08 (cgroup_mutex){....}-{3:3}, at: 
cgroup_bpf_release+0x5c/0x30
[  956.189051] 3 locks held by kworker/2:7/4643: