Re: [PATCH 5/5] cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
From: Bert Karwatzki
Date: Sun May 31 2026 - 14:46:01 EST
Am Sonntag, dem 31.05.2026 um 11:19 +0200 schrieb Bert Karwatzki:
> Am Freitag, dem 29.05.2026 um 22:08 +0100 schrieb Mark Brown:
> > On Fri, May 29, 2026 at 07:25:29AM -1000, Tejun Heo wrote:
> > > On Wed, May 27, 2026 at 11:45:54AM +0100, Mark Brown wrote:
> > > > On Mon, May 04, 2026 at 02:51:21PM -1000, Tejun Heo wrote:
> >
> > > > with no further output and given that this is a cgroup locking change
> > > > this does seem like a plausible commmit, though I didn't look into it in
> > > > detail. Bisect log and the list of LTP tests we're running in our test
> > > > job below. We are running multuple tests in parallel.
> >
> > > Unfortunately, I can't reproduce this in my environment. Any chance you can
> > > try testing on x86 tooa nd see whether it produces there?
> >
> > Not readily sadly, I'll see if I can figure something out. Our rootfs
> > images are based on Debian Trixie if that's relevant?
>
> Using debian unstable (sid/forky) I can at least detect a timeout when running
> the ltp controller testsuite:
>
> # LTPROOT=/home/bert/ltp-install/ ./kirk --run-suite controllers
> Host information
> Hostname: homer
> Python: 3.13.12 (main, Feb 4 2026, 15:06:39) [GCC 15.2.0]
> Directory: /tmp/kirk.root/tmp092in2yb
>
> Connecting to SUT: default
>
> Suite: controllers
> ──────────────────
> cgroup_core01: pass (0.024s)
> cgroup_core02: pass (0.004s)
> cgroup_core03: pass (0.017s)
> cgroup: skip (2m 41s)
> memcg_regression: skip (3.414s)
> memcg_test_3: pass (0.090s)
> memcg_failcnt: skip (0.019s)
> memcg_force_empty: skip (0.015s)
> memcg_limit_in_bytes: skip (0.017s)
> memcg_stat_rss: skip (0.015s)
> memcg_subgroup_charge: skip (0.015s)
> memcg_max_usage_in_bytes: skip (0.014s)
> memcg_move_charge_at_immigrate: skip (0.014s)
> memcg_memsw_limit_in_bytes: skip (0.015s)
> memcg_stat: skip (0.015s)
> memcg_use_hierarchy: skip (0.015s)
> memcg_usage_in_bytes: skip (0.014s)
> memcg_stress: pass (30m 4s)
> memcg_control: pass (6.058s)
> memcontrol01: pass (0.004s)
> memcontrol02: pass (0.636s)
> memcontrol03: pass (15.983s)
> memcontrol04: pass (0.890s)
> cgroup_fj_function_debug: skip (0.013s)
> cgroup_fj_function_cpuset: skip (0.044s)
> cgroup_fj_function_cpu: skip (0.050s)
> cgroup_fj_function_cpuacct: pass (0.052s)
> cgroup_fj_function_memory: skip (0.042s)
> cgroup_fj_function_freezer: pass (0.044s)
> cgroup_fj_function_devices: pass (0.066s)
> cgroup_fj_function_blkio: skip (0.009s)
> cgroup_fj_function_net_cls: pass (0.073s)
> cgroup_fj_function_perf_event: pass (0.072s)
>
>
> Execution time: 1h 33m 13s
>
> Disconnecting from SUT: default
>
> Target information
> ──────────────────
> Kernel: Linux 7.1.0-rc5-next-20260528-master-dirty #480 SMP PREEMPT_RT Thu May 28 19:55:12 CEST 2026
> Cmdline: BOOT_IMAGE=/boot/vmlinuz-7.1.0-rc5-next-20260528-master-dirty
> root=UUID=3d5cdc5d-1902-40bf-9e16-ca819372d350
> ro
> quiet
> Machine: unknown
> Arch: x86_64
> RAM: 63439380 kB
> Swap: 78125052 kB
> Distro: debian
>
> ────────────────────────
> TEST SUMMARY
> ────────────────────────
> Suite: controllers
> Runtime: 33m 13s
> Runs: 347
>
> Results:
> Passed: 181
> Failed: 0
> Broken: 0
> Skipped: 350
> Warnings: 0
>
> Session stopped
>
> In dmesg I get messages about task tst_cgtl hanging:
>
> [ 2212.794669] [ T346] INFO: task tst_cgctl:317896 blocked for more than 122 seconds.
> [ 2212.794674] [ T346] Not tainted 7.1.0-rc5-next-20260528-master-dirty #480
> [ 2212.794675] [ T346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>
> [...]
>
> [ 3318.721344] [ T346] INFO: task tst_cgctl:317896 blocked for more than 1228 seconds.
> [ 3318.721349] [ T346] Not tainted 7.1.0-rc5-next-20260528-master-dirty #480
> [ 3318.721351] [ T346] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>
>
>
>
>
>
> On 6.19.14 the Results of this testrun is:
>
> # LTPROOT=/home/bert/ltp-install/ ./kirk --run-suite controllers
>
> [...]
>
> Target information
> ──────────────────
> Kernel: Linux 6.19.14-stable #1238 SMP PREEMPT_RT Sat May 30 17:28:29 CEST 2026
> Cmdline: BOOT_IMAGE=/boot/vmlinuz-6.19.14-stable
> root=UUID=3d5cdc5d-1902-40bf-9e16-ca819372d350
> ro
> quiet
> Machine: unknown
> Arch: x86_64
> RAM: 63436188 kB
> Swap: 78125052 kB
> Distro: debian
>
> ────────────────────────
> TEST SUMMARY
> ────────────────────────
> Suite: controllers
> Runtime: 36m 12s
> Runs: 347
>
> Results:
> Passed: 1742
> Failed: 0
> Broken: 0
> Skipped: 97
> Warnings: 0
>
> Session stopped
>
> With 6.19.14 I also get no hung tasks.
>
> On 7.0.10 the tests also work:
>
> root@homer:/mnt/data/linux-forest/kirk# LTPROOT=/home/bert/ltp-install/ ./kirk --run-suite controllers
> Host information
> Hostname: homer
> Python: 3.13.12 (main, Feb 4 2026, 15:06:39) [GCC 15.2.0]
> Directory: /tmp/kirk.root/tmpq32b09g7
>
> Connecting to SUT: default
>
> Suite: controllers
> ──────────────────
> cgroup_core01: pass (0.016s)
>
> [...]
>
> pids_9_100: pass (0.107s)
>
> Execution time: 36m 15s
>
> Disconnecting from SUT: default
>
> Target information
> ──────────────────
> Kernel: Linux 7.0.10-stable #1239 SMP PREEMPT_RT Sun May 31 00:42:41 CEST 2026
> Cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.10-stable
> root=UUID=3d5cdc5d-1902-40bf-9e16-ca819372d350
> ro
> quiet
> Machine: unknown
> Arch: x86_64
> RAM: 63435940 kB
> Swap: 78125052 kB
> Distro: debian
>
> ────────────────────────
> TEST SUMMARY
> ────────────────────────
> Suite: controllers
> Runtime: 36m 13s
> Runs: 347
>
> Results:
> Passed: 1742
> Failed: 0
> Broken: 0
> Skipped: 97
> Warnings: 0
>
> Session stopped
>
>
>
> I'm not sure if this is related to the problems on arm64, but I'll try bisecting this.
>
> Bert Karwatzki
I finished my bisectiOn (from v7.0.0 to next-20260528) and it shows
commit 1dffd95575eb ("cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()")
as first bad commit, too. During the bisection I had to apply this patch (when it's cleanly applicable)
diff --git a/fs/filesystems.c b/fs/filesystems.c
index 771fc31a69b8..712316a1e3e0 100644
--- a/fs/filesystems.c
+++ b/fs/filesystems.c
@@ -269,7 +269,7 @@ static __cold noinline int regen_filesystems_string(void)
hlist_for_each_entry_rcu(p, &file_systems, list) {
if (!(p->fs_flags & FS_REQUIRES_DEV))
newlen += strlen("nodev");
- newlen += strlen("\t") + strlen(p->name) + strlen("\n");
+ newlen += strlen("\t") + strlen(p->name) + strlen("\n");
}
spin_unlock(&file_systems_lock);
@@ -289,6 +289,7 @@ static __cold noinline int regen_filesystems_string(void)
* Did someone beat us to it?
*/
if (old && old->gen == file_systems_gen) {
+ spin_unlock(&file_systems_lock);
kfree(new);
return 0;
}
@@ -297,6 +298,7 @@ static __cold noinline int regen_filesystems_string(void)
* Did the list change in the meantime?
*/
if (gen != file_systems_gen) {
+ spin_unlock(&file_systems_lock);
kfree(new);
goto retry;
}
@@ -321,13 +323,12 @@ static __cold noinline int regen_filesystems_string(void)
* generation above and messes it up.
*/
spin_unlock(&file_systems_lock);
- if (old)
- kfree_rcu(old, rcu);
+ kfree(new);
return -EINVAL;
}
/*
- * Paired with consume fence in READ_ONCE() in filesystems_proc_show()
+ * Paired with consume fence in rcu_dereference() in filesystems_proc_show()
*/
smp_store_release(&file_systems_string, new);
spin_unlock(&file_systems_lock);
to take care of a locking issue in commit
36b3306779ea ("fs: cache the string generated by reading /proc/filesystems")
https://lore.kernel.org/all/20260520225245.2962-1-spasswolf@xxxxxx/
The test that hang when running
# LTPROOT=/home/bert/ltp-install/ ./kirk --run-suite controllers
is always cgroup_fj_function_net_prio.
Also when bisecting this I disabled (i.e. commented out) the
memcg_stress test in ~/ltp-install/runtest/controllers as it takes a lot of
time (30min) and succeeds even in the version where hangs occur.
Bert Karwatzki