[PATCH v3] mm: memcontrol: don't miss processes whose leader has exited in OOM scans
From: Zhan Xusheng
Date: Sat Nov 21 2026 - 02:32:36 EST
From: Zhan Xusheng <zhanxusheng@xxxxxxxxxx>
mem_cgroup_scan_tasks() walks with CSS_TASK_ITER_PROCS, so it sees only
thread group leaders, plus a dying leader that still has live threads.
Moving a process whose leader has exited, by writing its PID to
cgroup.procs, moves the live threads but leaves the leader behind, as
cgroup_migrate_add_task() skips exiting tasks.
The destination's OOM killer then cannot see the process. If it is
alone there, every attempt ends with "Out of memory and no killable
processes..." and the charge is retried indefinitely, and
memory.oom.group kills nothing; otherwise another process is killed in
its place. The source's OOM killer still finds the leader, reaches the
moved threads' mm through find_lock_task_mm(), and kills a process that
has left the memcg.
Walk all tasks and let one task of each thread group in the memcg stand
for it: the first one that is not exiting, the first one that has not
exited yet once the whole group is exiting, or the leader once every
thread has exited, so that an OOM victim stays visible until it is gone,
as the leader does today. Exiting threads left behind do not count
while the group has live threads elsewhere.
Tested in a VM with memory.max=64M: in each case above the process using
the memory is now killed, and a process whose exited leader was not
moved, or whose leader is alive, is killed as before, with and without
memory.oom.group.
Fixes: f168a9a54ec3 ("mm: memcontrol: use CSS_TASK_ITER_PROCS at mem_cgroup_scan_tasks()")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Zhan Xusheng <zhanxusheng@xxxxxxxxxx>
---
v3:
- Rebase on "mm: memcontrol: don't lose track of exiting OOM victims"
in mm-hotfixes-unstable and keep CSS_TASK_ITER_WITH_DEAD
- Let the leader stand for the group once every thread has exited, so
such a victim stays visible as it does with that patch alone
Moving the exited leader along with its threads came up on v1, and
there has been no answer from the cgroup side. cgroup.kill had the
same case: da13e0e72a73 ("cgroup: fix cgroup.kill for exited thread
group leader") walks all threads instead of the leaders, and
5f4a3f743acc ("cgroup: don't kill migrated processes through their
exiting threads") then had to keep an exiting thread left behind from
standing for a process that has moved out. This patch does the same
for the memcg scan; the second case is what the live_elsewhere check
covers.
Retested on mm-hotfixes-unstable 2ba4d2c9ae3c: the cases in the
changelog behave as described, and with the reproducer from the review
of Chris's patch, memory.max=400M, 10 runs, oom_kill is 1 in every run
with and without this patch.
v2: https://lore.kernel.org/all/20261027144016.2582880-1-zhanxusheng1024@xxxxxxxxx/
v1: https://lore.kernel.org/all/20261026120248.2188024-1-zhanxusheng1024@xxxxxxxxx/
mm/memcontrol.c | 44 +++++++++++++++++++++++++++++++++++++++++---
1 file changed, 41 insertions(+), 3 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index afbb638e7e1c..3355341666db 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1426,6 +1426,44 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
dead_memcg);
}
+/*
+ * Let one task of each thread group in @memcg stand for the group: the
+ * first one that is not exiting, or, once the whole group is exiting, the
+ * first one that has not exited yet, so that an OOM victim stays visible.
+ * The leader is not enough: a group whose leader has exited can be moved
+ * without it. Exiting threads left behind in @memcg do not count while the
+ * group has live threads elsewhere.
+ */
+static bool mem_cgroup_scan_skip(struct mem_cgroup *memcg,
+ struct task_struct *task)
+{
+ struct task_struct *t, *dying = NULL;
+ bool live_elsewhere = false;
+
+ guard(rcu)();
+ __for_each_thread(task->signal, t) {
+ bool here;
+
+ if (READ_ONCE(t->exit_state))
+ continue;
+ here = mem_cgroup_from_task(t) == memcg;
+ if (!(READ_ONCE(t->flags) & PF_EXITING)) {
+ if (here)
+ return t != task;
+ live_elsewhere = true;
+ } else if (here && !dying) {
+ dying = t;
+ }
+ }
+ /*
+ * Once every thread has exited, the leader stands for the group until
+ * it is gone, as with CSS_TASK_ITER_PROCS.
+ */
+ if (!dying)
+ return live_elsewhere || !thread_group_leader(task);
+ return live_elsewhere || dying != task;
+}
+
/**
* mem_cgroup_scan_tasks - iterate over tasks of a memory cgroup hierarchy
* @memcg: hierarchy root
@@ -1456,10 +1494,10 @@ void mem_cgroup_scan_tasks(struct mem_cgroup *memcg,
* torn down. Keep it visible so oom_evaluate_task() can
* back off while there's still memory to free.
*/
- css_task_iter_start(&iter->css, CSS_TASK_ITER_PROCS |
- CSS_TASK_ITER_WITH_DEAD, &it);
+ css_task_iter_start(&iter->css, CSS_TASK_ITER_WITH_DEAD, &it);
while (!ret && (task = css_task_iter_next(&it))) {
- ret = fn(task, arg);
+ if (!mem_cgroup_scan_skip(iter, task))
+ ret = fn(task, arg);
/* Avoid potential softlockup warning */
cond_resched();
}
base-commit: 2ba4d2c9ae3cfdad96ee925c92b14d7acd886954
--
2.43.0