Re: [PATCH] mm: nommu: set MMF_OOM_SKIP in exit_mmap()

[Alexis Bouzigues]

On Thu, Oct 15, 2026 at 09:03:57AM +0200, Michal Hocko wrote:
> Are you really looking into the code or just relying on an LLM?
> Go and check to learn that the oom reaper is only compiled in with
> CONFIG_MMU=y

Hello,

Sorry about that, my earlier message used "reap" in the sense of
cleaning up the exited child in zombie state (waitpid), not in the sense
of oom-reaper.

I also went through the code in detail today manually using a 
backtrace from the reproducer. I set a breakpoint at oom_evaluate_task 
(containing the MMF_OOM_SKIP check) ran the reproducer on v7.3-rc7
through qemu-system-arm with gdb. I iterated to confirm the
breakpoint keeps getting hit.

Here is a backtrace of the reproducer when the parent PID is livelocked:

    oom_evaluate_task                  mm/oom_kill.c:354
    select_bad_process                 mm/oom_kill.c:373
    out_of_memory                      mm/oom_kill.c:1155
    __alloc_pages_may_oom              mm/page_alloc.c:4113
    __alloc_pages_slowpath             mm/page_alloc.c:5030
    __alloc_frozen_pages_noprof        mm/page_alloc.c:5449
    __alloc_pages_noprof               mm/page_alloc.c:5471
    alloc_pages_node_noprof            mm/page_alloc.c:5485
    alloc_pages_noprof                 include/linux/gfp.h:284
    get_free_pages_noprof              mm/page_alloc.c:5507
    alloc_pages_exact_noprof           mm/page_alloc.c:5641
    do_mmap_private                    mm/nommu.c:959
    do_mmap                            mm/nommu.c:1182
    vm_mmap_pgoff                      mm/util.c:581
    ksys_mmap_pgoff                    mm/nommu.c:1287


Inside oom_evaluate_task(), when the task is the zombie child pid
victim, without MMF_OOM_SKIP it aborts the loop from select_bad_process()
and sets oc->chosen to -1, preventing the oom killer from calling
oom_kill_process on oc->chosen

Since oc->chosen is not NULL, it does not hit the deadlock branch in
out_of_memory instead it returns !!oc->chosen (1).

Returning to __alloc_pages_may_oom(), out_of_memory(&oc) evaluates
non-zero, so did_some_progress from __alloc_pages_slowpath gets set
to 1.

Then in __alloc_pages_slowpath(), since did_some_progress was set
to 1, it jumps to the retry label

As long as the system is out of memory, __alloc_pages_may_oom and 
out_of_memory will get called, meaning that oom killer does not make
progress and that the parent PID busy spins in the allocation.

If the MMF_OOM_SKIP bit were to be set for the child zombie process, then
oom_evaluate_task would not abort and would select the parent, meaning the
oom killer can now call oom_kill_process on oc->chosen and progress.

So setting MMF_OOM_SKIP as soon as its memory gets freed (in exit_mmap)
would guard against the parent's mmap stalling oom killer, since
oom_evaluate_task would rule the child out and proceed.