Re: [BUG] mm: writes after MADV_FREE get lost in THP pages under cgroup limits

[Vernon Yang]

On Thu, Sep 03, 2026 at 12:31:51AM +0200, Orson Peters wrote:
> Dear reader(s),
>
> Below is a small user-space C reproducer that, when run with
> transparent huge pages and under cgroup limits, causes writes after
> MADV_FREE to be completely lost. I have reproduced this on
> 7.3.0-070300rc1-generic on an AWS c7a.8xlarge machine.
>
>
> Best,
>
> Orson Peters
>
>
>
> // Reproducer showing writes to madvise(MADV_FREE) pages getting lost. Requires
> // THP to be on and cgroup limits.
> //
> //   $ cat /sys/kernel/mm/transparent_hugepage/enabled
> //   always [madvise] never
> //   $ gcc -O2 -o madv_free_repro madv_free_repro.c
> //   $ systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0
> -p OOMPolicy=continue -- ./madv_free_repro
>
> #define _GNU_SOURCE
> #include <stdio.h>
> #include <string.h>
> #include <unistd.h>
> #include <fcntl.h>
> #include <sys/mman.h>
> #include <sys/wait.h>
>
> #define SIZE     (768UL << 20)
> #define PRESSURE (1536UL << 20)
> #define PAGE     4096UL
>
> // Touch PRESSURE bytes in a child so the cgroup has to reclaim the
> parent's buffer.
> // The child volunteers as the OOM victim, so it dies instead of us.
> static void squeeze(void) {
>     pid_t pid = fork();
>     if (pid == 0) {
>         int fd = open("/proc/self/oom_score_adj", O_WRONLY);
>         if (fd >= 0 && write(fd, "1000", 4)) close(fd);
>         for (size_t done = 0; done < PRESSURE; done += 64UL << 20) {
>             void *p = mmap(NULL, 64UL << 20, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>             if (p == MAP_FAILED) _exit(1);
>             memset(p, 1, 64UL << 20);
>         }
>         _exit(0);
>     }
>     waitpid(pid, NULL, 0);
>     sleep(1);
> }
>
> int main(void) {
>     int bad_rounds = 0;
>
>     for (int round = 0; round < 10; round++) {
>         char *buf = mmap(NULL, SIZE, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>         if (buf == MAP_FAILED) return perror("mmap"), 1;
>         madvise(buf, SIZE, MADV_HUGEPAGE);
>
>         // Set buffer to 0x5a, again after hinting MADV_FREE.
>         memset(buf, 0x5A, SIZE);
>         madvise(buf, SIZE, MADV_FREE);
>         memset(buf, 0x5A, SIZE);
>
>         // Trigger the bug - on a multi-NUMA region machine simply
> reading the buffer in a hot loop
>         // for 2 seconds also worked, likely due to migration.
>         mprotect(buf, SIZE, PROT_READ);
>         mprotect(buf, SIZE, PROT_READ | PROT_WRITE);
>
>         // Trigger cgroup oom killer.
>         squeeze();
>
>         size_t lost = 0;
>         long first = -1;
>         for (size_t off = 0; off < SIZE; off += PAGE)
>             if (buf[off] != 0x5A) {
>                 if (first < 0) first = off / PAGE;
>                 lost++;
>             }
>
>         printf("round %d: %6zu of %zu pages are no longer 0x5A",
> round, lost, SIZE / PAGE);
>         if (lost) printf("  (from page %ld, they read 0x%02x)", first,
> buf[first * PAGE]);
>         printf("\n");
>         fflush(stdout);
>
>         bad_rounds += lost > 0;
>         munmap(buf, SIZE);
>     }
>
>     printf("\n%d of 10 rounds lost data that had been written\n", bad_rounds);
>     return bad_rounds != 0;
> }
>

Hi Orson,

Thank you for your feedback. I have also reproduced this bug locally. I
am providing a fix patch [1] that can resolve this bug. Please test again to
see if it solves the issue on your end, Thanks!

[1] https://lore.kernel.org/linux-mm/20260903031608.1194238-1-vernon2gm@xxxxxxxxx/

--
Cheers,
Vernon