[BUG] shmem: FALLOC_FL_PUNCH_HOLE vs fault-around race corrupts page cache / rss counters

[Ayush Ranjan]

Hi,

We are seeing shmem/tmpfs page cache corruption on production hosts
running a workload that punches holes in a memfd (hole-punch based
memory reclaim) while other threads and forked children fault the
same MAP_SHARED mapping. The kernel taints but does not oops:

  BUG: Bad page cache in process ...  pfn:...
  page dumped because: still mapped when deleted
  ...
  dentry name(?): "memfd:..."

and, more frequently, a paired rss-counter imbalance when the mm is
torn down, always exactly one PMD-order folio (512 pages):

  BUG: Bad rss-counter state mm:... type:MM_FILEPAGES  val:-512
  BUG: Bad rss-counter state mm:... type:MM_SHMEMPAGES val:512

Seen on 6.12 and 6.18, x86_64, bare metal and VM, with
/sys/kernel/mm/transparent_hugepage/shmem_enabled = always.

This looks like the same corruption Gregory reported in March, which
as far as I can tell stalled without a fix; that report needed ~100
ballooning VMs to reproduce:

  https://patchew.org/linux/20260326162611.693539-1-gourry@xxxxxxxxxx/

The reproducer at the end of this mail trips it with a single memfd,
no VMs or ballooning, within a few minutes on a large machine, so
hopefully it makes the race easier to confirm (and to test a fix
against).

Here is my best understanding of the race -- corrections welcome:

shmem guards faults against an in-progress hole punch with
inode->i_private: shmem_fault() -> shmem_falloc_wait() waits while
shmem_fallocate(PUNCH_HOLE) holds i_private. But shmem's .map_pages
is the generic filemap_map_pages() (shmem_vm_ops /
shmem_anon_vm_ops), which does not consult i_private and does not
take invalidate_lock, and shmem does not use invalidate_lock to
serialize faults against truncation the way regular filesystems do --
the i_private + waitq scheme stands in for it, but only shmem_fault()
participates in that scheme.

So while shmem_fallocate(PUNCH_HOLE) is between
unmap_mapping_range() and shmem_truncate_range(), a concurrent
fault-around can (re-)install PTEs for folios that are about to be
truncated:

  - filemap_map_pages() samples mm_counter_file(folio) once per batch
    and applies it with add_mm_counter() after mapping; if the
    folio's swapbacked state changes while it is concurrently torn
    down, the map-time counter (MM_FILEPAGES) and the zap-time
    counter (MM_SHMEMPAGES) disagree by exactly one folio -- the
    +/-512 imbalance above.

  - a folio re-mapped in this window (by fault-around directly, or
    via a child VMA whose PTEs copy_page_range() installs after
    unmap_mapping_range() has already walked the i_mmap tree -- the
    dup_mmap() variant discussed in the earlier thread) can be
    deleted from the page cache while still mapped -> "still mapped
    when deleted".

Reproducer
----------

The race is on PMD-order folios, so khugepaged needs to scan
aggressively (with the default 10s scan interval the punched ranges
are not re-collapsed fast enough to reproduce quickly):

  echo always > /sys/kernel/mm/transparent_hugepage/shmem_enabled
  cd /sys/kernel/mm/transparent_hugepage/khugepaged
  echo 1    > scan_sleep_millisecs
  echo 4096 > pages_to_scan
  echo 511  > max_ptes_none

  cc -O2 -pthread -o repro repro_shmem_punch_race.c
  for i in $(seq $(( $(nproc) / 3 ))); do ./repro 60 & done; wait
  # watch: dmesg -w

On a 112-CPU host this trips within ~2-5 minutes; this capture is
from 6.12.0-204.92.4.4.3.el9uek.x86_64:

  BUG: Bad rss-counter state mm:0000000078314ee0 type:MM_FILEPAGES  val:-512
  BUG: Bad rss-counter state mm:0000000078314ee0 type:MM_SHMEMPAGES val:512

The rss-counter form is the most frequent. The "still mapped when
deleted" form is what we mostly see in production but is rarer under
the reproducer (as in the earlier thread); I do not have a fresh
capture of it to paste here and will follow up with a full splat if I
catch one.

For background: we originally hit this under a memfd-backed sandbox
runtime (gVisor), which reclaims memory by punching holes in a
MAP_SHARED memfd while it is being faulted -- hence the "memfd:..."
dentry in the splats. The reproducer below has no such dependency: it
only uses memfd_create + mmap(MAP_SHARED) + fallocate(PUNCH_HOLE) +
madvise, so this appears to be a plain shmem issue rather than
anything specific to our setup.

Thanks,
Ayush

---- repro_shmem_punch_race.c ----

/*
 * Race FALLOC_FL_PUNCH_HOLE against fault-around on a MAP_SHARED
 * memfd mapping.
 *
 * One memfd is mapped MAP_SHARED into the main process and several
 * forked peer processes. All of them fault (and MADV_DONTNEED,
 * forcing re-fault via fault-around) random windows of the file while
 * the main process punches holes at random offsets. Short-lived
 * fork() children exercise the dup_mmap()/copy_page_range() variant.
 *
 * Usage: ./repro [seconds] [file_MiB] [peers]
 */

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>
#include <linux/falloc.h>
#include <linux/memfd.h>

#define PAGE 4096UL
#define HPAGE (2UL << 20)        /* PMD-order folio */

static unsigned char *map;
static int fd;
static size_t file_sz;
static volatile int stop;
static _Atomic long n_punch;

static inline uint64_t xorshift(uint64_t *s) {
	*s ^= *s << 13; *s ^= *s >> 7; *s ^= *s << 17; return *s;
}
static uint64_t seed(void) {
	struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t);
	return (t.tv_nsec ^ ((uint64_t)getpid() << 20) ^ (uint64_t)pthread_self()) | 1;
}

/* Read a random ~128 KiB span (spanning several fault-around batches), then drop
 * it so the next touch faults again through filemap_map_pages(). */
static void *faulter(void *arg) {
	uint64_t s = seed();
	size_t span = 32 * PAGE;
	while (!stop) {
		size_t off = (xorshift(&s) % ((file_sz - span) / PAGE)) * PAGE;
		volatile unsigned char sink = 0;
		for (size_t o = 0; o < span; o += PAGE)
			sink += map[off + o];
		if (xorshift(&s) & 1)
			madvise(map + off, span, MADV_DONTNEED);
	}
	return NULL;
}

/* Keep folios present for the puncher to race against. */
static void *writer(void *arg) {
	uint64_t s = seed();
	size_t span = 64 * PAGE;
	while (!stop) {
		size_t off = (xorshift(&s) % ((file_sz - span) / PAGE)) * PAGE;
		memset(map + off, 0x5a, span);
	}
	return NULL;
}

/* Punch holes at random offsets, mixing PMD-aligned and unaligned/sub-PMD
 * ranges. */
static void *puncher(void *arg) {
	uint64_t s = seed();
	while (!stop) {
		size_t len, off;
		if (xorshift(&s) & 1) {                    /* unaligned, 4K..2M */
			len = ((xorshift(&s) % 512) + 1) * PAGE;
			off = (xorshift(&s) % ((file_sz - len) / PAGE)) * PAGE;
		} else {                                   /* PMD-aligned, 2M/4M */
			len = ((xorshift(&s) % 2) + 1) * HPAGE;
			off = (xorshift(&s) % ((file_sz - len) / HPAGE)) * HPAGE;
		}
		fallocate(fd, FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE,
			  (off_t)off, (off_t)len);
		atomic_fetch_add(&n_punch, 1);
	}
	return NULL;
}

/* dup_mmap()/copy_page_range() variant; bounded to one child per thread. */
static void *forker(void *arg) {
	uint64_t s = seed();
	size_t pages = file_sz / PAGE;
	while (!stop) {
		pid_t p = fork();
		if (p == 0) {
			volatile unsigned char sink = 0;
			for (int i = 0; i < 16; i++)
				sink += map[(xorshift(&s) % pages) * PAGE];
			_exit(0);
		}
		if (p > 0) waitpid(p, NULL, 0);
		else usleep(200);
	}
	return NULL;
}

/* A peer process: maps the same memfd and faults/forks it concurrently. */
static void peer(int secs) {
	pthread_t th[4];
	pthread_create(&th[0], NULL, faulter, NULL);
	pthread_create(&th[1], NULL, faulter, NULL);
	pthread_create(&th[2], NULL, writer,  NULL);
	pthread_create(&th[3], NULL, forker,  NULL);
	sleep(secs + 2);
	_exit(0);
}

int main(int argc, char **argv) {
	int secs  = argc > 1 ? atoi(argv[1]) : 60;
	file_sz   = (argc > 2 ? (size_t)atol(argv[2]) : 64) << 20;
	int peers = argc > 3 ? atoi(argv[3]) : 2;

	fd = memfd_create("repro", MFD_CLOEXEC);
	if (fd < 0) { perror("memfd_create"); return 1; }
	if (ftruncate(fd, file_sz)) { perror("ftruncate"); return 1; }
	map = mmap(NULL, file_sz, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
	if (map == MAP_FAILED) { perror("mmap"); return 1; }
	madvise(map, file_sz, MADV_HUGEPAGE);   /* request PMD-order folios */
	memset(map, 1, file_sz);

	pid_t pid[64];
	if (peers > 64) peers = 64;
	for (int i = 0; i < peers; i++) {
		pid[i] = fork();
		if (pid[i] == 0) peer(secs);    /* child shares the memfd + mapping */
	}

	enum { NFAULT = 3, NFORK = 1 };
	pthread_t th[NFAULT + NFORK + 2];
	int n = 0;
	for (int i = 0; i < NFAULT; i++) pthread_create(&th[n++], NULL, faulter, NULL);
	for (int i = 0; i < NFORK;  i++) pthread_create(&th[n++], NULL, forker,  NULL);
	pthread_create(&th[n++], NULL, writer,  NULL);
	pthread_create(&th[n++], NULL, puncher, NULL);

	sleep(secs);
	stop = 1;
	for (int i = 0; i < n; i++) pthread_join(th[i], NULL);
	for (int i = 0; i < peers; i++) { kill(pid[i], SIGKILL); waitpid(pid[i], NULL, 0); }
	while (waitpid(-1, NULL, WNOHANG) > 0) {}

	fprintf(stderr, "pid %d: punches=%ld\n", getpid(), atomic_load(&n_punch));
	munmap(map, file_sz);
	close(fd);
	return 0;
}