Re: [BUG] shmem: FALLOC_FL_PUNCH_HOLE vs fault-around race corrupts page cache / rss counters

[Baolin Wang]

On 9/24/26 4:34 PM, Pedro Falcato wrote:
> (please use the email I actually use for work, thanks; not sure how
> you got to that one)
>
> Hi,
>
> On Thu, Sep 24, 2026 at 06:16:21AM +0000, Ayush Ranjan wrote:
> > Hi,
> >
> > We are seeing shmem/tmpfs page cache corruption on production hosts
> > running a workload that punches holes in a memfd (hole-punch based
> > memory reclaim) while other threads and forked children fault the
> > same MAP_SHARED mapping. The kernel taints but does not oops:
> >
> >    BUG: Bad page cache in process ...  pfn:...
> >    page dumped because: still mapped when deleted
> >    ...
> >    dentry name(?): "memfd:..."
> >
> > and, more frequently, a paired rss-counter imbalance when the mm is
> > torn down, always exactly one PMD-order folio (512 pages):
> >
> >    BUG: Bad rss-counter state mm:... type:MM_FILEPAGES  val:-512
> >    BUG: Bad rss-counter state mm:... type:MM_SHMEMPAGES val:512
> >
> > Seen on 6.12 and 6.18, x86_64, bare metal and VM, with
> > /sys/kernel/mm/transparent_hugepage/shmem_enabled = always.
> >
> > This looks like the same corruption Gregory reported in March, which
> > as far as I can tell stalled without a fix; that report needed ~100
> > ballooning VMs to reproduce:
>
> Well, I thought (from the lack of replies) that it was probably a bug on
> their side. Perhaps that's not true :/
>
> >    https://patchew.org/linux/20260326162611.693539-1-gourry@xxxxxxxxxx/
> >
> > The reproducer at the end of this mail trips it with a single memfd,
> > no VMs or ballooning, within a few minutes on a large machine, so
> > hopefully it makes the race easier to confirm (and to test a fix
> > against).
> >
> > Here is my best understanding of the race -- corrections welcome:
> >
> > shmem guards faults against an in-progress hole punch with
> > inode->i_private: shmem_fault() -> shmem_falloc_wait() waits while
> > shmem_fallocate(PUNCH_HOLE) holds i_private. But shmem's .map_pages
> > is the generic filemap_map_pages() (shmem_vm_ops /
> > shmem_anon_vm_ops), which does not consult i_private and does not
> > take invalidate_lock, and shmem does not use invalidate_lock to
> > serialize faults against truncation the way regular filesystems do --
> > the i_private + waitq scheme stands in for it, but only shmem_fault()
> > participates in that scheme.
> >
> > So while shmem_fallocate(PUNCH_HOLE) is between
> > unmap_mapping_range() and shmem_truncate_range(), a concurrent
> > fault-around can (re-)install PTEs for folios that are about to be
> > truncated:
> >
> >    - filemap_map_pages() samples mm_counter_file(folio) once per batch
> >      and applies it with add_mm_counter() after mapping; if the
> >      folio's swapbacked state changes while it is concurrently torn
>
> But that cannot happen? We hold the folio lock in filemap_map_pages().
> The folio (naturally) cannot be torn down while we have the folio lock.
>
> >      down, the map-time counter (MM_FILEPAGES) and the zap-time
> >      counter (MM_SHMEMPAGES) disagree by exactly one folio -- the
> >      +/-512 imbalance above.
> >
> >    - a folio re-mapped in this window (by fault-around directly, or
> >      via a child VMA whose PTEs copy_page_range() installs after
> >      unmap_mapping_range() has already walked the i_mmap tree -- the
> >      dup_mmap() variant discussed in the earlier thread) can be
> >      deleted from the page cache while still mapped -> "still mapped
> >      when deleted".
>
> No, I don't think this paragraph is true. Page cache truncation (via
> truncate, or fallocate PUNCH_HOLE) takes the folio lock for each folio
> that is about to be truncated out. Mapping folios takes the folio lock
> as well, except in the fork() case where a myriad of weird interval tree
> + PTE lock interactions make it safe (AIUI).

Agree.

However, I did previously fix a race between filemap_map_pages() and 
truncation that caused incorrect folio mappings, and I believe this race 
also exists in shmem. Ayush, could you check whether that fix is present 
in your kernel?

f58df566524e ("mm: filemap: fix nr_pages calculation overflow in 
filemap_map_pages()")

> > Reproducer
> > ----------
> >
> > The race is on PMD-order folios, so khugepaged needs to scan
> > aggressively (with the default 10s scan interval the punched ranges
> > are not re-collapsed fast enough to reproduce quickly):
> >
> >    echo always > /sys/kernel/mm/transparent_hugepage/shmem_enabled
> >    cd /sys/kernel/mm/transparent_hugepage/khugepaged
> >    echo 1    > scan_sleep_millisecs
> >    echo 4096 > pages_to_scan
> >    echo 511  > max_ptes_none
> >
> >    cc -O2 -pthread -o repro repro_shmem_punch_race.c
> >    for i in $(seq $(( $(nproc) / 3 ))); do ./repro 60 & done; wait
> >    # watch: dmesg -w
> >
> > On a 112-CPU host this trips within ~2-5 minutes; this capture is
> > from 6.12.0-204.92.4.4.3.el9uek.x86_64:
>
> Awesome that you have a reproducer! Have you reproduced this on a mainline
> kernel? Enterprise kernels are not supported upstream.

I've been trying to reproduce the issue on v7.3.0-rc1 for half an hour 
now with Ayush's reproducer, but haven't been able to trigger it.