[PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio
From: David Hildenbrand (Arm)
Date: Wed Nov 18 2026 - 04:21:24 EST
Zapping PTEs in zap_pte_range() defers the TLB flush until after dropping
the PTL for clean page-cache folios.
A concurrent write fault in a MAP_PRIVATE mapping can then CoW the
page-cache folio and install a new anon folio without a prior TLB
flush:
CPU0 CPU1
LOAD(ADDR)
-> #PF
installs R/O PTE -> A
fuse_open()
-> invalidate_inode_pages2()
-> zap_pte_range()
clears R/O PTE
// delayed flush
// Other CPUs (incl CPU0) still cache ADDR -> A
STORE(ADDR)
-> #PF
do_cow_fault()
copies A to B, installs R/W PTE -> B
// no TLB flush: PTE was none
CPUs with stale TLB entries will still read from the old folio while writes
go to the new folio, until the TLB is flushed.
We really must process pending TLB flushes if we are not faulting the
same folio back in.
Fix it by flushing the TLB before installing an anon (CoW'ed) folio in
set_pte_range() if a TLB flush is pending. Zapping itself stays as fast
as before.
Other faults are not affected: unmap_mapping_*() users only remove
folios from the page-cache after their own TLB flush, so until then, a
refault maps the same folio again. The exception is a concurrent
invalidation that removes the already-unmapped folio from the
page-cache before the zapper flushed the TLB; that is an unlikely corner
case that we'll not handle for now.
Note that we don't care about MADV_DONTNEED followed by a new read
fault, as MADV_DONTNEED only guarantees its semantics once the operation
completed. CoW faults will flush the TLB either way.
For example, imagine that MADV_DONTNEED zaps a page-cache folio and
doesn't flush the TLB (but still holds a reference). If the page-cache
replaces the folio, a new read fault might map the new folio without
flushing the TLB. That's okay, as MADV_DONTNEED did not complete yet.
Same applies to MADV_DONTNEED of CoW'ed folios when re-faulting the
page-cache folio.
Memory reclaim, which removes folios from the page-cache before performing
its own batched TLB flush, will be handled separately.
Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Reported-by: Cenk Yılmaz <cenk@xxxxxxxxxxxxxxxxxxx>
Debugged-by: Cenk Yılmaz <cenk@xxxxxxxxxxxxxxxxxxx>
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: David Hildenbrand (Arm) <david@xxxxxxxxxx>
---
mm/memory.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/mm/memory.c b/mm/memory.c
index d2f48d78b0d7b..e1d9759e6fde1 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -5888,6 +5888,12 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
VM_BUG_ON_FOLIO(nr != 1, folio);
+ /*
+ * Concurrent zapping might have dropped the PTL but not flushed
+ * the TLB yet. Flush before installing a new folio.
+ */
+ if (mm_tlb_flush_pending(vma->vm_mm))
+ flush_tlb_page(vma, addr);
folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
folio_add_lru_vma(folio, vma);
} else {
--
2.56.1