[PATCH v2 0/2] mm/memory: reuse the whole exclusive large folio on a write fault

[Yuan-Hao Hsu]

After fork() and the child's exit, the parent's write faults reuse a
large anonymous folio one PTE at a time, although
wp_can_reuse_anon_folio() has already found the whole folio exclusive.

v2, after the comments on v1 [1]:
- Split in two.  1/2 handles an aligned block of 16 PTEs around the
  fault, the contpte-sized version David was fine with in the 2024
  discussion of Barry's RFC [2].  2/2 lifts that to the folio.
- The earlier discussion is linked; the changelogs say how its two
  reservations, the latency of the individual fault and how far to go
  around it, are answered.
- The bound of the walk is spelled out (Barry): folio, VMA and block or
  page table, each PTE read once by folio_pte_batch_flags().
- Changelogs cut to what is needed to judge the change.
- Same base as v1; 1/2 + 2/2 is the code of v1.

Two points from the AI review of v1, both done as mprotect() does them:
change_pte_range() does not flush_cache_range() before making PTEs
writable, and it makes clean exclusive anonymous PTEs writable without
pte_mkdirty() (the dirty rule is for shared file mappings, see
can_change_shared_pte_writable()).

Controls, unchanged: order-0 pages, PMD-mapped THPs, the COW copy path
and the order-0 and cow/fork/write-fault modes of David's
pte-mapped-folio-benchmarks.

What does not get faster on x86: stores to pages this CPU still holds
a read-only TLB entry for.  The fault makes the PTEs writable but, like
mprotect(), does not flush, so such a page takes one spurious fault,
about the cost of the reuse fault it replaces.  That is the case for
pages read since fork(), and for a loop that only stores one byte per
page in ascending order (David's reuse-byte mode): the CPU runs the
next stores speculatively while the first one faults and caches their
read-only translations.  Shown with kprobes (135,687 handle_mm_fault()
for 8,457 do_wp_page()) and an LFENCE after every store (4,100 faults
instead of 65,400); the untouched PMD-mapped case behaves the same.  A
flush_tlb_local() in the helper would fix it (that loop 28 -> 4 ms,
memset() 38 -> 21 ms, +140 ns per fault), but generic code has no way
to ask x86 for a flush that stays on this CPU, so that is for later.

Tested with DEBUG_VM, DEBUG_VM_PGTABLE, PROVE_LOCKING and
PAGE_TABLE_CHECK_ENFORCED: the mm selftests, a 12-scenario COW test
(child alive, vmsplice, PROT_READ VMA inside the folio, soft-dirty and
uffd-wp counts, mremap, holes, pageout, FOLL_FORCE), a
fork/pageout/mprotect/vmsplice stress, NUMA balancing on numa=fake=2
(protnone PTEs left alone), and arm64 under QEMU for the counters.
Cross-built for arm64 4K/16K/64K, i386 with and without PAE, x86
without THP, arm, arm nommu, riscv64, powerpc64le and s390x.

[1] https://lore.kernel.org/r/20260918064238.868-1-aa9736195201@xxxxxxxxx
[2] https://lore.kernel.org/r/20240831092339.66085-1-21cnbao@xxxxxxxxx

Yuan-Hao Hsu (2):
  mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault
  mm/memory: reuse the whole exclusive large folio on a write fault

 mm/memory.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)


base-commit: 238650ef6c7c7cca08e032527329424c9fbd70e5
-- 
2.43.0