[RFC PATCH v5 0/8] batch lookups in follow_page_mask()

[Rik van Riel]

follow_page_mask() walks the page tables one page at a time, even when the
caller asked for a whole run of contiguous pages. Every page of a large folio
re-enters the pmd/pud/pte walk and re-takes the page table lock.

This series changes follow_page_mask() to return a page count and fill an
array of pages, instead of a single struct page, so a walker can hand back
more than one page per call.

Patches 1 to 4 are preparation, no functional change:

  1: move __get_user_pages()'s open-coded pages[] fill and cache flush into a
     gup_fill_pages() helper, which the rest of the series reuses.
  2: convert the follow_page_mask()/follow_p4d_mask()/follow_pud_mask()/
     follow_pmd_mask()/follow_page_pte() call chain to return a long instead of
     a struct page pointer or ERR_PTR(). Every path still handles one page.
  3: split the "grab a folio reference and store the pages" tail of
     follow_page_pte() into gup_grab_folio_pages(). Storing runs under the
     lock; the cache flushes run after it, via gup_flush_pages().
  4: split the "work out which page this PTE maps" half of follow_page_pte()
     into gup_lookup_pte(), leaving one unlock and one exit.

Patch 5 has the huge page paths store the page and leave the array fill to
follow_pud_mask()/follow_pmd_mask() after they unlock, so the cache flushes
happen outside the pud/pmd critical section.

Patch 6 has follow_huge_pud()/follow_huge_pmd() report the huge page's real
subpage count instead of a separate *page_mask output, and retires *page_mask
and __get_user_pages()'s try_grab_folio() call and subpage loop.

Patch 7 walks every PTE in a page table in one follow_page_pte() call instead
of one per page. The slots are stored under the lock and flushed after it is
dropped, so a whole-table hold does no cache maintenance under the spinlock.

Patch 8 adds gup_pte_batch_len() so a contiguous same-folio run is committed
with one refcount grab. This is the only patch whose benefit depends on folio
size; patch 7 alone covers plain base pages.

Patches 7 and 8 carry their own benchmark tables, both measured against the
base of the series, so the split between the two mechanisms is visible: the
single-call walk is worth 2.3x on base pages and 2.2x on 64 kB mTHP, and
refcount batching adds a further 5.9x on the mTHP case.

Yuan-Hao Hsu ran the v3 series through mm selftests (DEBUG_VM, PROVE_LOCKING),
a GUP-vs-mprotect/MADV_DONTNEED/MADV_COLLAPSE/fork race test, a mixed-mapping
core dump test, W=1/sparse and arm64/i386/ARM builds, all clean, and confirmed
the wins with pin_user_pages(FOLL_LONGTERM): 64 kB mTHP at 256 MB goes from
2318 to 249 us, 4 kB pages from 2361 to 1065 us, 2 MB THP unchanged.

v5:
 - rename the new helpers to gup_lookup_pte(), gup_pte_batch_len(), and
   gup_grab_folio_pages() (David)
 - rebase to 7.4-rc1
 - document that folio_mark_accessed() now runs once per run,
   leaving a freshly faulted large folio referenced but inactive after
   the first call (Yuan-Hao's measurement)
 - note (in patch 4) why the recorded no_page_table() call cannot newly trigger
   on a FOLL_WRITE failure (Suren's question; answered in changelog)
 - add Suren's Reviewed-by tags on patches 1, 2 and 5

Not in this series: a single bio_vec straight through pin_user_pages_fast
for iov_iter_extract_bvecs (Christoph's idea) stays separate work, as David
suggested. The series does not change dirty-marking semantics -- a batch
only gathers the dirty bits its PTEs already had -- so the wider question
of whether GUP should mark folios dirty at all stays where it was
(John Hubbard pointed out the balrog in the room).

Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
Cc: David Hildenbrand <david@xxxxxxxxxx>
Cc: Jason Gunthorpe <jgg@xxxxxxxx>
Cc: John Hubbard <jhubbard@xxxxxxxxxx>
Cc: Peter Xu <peterx@xxxxxxxxxx>
Cc: Suren Baghdasaryan <surenb@xxxxxxxxxx>
Cc: Yuan-Hao Hsu <aa9736195201@xxxxxxxxx>
Cc: Christoph Hellwig <hch@xxxxxxxxxxxxx>
Cc: linux-mm@xxxxxxxxx
Link: https://lore.kernel.org/r/20260811025157.1632867-1-riel@xxxxxxxxxxx/ [RFC v3]
Link: https://lore.kernel.org/r/20260801031540.2742891-1-riel@xxxxxxxxxxx/ [RFC v2]
Link: https://lore.kernel.org/r/20260730035350.1fc95dd8@fangorn/ [RFC]
Suggested-by: David Hildenbrand <david@xxxxxxxxxx>

Rik van Riel (8):
  mm/gup: break out gup_fill_pages() helper
  mm/gup: convert follow_page_mask() to return a long
  mm/gup: split out folio grab-and-store helper
  mm/gup: break out gup_lookup_pte() helper
  mm/gup: fill the pages array outside the pud/pmd lock
  mm/gup: return a huge page's full count from follow_page_mask()
  mm/gup: walk multiple PTEs per follow_page_pte() call
  mm/gup: batch contiguous same-folio PTEs into one refcount grab