[PATCH 0/5] mm/gup: batch contiguous pages in follow_page_mask()

[Rik van Riel]

follow_page_mask() walks the page tables one page at a time, even when
the caller asked for a whole run of contiguous pages. Every page of a
large folio re-enters the pmd/pud/pte walk and re-takes the page table
lock.

This series changes follow_page_mask() to return a page count and a
new @end argument bounding how many pages remain, instead of a single
struct page, so a walker can hand back more than one page per call.

Patch 1 converts the follow_page_mask()/follow_p4d_mask()/
follow_pud_mask()/follow_pmd_mask()/follow_page_pte() call chain to
return a long instead of a struct page pointer or ERR_PTR(), with no
functional change: every path still handles exactly one page.

Patch 2 is pure code motion, splitting the "commit to a resolved page"
tail of follow_page_pte() into its own follow_page_pte_commit(), no
functional change.

Patch 3 adds gup_fill_pages(), a small helper that fills pages[] and
flushes caches for a run of subpages, and converts the three existing
per-page call sites to use it with nr == 1, no functional change.

Patch 4 has follow_huge_pud()/follow_huge_pmd() report the huge page's
real subpage count instead of a separate *page_mask output, and
retires *page_mask and __get_user_pages()'s dead second
try_grab_folio() call and subpage loop.

It also defers gup_fill_pages() past the pud/pmd unlock, so a 1 GB
PUD-mapped folio doesn't hold that lock for a full array fill and
cache flush.

Patch 5 adds follow_pte_batch() and has follow_page_pte() call it once
per contiguous same-folio run instead of once per page, so a
PTE-mapped mTHP no longer restarts the walk and re-takes the PTE lock
per subpage.

This is the only patch that changes the number of page table walks or
lock acquisitions.

Benchmarked with mm/gup_test.c (PIN_LONGTERM_BENCHMARK, pin_user_pages
+ FOLL_LONGTERM, 256 MB region, median of 16 runs, folio formation
verified via the per-size anon_fault_alloc counter):

                       before      after
  4 kB base pages      2721 us     1198 us   (2.3x)
  64 kB mTHP           2929 us      201 us  (14.6x)
  2 MB THP               73 us       69 us   (flat)

The 4 kB result comes entirely from patch 5 merging two separate
try_grab_folio() calls and lock acquisitions into one; folio size and
PTE batching play no part in it.

64 kB mTHP adds the walk-restart avoidance on top. 2 MB THP is
unaffected, since follow_huge_pmd() already handled it in one call.

Patch 4's lock-hold-time change is a scalability argument, not a
measured one -- it is not visible in this single-threaded benchmark.

Suggested-by: David Hildenbrand <david@xxxxxxxxxx>

Rik van Riel (5):
  mm/gup: convert follow_page_mask() to return a long
  mm/gup: split follow_page_pte_commit() out of follow_page_pte()
  mm/gup: add gup_fill_pages() and use it
  mm/gup: return a huge page's full count from follow_page_mask()
  mm/gup: walk multiple PTEs per follow_page_pte() call

 mm/gup.c | 532 +++++++++++++++++++++++++++++++++----------------------
 1 file changed, 322 insertions(+), 210 deletions(-)

base-commit: fc02acf6ac0c
-- 
2.53.0-Meta