Re: [PATCH] mm/gup: batch PTE-mapped large folios in gup_fast_pte_range()

[David Hildenbrand (Arm)]

On 9/17/26 10:42, David Hildenbrand (Arm) wrote:
> On 9/17/26 10:37, Yuan-Hao Hsu wrote:
>> GUP-fast grabs a PTE-mapped large folio one page at a time.  Every PTE
>> costs a try_grab_folio_fast() (a refcount cmpxchg, plus the pincount
>> atomic, a full barrier and a node stat update for FOLL_PIN), a
>> gup_fast_folio_allowed() and a folio_set_referenced(), so a 64 kB mTHP
>> pays sixteen of each and a PTE-mapped 2 MB THP pays 512.  The PMD and
>> PUD leaf paths already take the whole range with one
>> try_grab_folio_fast() call, and the slow path is getting the same
>> treatment for PTEs in Rik's follow_page_mask() series.
>>
>> The users are the hot ones: iov_iter_extract_user_pages() pins every
>> O_DIRECT buffer through pin_user_pages_fast(), io_uring registers its
>> buffers through it, and so does RDMA memory registration.  With mTHP
>> those buffers are PTE-mapped large folios.
>>
>> After the head page has been grabbed and verified as before, extend the
>> run to the following PTEs that map the next pages of the same folio
>> with the same protection bits, take the remaining references for the
>> run with try_grab_folio(), and fill pages[] from the run.  The PTEs of
>> the run are read while the reference on the folio is held, so the folio
>> can neither be freed nor split underneath the scan; a THP collapse can
>> still detach the page table, so the pmd is checked again after the run
>> is grabbed and the run is dropped if it changed, leaving the next
>> iteration to hit the existing pmd check and bail out.  A read-only run
>> of an anonymous folio stops at the first page that gup_must_unshare()
>> rejects, so the per-page exclusivity guarantee for FOLL_PIN is kept.
>>
>> folio_pte_batch_flags() does the scan.  It reads the entries with
>> ptep_get(), which is not safe without the page table lock on
>> CONFIG_GUP_GET_PXX_LOW_HIGH (x86 PAE, mips32, sh) or with arm64 contpte,
>> and GUP-fast reads every PTE with ptep_get_lockless() for that reason.
>> Add FPB_LOCKLESS to make the helper do the same; it is inlined per call
>> site, so the other callers do not change.
>>
>> The order-0 path keeps its instruction sequence.  The folio flags are
>> tested before folio_set_referenced() on purpose: a load of the flags
>> right after that locked instruction cost about 25% on order-0 pages in
>> the gup_test benchmark below, and reading them first brings that path
>> back to the baseline.
>>
>> gup_test ioctl over a 256 MB anonymous region, FOLL_WRITE, 65536 pages
>> per call, median of 15 runs, two baseline boots / one patched boot, on
>> an x86-64 VM:
>>
>>                        get_user_pages_fast()   pin_user_pages_fast()
>>                        before       after      before       after
>>   4 kB pages           849/789 us   783 us     1174/1138 us 1094 us
>>   64 kB mTHP           812/811 us   126 us     1215/1113 us  188 us
>>   1 MB mTHP            799/774 us    59 us     1185/1077 us   64 us
>>   2 MB THP, PTE-mapped 803/786 us    57 us     1159/1120 us   58 us
>>   2 MB THP, PMD-mapped  42/40 us     40 us       46/41 us     42 us
>>
>> fio O_DIRECT randread of a null_blk device (CPU bound) with the I/O
>> buffer in 64 kB mTHP, median of 15 five-second runs:
>>
>>   bs=1M psync                 126.2/125.6 GB/s  ->  220.6 GB/s
>>   bs=1M io_uring, iodepth 16   71.2/71.6 GB/s   ->  112.1 GB/s
>>   bs=64k io_uring, iodepth 16  50.2/50.7 GB/s   ->   60.1 GB/s
>>
>> IORING_REGISTER_BUFFERS of 1 GB, median of 15:
>>
>>   64 kB mTHP           6179/5998 us  ->  1599 us
>>   1 MB mTHP            5790/5801 us  ->   893 us
>>   4 kB pages           7173/6741 us  ->  6763 us
>>
> 
> I think Rik was already working on this and sent some patches. Anyhow, there is
> quite some GUP review backlog I have t go through, so this will have to wait.
> 

Ah, no, Rik looked at GUP not GUP-fast. So I'll have a look but it will take a
while.

-- 
Cheers,

David