Re: [PATCH RFC 00/39] mm/rmap: interface overhaul

[Ryan Roberts]

On 04/12/2023 19:53, Ryan Roberts wrote:
> On 04/12/2023 14:21, David Hildenbrand wrote:
>> Baed on mm-stable from a couple of days.
>>
>> This series proposes an overhaul to our rmap interface, to get rid of the
>> "bool compound" / RMAP_COMPOUND parameter with the goal of making the
>> interface less error prone, more future proof, and more natural to extend
>> to "batching". Also, this converts the interface to always consume
>> folio+subpage, which speeds up operations on large folios.
>>
>> Further, this series adds PTE-batching variants for 4 rmap functions,
>> whereby only folio_add_anon_rmap_ptes() is used for batching in this series
>> when PTE-remapping a PMD-mapped THP.
> 
> I certainly support the objective you have here; making the interfaces clearer,
> more consistent and more amenable to batching. I'll try to find some time this
> week to review.
> 
>>
>> Ryan has series where we would make use of folio_remove_rmap_ptes() [1]
>> -- he carries his own batching variant right now -- and
>> folio_try_dup_anon_rmap_ptes()/folio_dup_file_rmap_ptes() [2].
> 
> Note that the contpte series at [2] has a new patch in v3 (patch 2), which could
> benefit from folio_remove_rmap_ptes() or equivalent. My plan was to revive [1]
> on top of [2] once it is merged.
> 
>>
>> There is some overlap with both series (and some other work, like
>> multi-size THP [3]), so that will need some coordination, and likely a
>> stepwise inclusion.
> 
> Selfishly, I'd really like to get my stuff merged as soon as there is no
> technical reason not to. I'd prefer not to add this as a dependency if we can
> help it.
> 
>>
>> I got that started [4], but it made sense to show the whole picture. The
>> patches of [4] are contained in here, with one additional patch added
>> ("mm/rmap: introduce and use hugetlb_try_share_anon_rmap()") and some
>> slight patch description changes.
>>
>> In general, RMAP batching is an important optimization for PTE-mapped
>> THP, especially once we want to move towards a total mapcount or further,
>> as shown with my WIP patches on "mapped shared vs. mapped exclusively" [5].
>>
>> The rmap batching part of [5] is also contained here in a slightly reworked
>> fork [and I found a bug du to the "compound" parameter handling in these
>> patches that should be fixed here :) ].
>>
>> This series performs a lot of folio conversion, that could be separated
>> if there is a good reason. Most of the added LOC in the diff are only due
>> to documentation.
>>
>> As we're moving to a pte/pmd interface where we clearly express the
>> mapping granularity we are dealing with, we first get the remainder of
>> hugetlb out of the way, as it is special and expected to remain special: it
>> treats everything as a "single logical PTE" and only currently allows
>> entire mappings.
>>
>> Even if we'd ever support partial mappings, I strongly
>> assume the interface and implementation will still differ heavily:
>> hopefull we can avoid working on subpages/subpage mapcounts completely and
>> only add a "count" parameter for them to enable batching.
>>
>>
>> New (extended) hugetlb interface that operate on entire folio:
>>  * hugetlb_add_new_anon_rmap() -> Already existed
>>  * hugetlb_add_anon_rmap() -> Already existed
>>  * hugetlb_try_dup_anon_rmap()
>>  * hugetlb_try_share_anon_rmap()
>>  * hugetlb_add_file_rmap()
>>  * hugetlb_remove_rmap()
>>
>> New "ordinary" interface for small folios / THP::
>>  * folio_add_new_anon_rmap() -> Already existed
>>  * folio_add_anon_rmap_[pte|ptes|pmd]()
>>  * folio_try_dup_anon_rmap_[pte|ptes|pmd]()
>>  * folio_try_share_anon_rmap_[pte|pmd]()
>>  * folio_add_file_rmap_[pte|ptes|pmd]()
>>  * folio_dup_file_rmap_[pte|ptes|pmd]()
>>  * folio_remove_rmap_[pte|ptes|pmd]()
> 
> I'm not sure if there are official guidelines, but personally if we are
> reworking the API, I'd take the opportunity to move "rmap" to the front of the
> name, rather than having it burried in the middle as it is for some of these:
> 
> rmap_hugetlb_*()
> 
> rmap_folio_*()

In fact, I'd be inclined to drop the "folio" to shorten the name; everything is
a folio, so its not telling us much. e.g.:

New (extended) hugetlb interface that operate on entire folio:
 * rmap_hugetlb_add_new_anon() -> Already existed
 * rmap_hugetlb_add_anon() -> Already existed
 * rmap_hugetlb_try_dup_anon()
 * rmap_hugetlb_try_share_anon()
 * rmap_hugetlb_add_file()
 * rmap_hugetlb_remove()

New "ordinary" interface for small folios / THP::
 * rmap_add_new_anon() -> Already existed
 * rmap_add_anon_[pte|ptes|pmd]()
 * rmap_try_dup_anon_[pte|ptes|pmd]()
 * rmap_try_share_anon_[pte|pmd]()
 * rmap_add_file_[pte|ptes|pmd]()
 * rmap_dup_file_[pte|ptes|pmd]()
 * rmap_remove_[pte|ptes|pmd]()


> 
> I guess reading the patches will tell me, but what's the point of "ptes"? Surely
> you're either mapping at pte or pmd level, and the number of pages is determined
> by the folio size? (or presumably nr param passed in)
> 
> Thanks,
> Ryan
> 
>>
>> folio_add_new_anon_rmap() will always map at the biggest granularity
>> possible (currently, a single PMD to cover a PMD-sized THP). Could be
>> extended if ever required.
>>
>> In the future, we might want "_pud" variants and eventually "_pmds" variants
>> for batching. Further, if hugepd is ever a thing outside hugetlb code,
>> we might want some variants for that. All stuff for the distant future.
>>
>>
>> I ran some simple microbenchmarks from [5] on an Intel(R) Xeon(R) Silver
>> 4210R: munmap(), fork(), cow, MADV_DONTNEED on each PTE ... and PTE
>> remapping PMD-mapped THPs on 1 GiB of memory.
>>
>> For small folios, there is barely a change (< 1 % performance improvement),
>> whereby fork() still stands out with 0.74% performance improvement, but
>> it might be just some noise. Folio optimizations don't help that much
>> with small folios.
>>
>> For PTE-mapped THP:
>> * PTE-remapping a PMD-mapped THP is more than 10% faster.
>>   -> RMAP batching
>> * fork() is more than 4% faster.
>>   -> folio conversion
>> * MADV_DONTNEED is 2% faster
>>   -> folio conversion
>> * COW by writing only a single byte on a COW-shared PTE
>>   -> folio conversion
>> * munmap() is only slightly faster (< 1%).
>>
>> [1] https://lkml.kernel.org/r/20230810103332.3062143-1-ryan.roberts@xxxxxxx
>> [2] https://lkml.kernel.org/r/20231204105440.61448-1-ryan.roberts@xxxxxxx
>> [3] https://lkml.kernel.org/r/20231204102027.57185-1-ryan.roberts@xxxxxxx
>> [4] https://lkml.kernel.org/r/20231128145205.215026-1-david@xxxxxxxxxx
>> [5] https://lkml.kernel.org/r/20231124132626.235350-1-david@xxxxxxxxxx
>>
>> Cc: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>> Cc: "Matthew Wilcox (Oracle)" <willy@xxxxxxxxxxxxx>
>> Cc: Hugh Dickins <hughd@xxxxxxxxxx>
>> Cc: Ryan Roberts <ryan.roberts@xxxxxxx>
>> Cc: Yin Fengwei <fengwei.yin@xxxxxxxxx>
>> Cc: Mike Kravetz <mike.kravetz@xxxxxxxxxx>
>> Cc: Muchun Song <muchun.song@xxxxxxxxx>
>> Cc: Peter Xu <peterx@xxxxxxxxxx>
>>
>> David Hildenbrand (39):
>>   mm/rmap: rename hugepage_add* to hugetlb_add*
>>   mm/rmap: introduce and use hugetlb_remove_rmap()
>>   mm/rmap: introduce and use hugetlb_add_file_rmap()
>>   mm/rmap: introduce and use hugetlb_try_dup_anon_rmap()
>>   mm/rmap: introduce and use hugetlb_try_share_anon_rmap()
>>   mm/rmap: add hugetlb sanity checks
>>   mm/rmap: convert folio_add_file_rmap_range() into
>>     folio_add_file_rmap_[pte|ptes|pmd]()
>>   mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]()
>>   mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd()
>>   mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte()
>>   mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte()
>>   mm/rmap: remove page_add_file_rmap()
>>   mm/rmap: factor out adding folio mappings into __folio_add_rmap()
>>   mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]()
>>   mm/huge_memory: batch rmap operations in __split_huge_pmd_locked()
>>   mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd()
>>   mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
>>   mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
>>   mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
>>   mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte()
>>   mm/rmap: remove page_add_anon_rmap()
>>   mm/rmap: remove RMAP_COMPOUND
>>   mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]()
>>   kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte()
>>   mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd()
>>   mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte()
>>   mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte()
>>   mm/memory: page_remove_rmap() -> folio_remove_rmap_pte()
>>   mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte()
>>   mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte()
>>   Documentation: stop referring to page_remove_rmap()
>>   mm/rmap: remove page_remove_rmap()
>>   mm/rmap: convert page_dup_file_rmap() to
>>     folio_dup_file_rmap_[pte|ptes|pmd]()
>>   mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]()
>>   mm/huge_memory: page_try_dup_anon_rmap() ->
>>     folio_try_dup_anon_rmap_pmd()
>>   mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte()
>>   mm/rmap: remove page_try_dup_anon_rmap()
>>   mm: convert page_try_share_anon_rmap() to
>>     folio_try_share_anon_rmap_[pte|pmd]()
>>   mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED
>>
>>  Documentation/mm/transhuge.rst       |   4 +-
>>  Documentation/mm/unevictable-lru.rst |   4 +-
>>  include/linux/mm.h                   |   6 +-
>>  include/linux/rmap.h                 | 380 +++++++++++++++++++-----
>>  kernel/events/uprobes.c              |   2 +-
>>  mm/gup.c                             |   2 +-
>>  mm/huge_memory.c                     |  85 +++---
>>  mm/hugetlb.c                         |  21 +-
>>  mm/internal.h                        |  12 +-
>>  mm/khugepaged.c                      |  17 +-
>>  mm/ksm.c                             |  15 +-
>>  mm/memory-failure.c                  |   4 +-
>>  mm/memory.c                          |  60 ++--
>>  mm/migrate.c                         |  12 +-
>>  mm/migrate_device.c                  |  41 +--
>>  mm/mmu_gather.c                      |   2 +-
>>  mm/rmap.c                            | 422 ++++++++++++++++-----------
>>  mm/swapfile.c                        |   2 +-
>>  mm/userfaultfd.c                     |   2 +-
>>  19 files changed, 709 insertions(+), 384 deletions(-)
>>
>