[RFC PATCH 0/2] mm/huge_memory: find partially mapped shared folios with NO_PAGE_MAPCOUNT

[Jose A. Perez de Azpillaga]

With CONFIG_NO_PAGE_MAPCOUNT, an anon large folio mapped by several MMs
isn't seen as partially mapped while its mapcount stays at or above the
number of pages. If parent and child zap the same half of a THP after
fork(), that half is mapped by nobody, the folio is never queued for
deferred split, and the memory stays allocated until every mapper is
gone.

Patch 1 moves the split_underused_thp check out of
deferred_split_folio(). Patch 2 queues such folios as candidates and has
the shrinker walk the rmap to see which pages are still mapped.

512 MB of THPs with the same half zapped by every mapper, memory freed
by the shrinker:

                                before    after
  2M,  2 processes                0 MB   252 MB
  2M,  3 processes                0 MB   237 MB
  2M,  2 children + parent PMD    0 MB   244 MB
  64K, 2 processes                0 MB   252 MB
  64K, 3 processes                0 MB   252 MB

With patch 2 the shrinker reads shrink_underused itself, so folios
queued for the underused check aren't split anymore once it's turned
off.

David suggested a check from the shrinker for the opposite case in [1],
folios marked partially mapped by a PTE by PTE unmap. The walk in patch 2
could clear the flag there too, this series doesn't do it.

Questions:

- Is an rmap walk from the shrinker fine for this? Every CoW of a page
  in a shared PTE-mapped folio queues a candidate, and so does the last
  PMD mapper when it splits its PMD.

- The walk needs one bit per page, which is too much for the stack with
  64K pages, so I used a static bitmap under a mutex. That serializes
  the walks of all nodes and memcgs, and the mutex is taken with the
  folio locked. Where should the bitmap live?

No selftest yet, I'll turn the reproducer into one if the approach is
ok.

Based on mm-new.

Tested on x86_64 in QEMU, so no numbers from real hardware or with
concurrent reclaim.

[1] https://lore.kernel.org/all/20261109220958.2071687-1-joannelkoong@xxxxxxxxx/

Jose A. Perez de Azpillaga (2):
  mm/huge_memory: check split_underused_thp in the caller of
    deferred_split_folio()
  mm/huge_memory: detect partially mapped shared folios without per-page
    mapcounts

 mm/huge_memory.c | 86 ++++++++++++++++++++++++++++++++++++++++++++----
 mm/rmap.c        | 21 ++++++++++--
 2 files changed, 98 insertions(+), 9 deletions(-)

-- 
2.56.0