[RFC PATCH v2 00/11] add shmem mTHP collapse support

From: Baolin Wang

Date: Wed Jun 10 2026 - 06:40:42 EST


(Note: this patchset is not targeting v7.2, but posted for early feedback.)

This is a follow-up patchset for mTHP collapse to support shmem mTHP collapse,
which is based on Nico's patchset[1].

The shmem mTHP collapse strategy follows the anonymous mTHP collapse approach:
track present pages via a bitmap while scanning PMD ranges for collapse candidates,
then use the bitmap after the scan completes to determine the most efficient
mTHP order to collapse to. Built on the basic framework added for anonymous
mTHP collapse, the shmem mTHP collapse implementation is straightforward
(Thanks for Nico's work).

In addition, I have added some anon/shmem mTHP collapse selftests, and now all
khugepaged test cases can pass.

Note: I have not yet enabled large order collapse for file folios (file folios
currently only support PMD-sized large folio collapse). Although file large order
collapse would be more straightforward to implement after shmem mTHP collapse
support is added (requiring some changes to file_thp_enabled()), I think this
still need some discussion on whether it is necessary to support other large
orders collapse for file folios.

Comments are welcome. Thanks.

Changes from RFC v1:
https://lore.kernel.org/all/cover.1755677674.git.baolin.wang@xxxxxxxxxxxxxxxxx/
- Rebase on the new code, and update to use the new functions.
- Add more test cases.

[1] https://lore.kernel.org/all/20260605161422.213817-1-npache@xxxxxxxxxx/

Baolin Wang (11):
mm: khugepaged: add max_ptes_none check in collapse_file()
mm: khugepaged: generalize collapse_file() for shmem mTHP support
mm: khugepaged: add an order check for PMD-sized THP statistics
mm: khugepaged: add shmem mTHP collapse support
mm: shmem: run khugepaged for all shmem mTHP orders
mm: khugepaged: allow khugepaged to check all shmem mTHP-sized orders
mm: khugepaged: skip large folios that don't need to be collapsed
selftests: mm: extend the check_huge() to support mTHP check
selftests: mm: move gather_after_split_folio_orders() into vm_util.c
file
selftests: mm: implement the mTHP-sized hugepage check helpers
selftests: mm: add mTHP collapse test cases

include/linux/shmem_fs.h | 4 +-
mm/khugepaged.c | 174 ++++++++++++----
mm/shmem.c | 10 +-
.../selftests/mm/folio_split_race_test.c | 2 +-
tools/testing/selftests/mm/khugepaged.c | 195 +++++++++++++-----
.../testing/selftests/mm/prctl_thp_disable.c | 2 +-
tools/testing/selftests/mm/run_vmtests.sh | 4 +
tools/testing/selftests/mm/soft-dirty.c | 2 +-
.../selftests/mm/split_huge_page_test.c | 139 +------------
tools/testing/selftests/mm/uffd-common.c | 4 +-
tools/testing/selftests/mm/vm_util.c | 184 ++++++++++++++++-
tools/testing/selftests/mm/vm_util.h | 8 +-
12 files changed, 492 insertions(+), 236 deletions(-)

--
2.47.3