[PATCH mm-unstable v1 2/3] mm/migrate.c: Prevent folio splitting from interacting with KSM

[Nico Pache]

Since commit b1f202060afe ("mm: remap unused subpages to shared zeropage
when splitting isolated thp"), splitting an anonymous THP remaps all
zero-filled subpages to the shared zeropage via TTU_USE_SHARED_ZEROPAGE.
This flag is set unconditionally for every anonymous folio split,
including splits triggered by KSM.

When KSM is enabled with THP=always, this causes two regressions:

1. use_zero_pages=1: KSM calls try_to_merge_one_page() which triggers
   split_huge_page(). The split remaps all 512 zero-filled subpages to
   the shared zeropage at once, freeing the entire 2MB THP when KSM only
   intended to process a single 4KB page. This bypasses KSM's
   pages_to_scan rate limiting, causing ~1GB to be freed almost
   instantly.

2. use_zero_pages=0: The same split side-effect occurs through the
   stable/unstable tree merge paths. Each pages_to_scan iteration
   triggers an expensive split_huge_page() that silently frees 2MB,
   while the scanner wastes cycles on tree searches for zero-filled
   pages that were already freed as a side-effect.

Fix this by restricting TTU_USE_SHARED_ZEROPAGE being set in the case that
KSM is running and the VMA has VM_MERGEABLE.

Fixes: b1f202060afe ("mm: remap unused subpages to shared zeropage when splitting isolated thp")
Signed-off-by: Nico Pache <npache@xxxxxxxxxx>
---
 mm/migrate.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/mm/migrate.c b/mm/migrate.c
index d9b23909d716..f410f972fc5e 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -304,6 +304,15 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
 	if (PageCompound(page) || PageHWPoison(page))
 		return false;
 
+	/*
+	 * Let KSM handle the zero-filled page deduplication according to its
+	 * own rate limit (pages_to_scan) and policy (use_zero_pages). Without
+	 * this, a KSM-triggered THP split would remap all zero-filled subpages
+	 * to the shared zeropage as a side effect.
+	 */
+	if (ksm_is_running() && (pvmw->vma->vm_flags & VM_MERGEABLE))
+		return false;
+
 	VM_BUG_ON_PAGE(!PageAnon(page), page);
 	VM_BUG_ON_PAGE(!PageLocked(page), page);
 	VM_BUG_ON_PAGE(pte_present(old_pte), page);
-- 
2.54.0