[RFC PATCH] mm: preserve physical order in swap fallback splitting

[Chengfeng Lin]

When swap allocation falls back to splitting an anonymous PMD-sized
folio, the new children are appended in ascending physical order.
shrink_folio_list() consumes its list from the tail, so it processes
those children in descending order after the original head. On the
tested NVMe/IOMMU path, this prevents much of the DMA mapping coalescing.

Keep this ordering change local to the swap-allocation fallback. Collect
the new children on a temporary list and reverse their insertion into
the reclaim list after a successful split. Leave the original head and
existing reclaim entries in place. On failure, append any children in
their original order and return the original error. Other split callers
are unchanged.

Assisted-by: LLM
Signed-off-by: Chengfeng Lin <lin2530632123@xxxxxxxxx>
---
Hi,

This prototype reduced PAGEOUT plus cached-refault worker CPU time by 14.84%
with one worker and 13.14% with two workers in a synthetic 8 MiB test
starting from anonymous THPs. Is this reclaim fallback a reasonable place
to preserve physical page order, or would you prefer addressing it in
the split or block/DMA mapping code?

The patch and both measured kernels use mm-stable 2967f3ccdb5f
(7.3-rc5). The machine is an Intel Core i7-12700KF with 32 GiB RAM,
an ext4 swap file on NVMe, and translated IOMMU DMA mappings. The test
uses full preemption, the performance governor, turbo off and zswap off.
Workers run on CPU 2, or CPUs 2 and 4; the total size stays at 8 MiB.

Each cycle fills and collapses the memory, calls MADV_PAGEOUT, waits
20 ms and checks swap-cache readiness, then touches one byte per base page.
Each invocation repeats this four times. The primary metric adds the
workers' CPU time in PAGEOUT and these cached refaults, then averages it
over the four 8 MiB cycles. Setup, waiting, full data checks and cleanup
are outside this metric. It measures CPU cost, not concurrent throughput
or storage-read latency. The extra traversal of the split children runs
inside PAGEOUT, so its CPU cost is included.

                       Baseline A   Prototype   Baseline B
  Initial THP, 1 worker   3804.69 us   3243.69 us   3812.94 us
  Initial THP, 2 workers  3978.96 us   3445.82 us   3955.02 us

Each point has two warmups and nine measured invocations per condition.
Relative to the two baseline points combined, the savings are 565.13 us
and 521.17 us per 8 MiB. Baseline return drift is +0.22% and -0.60%.
The two base-page controls showed no clear change. All 108 measured
invocations, including controls, are retained. This is one A/B/A boot
sequence, not evidence of the same gain in an application.

Separate non-timing checks confirmed the changed page order. Both
kernels submitted 65 BIOs per 8 MiB. Within the observed PAGEOUT caller,
the same 8,384,512 bytes required 2,047 IOVA links on the baseline and
64 with the prototype; these counts exclude work outside that caller.
The checks covered data, page state and cleanup on both kernels.
Failure/partial-output handling was source-reviewed and checked with a
local list model, not exercised by kernel fault injection.

The reproducer and patch are attached. Configurations, measurements and
review notes are available at [1]. The numbers above are from this
MM-tree pair, not reused from older kernels.
Codex assisted with the workload, prototype, tests, analysis and message.

[1] https://github.com/lcf0399/linux-regression-evidence/tree/a233ed8edb1e91a6a0ea0d326668bb1f52384489/mm/mm-swap-fallback-folio-order

--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1153,6 +1153,37 @@
                        SWAP_OPS_F_REQUIRE_NOFS))
                return true;
        return false;
+}
+
+/*
+ * Keep this ordering local to the swap-allocation fallback.  Reclaim takes
+ * folios from the tail, so reverse only the newly split children to process
+ * physically adjacent pages in ascending order after the original head.
+ */
+static int split_folio_for_swap(struct folio *folio, struct list_head *list)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+       if (folio_test_anon(folio) && !folio_test_swapcache(folio) &&
+           folio_order(folio) == HPAGE_PMD_ORDER) {
+               LIST_HEAD(split_folios);
+               struct folio *child;
+               int ret;
+
+               ret = split_folio_to_list(folio, &split_folios);
+               if (ret) {
+                       /* A failed split may still have produced some
children. */
+                       list_splice_tail_init(&split_folios, list);
+                       return ret;
+               }
+
+               while (!list_empty(&split_folios)) {
+                       child = list_last_entry(&split_folios, struct
folio, lru);
+                       list_move_tail(&child->lru, list);
+               }
+               return 0;
+       }
+#endif
+       return split_folio_to_list(folio, list);
 }

 /*
@@ -1391,7 +1422,7 @@
                                if (ret != -E2BIG)
                                        goto activate_locked;
                                /* Fallback to swap normal pages */
-                               if (split_folio_to_list(folio, folio_list))
+                               if (split_folio_for_swap(folio, folio_list))
                                        goto activate_locked;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
                                if (nr_pages >= HPAGE_PMD_NR) {

Attachment: mm-swap-fallback-folio-order-reproducer.tar.gz
Description: GNU Zip compressed data

--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1153,6 +1153,37 @@
 			SWAP_OPS_F_REQUIRE_NOFS))
 		return true;
 	return false;
+}
+
+/*
+ * Keep this ordering local to the swap-allocation fallback.  Reclaim takes
+ * folios from the tail, so reverse only the newly split children to process
+ * physically adjacent pages in ascending order after the original head.
+ */
+static int split_folio_for_swap(struct folio *folio, struct list_head *list)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	if (folio_test_anon(folio) && !folio_test_swapcache(folio) &&
+	    folio_order(folio) == HPAGE_PMD_ORDER) {
+		LIST_HEAD(split_folios);
+		struct folio *child;
+		int ret;
+
+		ret = split_folio_to_list(folio, &split_folios);
+		if (ret) {
+			/* A failed split may still have produced some children. */
+			list_splice_tail_init(&split_folios, list);
+			return ret;
+		}
+
+		while (!list_empty(&split_folios)) {
+			child = list_last_entry(&split_folios, struct folio, lru);
+			list_move_tail(&child->lru, list);
+		}
+		return 0;
+	}
+#endif
+	return split_folio_to_list(folio, list);
 }
 
 /*
@@ -1391,7 +1422,7 @@
 				if (ret != -E2BIG)
 					goto activate_locked;
 				/* Fallback to swap normal pages */
-				if (split_folio_to_list(folio, folio_list))
+				if (split_folio_for_swap(folio, folio_list))
 					goto activate_locked;
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 				if (nr_pages >= HPAGE_PMD_NR) {