[RFC PATCH 31/45] mm: page_alloc: cross-non-movable buddy borrow within tainted SPBs

[Rik van Riel]

From: Rik van Riel <riel@xxxxxxxx>

When pages get freed via __free_one_page, they're placed on the per-SPB
free_list determined by their pageblock's migratetype, not the original
allocation's migratetype. Slab-heavy workloads expose a structural
mismatch:

  - RECLAIMABLE pageblocks fill up densely with live slab objects (e.g.
    btrfs_inode caches), leaving very few sub-pageblock free fragments
    on the RECL free list.

  - UNMOVABLE pageblocks accumulate sparse free space from vmalloc and
    raw-alloc churn — tens of thousands of free pages, all on the UNMOV
    free list.

Net effect: a tainted SPB can show 87,000+ free pages in metadata while
having ZERO free buddies on the RECL list. A new RECL allocation walking
__rmqueue_smallest's preferred-SB Pass 1 finds nothing, falls through
Pass 2 (claim_whole_block on MOVABLE — but mov=0 in tainted SBs),
Pass 2b (sub-PB MOVABLE — same), and reaches Pass 3, which taints a
fresh clean SPB. Repeat per RECL burst.

Add a Pass 2c between 2b and 3: for non-movable allocations that
couldn't find their own migratetype, try borrowing a sub-pageblock buddy
from the *opposite* non-movable migratetype's free list within tainted
SPBs. UNMOV alloc → check RECL free list; RECL alloc → check UNMOV
free list. The pageblock tag is NOT changed — page_del_and_expand uses
the source migratetype for both delete and re-list, so the splits stay
on the source list, and when our borrowed page is later freed
__free_one_page returns it to the source list (based on pageblock tag).
The "borrow" is purely transient: physical page goes to a foreign-type
caller, returns to its native list on free.

PB_has_<requested_type> is set via __spb_set_has_type so spb_defrag
accounting reflects that the pageblock now hosts our type's content.
PB_has_<source_type> stays set since other buddies of that type remain.

Restricted to UNMOV ↔ RECL within SB_TAINTED — movable allocations have
their own Pass 4 fallback, and clean SPBs must not be polluted with
cross-type mixing (that's what the existing migratetype-isolation
machinery exists to prevent).

Live measurement on a 247 GB devvm with btrfs root, kernel 397 (Stage 1
+ simplified Stage 2a) at boot+7min: 12 tainted Normal-zone SPBs grew
from 4 baseline despite the existing 11 having between 825 and 87,062
free pages each, ALL on the UNMOV list while the workload kept
allocating RECL btrfs_inode slab pages. Pass 2c lets those allocs
absorb into the existing UNMOV-listed free pool rather than creating
fresh tainted SPBs.

Signed-off-by: Rik van Riel <riel@xxxxxxxxxxx>
Assisted-by: Claude:claude-opus-4.7 syzkaller
---
 mm/page_alloc.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 85 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index a72cb2da606d..f2db3dd86a84 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2806,6 +2806,7 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
 	struct page *page;
 	int full;
 	struct superpageblock *sb;
+	int opposite_mt;
 	/*
 	 * Category search order: 2 passes.
 	 * Movable: clean first, then tainted (pack into clean SBs).
@@ -2985,6 +2986,90 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
 				}
 			}
 		}
+
+		/*
+		 * Pass 2c: cross-non-movable borrow within tainted SPBs.
+		 *
+		 * If we're a non-movable alloc and Pass 1/2/2b couldn't find a
+		 * buddy on our migratetype's free list anywhere, but tainted
+		 * SPBs have free buddies on the *opposite* non-movable type's
+		 * free list, take one of those.
+		 *
+		 * Why this happens: when pages are freed, __free_one_page puts
+		 * them on the free_list determined by their pageblock's tag,
+		 * not the original allocation's migratetype. Slab caches tend
+		 * to be dense (RECL pageblocks fill up; few sub-PB fragments),
+		 * while UNMOV pageblocks accumulate sparse free space from
+		 * vmalloc/raw alloc churn. Net effect: tainted SPBs frequently
+		 * have tens of thousands of free pages all on the UNMOV list,
+		 * invisible to RECL allocs (or vice versa). Without this pass,
+		 * the alloc falls through to Pass 3 and taints a fresh clean
+		 * SPB even though the existing tainted ones have plenty of
+		 * unused space.
+		 *
+		 * We do NOT relabel the source pageblock. The buddy is taken
+		 * from @opposite_mt's free list and the splits go back on
+		 * @opposite_mt's list (page_del_and_expand uses the same mt
+		 * for delete and expand). The pageblock tag is unchanged, so
+		 * the page returns to @opposite_mt's list when freed via
+		 * __free_one_page. Effectively a borrow: the alloc takes a
+		 * physical page from a UNMOV-tagged pageblock for a RECL
+		 * use, and the page cycles back to UNMOV's list on free.
+		 *
+		 * We do set PB_has_<migratetype> via __spb_set_has_type so
+		 * spb_defrag accounting reflects that this pageblock now hosts
+		 * our migratetype's content too. PB_has_<opposite_mt> stays
+		 * set since other buddies of that type remain.
+		 *
+		 * Restricted to UNMOV ↔ RECL. Movable allocations don't
+		 * participate (they have their own Pass 4 fallback path).
+		 *
+		 * Restricted to SB_TAINTED to avoid spreading mixing into
+		 * clean SPBs.
+		 */
+		opposite_mt = -1;
+		if (migratetype == MIGRATE_UNMOVABLE)
+			opposite_mt = MIGRATE_RECLAIMABLE;
+		else if (migratetype == MIGRATE_RECLAIMABLE)
+			opposite_mt = MIGRATE_UNMOVABLE;
+
+		if (opposite_mt >= 0) {
+			for (full = SB_FULL; full < __NR_SB_FULLNESS; full++) {
+				list_for_each_entry(sb,
+					&zone->spb_lists[SB_TAINTED][full], list) {
+					int co;
+
+					if (!sb->nr_free_pages)
+						continue;
+					for (co = min_t(int, pageblock_order - 1,
+							NR_PAGE_ORDERS - 1);
+					     co >= (int)order;
+					     --co) {
+						current_order = co;
+						area = &sb->free_area[current_order];
+						page = get_page_from_free_area(
+							area, opposite_mt);
+						if (!page)
+							continue;
+						if (get_pageblock_isolate(page))
+							continue;
+						if (is_migrate_cma(
+						    get_pageblock_migratetype(page)))
+							continue;
+						page_del_and_expand(zone, page,
+							order, current_order,
+							opposite_mt);
+						__spb_set_has_type(page,
+							migratetype);
+						trace_mm_page_alloc_zone_locked(
+							page, order, migratetype,
+							pcp_allowed_order(order) &&
+							migratetype < MIGRATE_PCPTYPES);
+						return page;
+					}
+				}
+			}
+		}
 	}
 
 	/*
-- 
2.52.0