[PATCH v3 0/3] mm: make watermark checks honour ALLOC_NOFRAGMENT

From: Kiryl Shutsemau (Meta)

Date: Mon Nov 16 2026 - 07:51:17 EST


With defrag_mode, 64K mTHP faults under memory pressure stall in
compaction 18 times as often as without it, and get about a third of the
folios. Dev Jain reported it [1].

The high-order watermark check counts free pages of every migratetype.
With defrag_mode, every slowpath allocation is ALLOC_NOFRAGMENT, and a
request below pageblock_order cannot take free pages from another
migratetype's pageblock. Under pressure, that is where the large free
pages end up. Two callers act on the wrong answer:

- direct compaction's up-front check returns COMPACT_SUCCESS without
compacting, so compaction is never deferred;
- direct reclaim for a costly order stops early, having freed little.

Patches 1-2 give the check the request's migratetype and make it skip
the free lists the request cannot take. Patch 3 tells direct reclaim
that the allocation is ALLOC_NOFRAGMENT.

Dev's benchmark in a 7G VM with only 64K mTHP enabled, medians of 15
runs:

defrag_mode=0 defrag_mode=1 defrag_mode=1
+ series
compact_stall 1690 30669 7366
64K faults allocated 20750 7391 20129
64K faults falling back to 4K 725 13832 1339

defrag_mode=1 still compacts more than defrag_mode=0. That is by design,
as it keeps 64K faults out of unmovable and reclaimable pageblocks [2].
With the series, defrag_mode=0 is unchanged at 20698 64K faults
allocated.

Based on page_alloc/for-7.5/misc. Two trivial conflicts with other trees:

- v7.4-rc2 changed the trace_reclaim_retry_zone() call next to patch
2's change in should_reclaim_retry();
- mm-new moved struct scan_control to mm/vmscan.h, where patch 3's
new field goes.

v3:
- Patch 1: declare the watermark functions in mm/page_alloc.h; rewrite
the comment above __zone_watermark_ok() and the changelog
(Vlastimil, Harry)
- Patch 2: plainer comments in should_reclaim_retry() and
__compact_finished()
- Patch 3: pass alloc_flags to try_to_free_pages() (Vlastimil)
- Rebased onto page_alloc/for-7.5/misc

v2:
- Patch 2 changelog: describe what the fast path change does to pages
on the per-cpu lists (Andrew); drop the should_reclaim_retry()
paragraph
- Patch 3 changelog: say that the benchmark baseline already has the
high-order check honouring ALLOC_NOFRAGMENT (Andrew)
- Rebased onto v7.4-rc1

v2: https://lore.kernel.org/all/20261102093939.3340540-1-kas@xxxxxxxxxx/
v1: https://lore.kernel.org/all/20261028132638.2560361-1-kirill@xxxxxxxxxxxxx/

[1] https://lore.kernel.org/all/4aa14b41-9b89-494c-8825-55fffd5a541c@xxxxxxx/
[2] https://lore.kernel.org/all/auCDnqJc7fsBpUW2@thinkstation/

Kiryl Shutsemau (Meta) (3):
mm: page_alloc: add a watermark check that knows the migratetype
mm: page_alloc: make the high-order watermark check honour
ALLOC_NOFRAGMENT
mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT

include/linux/mmzone.h | 3 --
mm/compaction.c | 16 +++++++----
mm/internal.h | 3 +-
mm/page_alloc.c | 62 ++++++++++++++++++++++++++++++++----------
mm/page_alloc.h | 8 ++++++
mm/vmscan.c | 24 ++++++++++++----
6 files changed, 88 insertions(+), 28 deletions(-)

Range-diff against v2:
1: ed3ef9f9fd13 ! 1: e3c2bded052a mm: page_alloc: add a watermark check that knows the migratetype
@@ Metadata
## Commit message ##
mm: page_alloc: add a watermark check that knows the migratetype

- The high-order check in __zone_watermark_ok() accepts a free page of any
- migratetype. Under ALLOC_NOFRAGMENT, a request below pageblock_order can
- only take a page of its own migratetype or claim a whole free block, so
- the check can pass when the allocation cannot succeed.
+ For a high-order request, __zone_watermark_ok() accepts a free page of
+ any migratetype. An ALLOC_NOFRAGMENT request below pageblock_order can
+ take a page of its own migratetype or claim a whole free pageblock, but
+ not steal from another migratetype's pageblock. So the check can pass
+ for an allocation that is going to fail.

- Telling those apart needs the request's migratetype, but not every check
- wants them told apart. A check whether an allocation can be served now
- has to count only the pages it can take.
+ To tell these apart, the check needs the migratetype of the request.
+ Add __zone_watermark_ok_mt(), which takes it, and use it where the
+ caller asks whether this allocation can succeed:

- A check whether there is enough free memory counts every migratetype:
- kswapd and kcompactd balancing a zone, order-0 thresholds, the
- slowpath's forecast whether reclaim and compaction can still make
- progress. Reclaim and compaction work across migratetypes to free whole
- pageblocks.
+ - zone_watermark_fast();
+ - the recheck after node reclaim in get_page_from_freelist();
+ - compaction_suit_allocation_order().

- Move the body of __zone_watermark_ok() into __zone_watermark_ok_mt(),
- which takes the migratetype as well, and make __zone_watermark_ok() a
- wrapper around it for the second kind of check. These use the new
- function:
+ Other callers keep __zone_watermark_ok(), which counts every
+ migratetype. kswapd and kcompactd balance the zone as a whole.
+ should_reclaim_retry() asks whether the allocation could succeed if all
+ reclaimable pages were freed, and compaction can then free a whole
+ pageblock for it to claim.

- - the allocator fast path, zone_watermark_fast();
- - its recheck after node reclaim in get_page_from_freelist();
- - compaction's up-front check, compaction_suit_allocation_order().
+ kcompactd passes MIGRATE_MOVABLE to compaction_suit_allocation_order().
+ It never sets ALLOC_NOFRAGMENT, so the value is not used.

- Declare both in mm/internal.h: nothing outside mm uses them.
+ Nothing outside mm/ uses __zone_watermark_ok(). Move its declaration
+ to mm/page_alloc.h, next to the new one.

- Nothing reads the migratetype yet: no functional change.
+ Nothing reads the migratetype yet, so there is no functional change.
+ Preparation for making the high-order check honour ALLOC_NOFRAGMENT.

+ Reviewed-by: Harry Yoo (Meta) <harry@xxxxxxxxxx>
Tested-by: Dev Jain <dev.jain@xxxxxxx>
Signed-off-by: Kiryl Shutsemau (Meta) <kas@xxxxxxxxxx>

@@ mm/compaction.c: static void kcompactd_do_work(pg_data_t *pgdat)
continue;


- ## mm/internal.h ##
-@@ mm/internal.h: extern int min_free_kbytes;
- extern int defrag_mode;
-
- void setup_per_zone_wmarks(void);
-+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
-+ int highest_zoneidx, unsigned int alloc_flags,
-+ long free_pages);
-+bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
-+ unsigned long mark, int highest_zoneidx,
-+ unsigned int alloc_flags, long free_pages,
-+ int migratetype);
- void calculate_min_free_kbytes(void);
- int __meminit init_per_zone_wmark_min(void);
-
-
## mm/page_alloc.c ##
@@ mm/page_alloc.c: static inline long __zone_watermark_unusable_free(struct zone *z,
* one free page of a suitable size. Checking now avoids taking the zone lock
@@ mm/page_alloc.c: bool __zone_watermark_ok(struct zone *z, unsigned int order, un
}

+/*
-+ * For checks that are not about serving one allocation now: kswapd and
-+ * kcompactd balancing a zone, order-0 thresholds, forecasts whether
-+ * reclaim and compaction can still make progress. They count free pages
-+ * of every migratetype: reclaim and compaction work across migratetypes
-+ * to free whole pageblocks.
++ * Return true if free base pages in the zone are above 'mark' regardless
++ * of migratetype. Use it to check whether the zone as a whole has enough
++ * free memory. Use __zone_watermark_ok_mt() instead to check whether a
++ * particular allocation can be served.
+ */
+bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
+ int highest_zoneidx, unsigned int alloc_flags,
@@ mm/page_alloc.c: get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int
continue;
}

+
+ ## mm/page_alloc.h ##
+@@ mm/page_alloc.h: extern bool free_pages_prepare(struct page *page, unsigned int order);
+
+ extern int user_min_free_kbytes;
+
++bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
++ int highest_zoneidx, unsigned int alloc_flags,
++ long free_pages);
++bool __zone_watermark_ok_mt(struct zone *z, unsigned int order,
++ unsigned long mark, int highest_zoneidx,
++ unsigned int alloc_flags, long free_pages,
++ int migratetype);
++
+ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order, int nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+ #define __alloc_frozen_pages(...) \
2: 3347b0a24b0e ! 2: c991756a139a mm: page_alloc: make the high-order watermark check honour ALLOC_NOFRAGMENT
@@ Commit message
With defrag_mode, every slowpath allocation carries ALLOC_NOFRAGMENT. A
request below pageblock_order may take a page of its own migratetype or
claim a whole free block, but not steal from another migratetype's block.
- The high-order part of the watermark check still accepts a free page of
- any migratetype.
+ For a high-order request, the watermark check still accepts a free page
+ of any migratetype.

Under memory pressure, movable allocations take free movable pages as
soon as reclaim frees them. Free space in unmovable and reclaimable
@@ mm/compaction.c: static enum compact_result __compact_finished(struct compact_co
!free_area_empty(area, MIGRATE_CMA))
return COMPACT_SUCCESS;
#endif
-+ /* Under ALLOC_NOFRAGMENT it may only claim a whole block */
++ /* An ALLOC_NOFRAGMENT request may only claim a whole block */
+ if ((cc->alloc_flags & ALLOC_NOFRAGMENT) &&
+ order < pageblock_order)
+ continue;
@@ mm/page_alloc.c: should_reclaim_retry(gfp_t gfp_mask, unsigned order,
/*
* Would the allocation succeed if we reclaimed all
- * reclaimable pages?
-+ * reclaimable pages? Count free pages of every migratetype:
-+ * reclaim and compaction work across migratetypes to free a
-+ * whole pageblock, which the allocation can then claim.
++ * reclaimable pages? Count free pages of all migratetypes.
++ * Compaction can then free a whole pageblock for the
++ * allocation to claim.
*/
wmark = __zone_watermark_ok(zone, order, min_wmark,
- ac->highest_zoneidx, alloc_flags, available);
3: 0ecbcc5f5fb0 ! 3: fd8bb9835186 mm: vmscan: tell direct reclaim the allocation is ALLOC_NOFRAGMENT
@@ Commit message
64K faults allocated 20750 11538 19759
64K faults falling back to 4K 725 9945 1664

- Pass ALLOC_NOFRAGMENT from the allocation down to try_to_free_pages().
- Both checks then ask __zone_watermark_ok_mt() for a free page the
- allocation can take, and an order-4 direct reclaim frees 40 pages on
- average.
+ Pass the allocation's alloc_flags down to try_to_free_pages(), and
+ record ALLOC_NOFRAGMENT in scan_control. Both checks then ask
+ __zone_watermark_ok_mt() for a free page the allocation can take, and an
+ order-4 direct reclaim frees 40 pages on average.

Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode")
Reported-by: Dev Jain <dev.jain@xxxxxxx>
@@ mm/internal.h: void folio_mark_lazyfree(struct folio *folio);
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *mask);
+ gfp_t gfp_mask, const nodemask_t *mask,
-+ bool nofragment);
++ unsigned int alloc_flags);
unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
int zone_idx);

@@ mm/page_alloc.c: __perform_reclaim(gfp_t gfp_mask, unsigned int order,

progress = try_to_free_pages(ac->zonelist, order, gfp_mask,
- ac->nodemask);
-+ ac->nodemask,
-+ alloc_flags & ALLOC_NOFRAGMENT);
++ ac->nodemask, alloc_flags);

memalloc_noreclaim_restore(noreclaim_flag);
fs_reclaim_release(gfp_mask);
@@ mm/vmscan.c: struct scan_control {
/* One of the zones is ready for compaction */
unsigned int compaction_ready:1;

-+ /* The allocation cannot fall back to other migratetypes' blocks */
++ /* The allocation may not steal from other migratetypes' blocks */
+ unsigned int nofragment:1;
+
/* There is easily reclaimable cold cache in the current node */
@@ mm/vmscan.c: static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
- gfp_t gfp_mask, const nodemask_t *nodemask)
+ gfp_t gfp_mask, const nodemask_t *nodemask,
-+ bool nofragment)
++ unsigned int alloc_flags)
{
unsigned long nr_reclaimed;
struct scan_control sc = {
@@ mm/vmscan.c: unsigned long try_to_free_pages(struct zonelist *zonelist, int orde
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
-+ .nofragment = nofragment,
++ .nofragment = !!(alloc_flags & ALLOC_NOFRAGMENT),
};

/*

base-commit: 3a99594e62fce8c646e8aac01d38af16d6833a40
--
2.54.0