[PATCH] mm/mglru: skip oom killing for high order allocs in kswapd

[Bingfang Guo via B4 Relay]

From: Bingfang Guo <bingfangguo@xxxxxxxxxxx>

We observed that when MGLRU min_ttl is set and the system is highly
fragmented, high order allocations (e.g. order == 9) will fail and wake
up kswapd. It can not make any progress, while pushing generations of
memcgs so that they all end up younger than min_ttl_ms, repeatedly
triggering the OOM killer in lru_gen_age_node(). Then the whole process
repeats until all processes are killed and the kernel panics.

Fix this by skipping OOM killer invocations in lru_gen_age_node() for
sc->order > PAGE_ALLOC_COSTLY_ORDER, which is also what the normal
allocation path will do, to fail high order allocations without OOM
killing.

==== Testing ====

Tested on a VM with 6 cores and 6GiB of memory, HZ=1000, MGLRU enabled,
min_ttl_ms set to 1000, and a workload that fragments the memory then
allocates 2M pages.

Without the patch, kswapd kills all processes and panics the system even
with plenty of memory available. With the patch, allocations fail but
the system remains stable.

Fixes: 30d77b7eef019 ("mm/mglru: fix ineffective protection calculation")
Signed-off-by: Bingfang Guo <bingfangguo@xxxxxxxxxxx>
---
 mm/vmscan.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 9959ca158ce83..b32777468ead8 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4407,6 +4407,12 @@ static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_control *sc)
 			reclaimable = lruvec_is_reclaimable(lruvec, sc, min_ttl);
 	} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
 
+	/*
+	 * OOM killing won't help high order allocations. Skip and let them fail.
+	 */
+	if (sc->order > PAGE_ALLOC_COSTLY_ORDER)
+		return;
+
 	/*
 	 * The main goal is to OOM kill if every generation from all memcgs is
 	 * younger than min_ttl. However, another possibility is all memcgs are

---
base-commit: 39274b454e9993cea319f4ac342a8082a9ee1730
change-id: 20261026-fix-mglru-kswapd-age-98390614c746

Best regards,
--  
Bingfang Guo <bingfangguo@xxxxxxxxxxx>