[PATCH 49/60] mm/vmscan: reclaim private nodes for memcg

From: Gregory Price via B4 Relay

Date: Fri Nov 20 2026 - 13:18:18 EST


From: "Gregory Price (Meta)" <gourry@xxxxxxxxxx>

Memcg reclaim does not serve one allocation, so it should cover every
node that permits reclaim. It walks the current node's fallback
zonelist, which holds no private node, so a private node that permits
reclaim is never scanned for a cgroup over its limit.

After the zonelist, visit the nodes that permit reclaim and that the
zonelist did not cover, and give them the same refault snapshot and
congestion reset. The zonelist keeps setting the order, which matters
because memcg reclaim stops once it has freed enough.

Direct reclaim for an allocation still walks only that allocation's
zonelist. A private allocation's zonelist covers its own node, and
reclaiming private memory cannot satisfy a common allocation.

Without private nodes the zonelist covers every node that permits
reclaim, so behavior is unchanged.

Another way to do this would be to implement a reclaim zonelist that
only reclaim uses, but this implementation does not carry the static
per-node size overhead that adding another zonelist does.

Signed-off-by: Gregory Price (Meta) <gourry@xxxxxxxxxx>
---
mm/vmscan.c | 43 ++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 42 insertions(+), 1 deletion(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 03d0919d11be..49ec85ed00e0 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6636,6 +6636,21 @@ static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS);
}

+/* Whether shrink_zones() reaches @nid through @zonelist. */
+static bool zonelist_reaches_node(struct zonelist *zonelist, int nid,
+ struct scan_control *sc)
+{
+ struct zoneref *z;
+ struct zone *zone;
+
+ for_each_zone_zonelist_nodemask(zone, z, zonelist, sc->reclaim_idx,
+ sc->nodemask) {
+ if (zone_to_nid(zone) == nid)
+ return true;
+ }
+ return false;
+}
+
/*
* This is the direct reclaim path, for page-allocating processes. We only
* try to reclaim pages from zones which will satisfy the caller's allocation
@@ -6651,6 +6666,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
gfp_t orig_mask;
pg_data_t *last_pgdat = NULL;
pg_data_t *first_pgdat = NULL;
+ int nid;

/*
* If the number of buffer_heads in the machine exceeds the maximum
@@ -6706,6 +6722,14 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
shrink_node(zone->zone_pgdat, sc);
}

+ /* A cgroup's memory may be on nodes that are not on the zonelist. */
+ if (cgroup_reclaim(sc)) {
+ for_each_node_state(nid, N_MEMORY_RECLAIM) {
+ if (!zonelist_reaches_node(zonelist, nid, sc))
+ shrink_node(NODE_DATA(nid), sc);
+ }
+ }
+
if (first_pgdat)
consider_reclaim_throttle(first_pgdat, sc);

@@ -6754,6 +6778,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
pg_data_t *last_pgdat;
struct zoneref *z;
struct zone *zone;
+ int nid;
retry:
delayacct_freepages_start();

@@ -6792,6 +6817,21 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
}
}

+ /*
+ * Also cover the nodes shrink_zones() reclaimed beyond the zonelist.
+ * Finishing a zonelist node again is harmless.
+ */
+ if (cgroup_reclaim(sc)) {
+ for_each_node_state(nid, N_MEMORY_RECLAIM) {
+ struct lruvec *lruvec;
+
+ snapshot_refaults(sc->target_mem_cgroup, NODE_DATA(nid));
+ lruvec = mem_cgroup_lruvec(sc->target_mem_cgroup,
+ NODE_DATA(nid));
+ clear_bit(LRUVEC_CGROUP_CONGESTED, &lruvec->flags);
+ }
+ }
+
delayacct_freepages_end();

if (sc->nr_reclaimed)
@@ -7038,7 +7078,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
/*
* Traverse the ZONELIST_FALLBACK zonelist of the current node to put
* equal pressure on all the nodes. This is based on the assumption that
- * the reclaim does not bail out early.
+ * the reclaim does not bail out early. shrink_zones() also visits the
+ * nodes that permit reclaim and are not on the zonelist.
*/
struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask);


--
2.56.1