[PATCH] mm: workingset: account MGLRU pages in count_shadow_nodes()

[Hui Zhu]

From: Hui Zhu <zhuhui@xxxxxxxxxx>

Commit 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the
number of lru pages") switched count_shadow_nodes() to
lruvec_lru_size().  With CONFIG_MEMCG enabled, lruvec_lru_size() reads
mz->lru_zone_size, which is only maintained through
mem_cgroup_update_lru_size().  MGLRU never goes there: all of its
accounting (lru_gen_update_size(), reset_batch_size(), inc_max_seq()
and __lru_gen_reparent_memcg()) uses __update_lru_size(), which skips
the memcg array, and the classic lruvec_add/del_folio() paths return
early once lru_gen_add/del_folio() succeeded.

The four evictable LRU lists are therefore always seen as empty when
MGLRU is on, and the shadow node budget (pages >> 3) is based on slab
and unevictable pages only.  For a memcg holding 1 GiB of page cache
and 64 MiB of slab the budget drops from ~35k nodes to ~2k, so the
workingset shadow shrinker reclaims eviction tokens almost as fast as
they are created.  Refaults then find live pages instead of shadow
entries, lru_gen_refault() cannot restore the workingset state of hot
pages, and thrashing protection is lost.  This hits every memcg reclaim
and, since the root memcg is iterated as well, global reclaim too.

Fix this by adding the per-generation counters of an enabled multi-gen
LRU to the budget.  lrugen->nr_pages[] is eventually consistent and may
go transiently negative while batched updates are pending, so clamp the
sum at zero.  Keep lruvec_lru_size() as it still accounts for the
unevictable pages.

Fixes: 7404bd37cfbe ("mm: workingset: use lruvec_lru_size() to get the number of lru pages")
Signed-off-by: Hui Zhu <zhuhui@xxxxxxxxxx>
---
 mm/workingset.c | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/mm/workingset.c b/mm/workingset.c
index f351798e723a..da20034d28fc 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -653,6 +653,37 @@ void workingset_update_node(struct xa_node *node)
 	}
 }
 
+#ifdef CONFIG_LRU_GEN
+/*
+ * The per-generation counters are eventually consistent and may go
+ * transiently negative while batched updates are pending, so clamp
+ * the sum at zero.
+ */
+static unsigned long count_lru_gen_pages(struct lruvec *lruvec)
+{
+	struct lru_gen_folio *lrugen = &lruvec->lrugen;
+	long nr_pages = 0;
+	int gen, type, zone;
+
+	if (!lrugen->enabled)
+		return 0;
+
+	for (gen = 0; gen < MAX_NR_GENS; gen++) {
+		for (type = 0; type < ANON_AND_FILE; type++) {
+			long *cnt = lrugen->nr_pages[gen][type];
+
+			for (zone = 0; zone < MAX_NR_ZONES; zone++)
+				nr_pages += READ_ONCE(cnt[zone]);
+		}
+	}
+
+	if (nr_pages <= 0)
+		return 0;
+
+	return nr_pages;
+}
+#endif
+
 static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 					struct shrink_control *sc)
 {
@@ -697,6 +728,15 @@ static unsigned long count_shadow_nodes(struct shrinker *shrinker,
 		for (pages = 0, i = 0; i < NR_LRU_LISTS; i++)
 			pages += lruvec_lru_size(lruvec, i, MAX_NR_ZONES - 1);
 
+#ifdef CONFIG_LRU_GEN
+		/*
+		 * MGLRU accounts for the evictable pages in
+		 * lrugen->nr_pages[] instead of mz->lru_zone_size, which
+		 * lruvec_lru_size() reads, so the loop above misses them.
+		 */
+		pages += count_lru_gen_pages(lruvec);
+#endif
+
 		pages += lruvec_page_state_local(
 			lruvec, NR_SLAB_RECLAIMABLE_B) >> PAGE_SHIFT;
 		pages += lruvec_page_state_local(
-- 
2.53.0