[RFC PATCH 3/3] mm/mglru: move lrugen atomic stats off the read-mostly lines

[Ridong Chen]

From: Ridong Chen <chenridong@xxxxxxxxxx>

lru_gen_eviction() and lru_gen_refault() update lrugen->evicted[] and
lrugen->refaulted[] locklessly with atomic_long_add(). The arrays sit at
the tail of struct lru_gen_folio, right before enabled/gen/seg/list:
with LRU_GEN_STATS=n the file type counters share a cacheline with them.

Those fields are read far more often than they are written:
lru_gen_add_folio() reads ->enabled on every LRU add, and shrink_many()
walks ->list and reads ->gen/->seg under RCU. With frequent file refaults
from several CPUs, every refault invalidates that line on the CPUs doing
LRU adds.

Move enabled/gen/seg/list next to max_seq/min_seq/timestamps, which are
also mostly read, and put evicted[]/refaulted[] in their own cacheline
group at the end of the struct. This grows struct lru_gen_folio from
1360 to 1408 bytes and struct mem_cgroup_per_node from 2176 to 2240
bytes on x86_64 (one more cacheline per memcg per node).

Tested on an i7-13700HX (24 threads) with turbo disabled and the
performance governor. 24 files of 128MiB on ext4 on brd, read with
fio (psync, randread, bs=4k, fadvise_hint=random, buffered), all jobs
in one memcg with memory.max=1G, so nearly every read refaults.
IOPS, median of 5 runs:

  jobs      base   patched
     1    270537    274303   +1.4%  (sd 3.7%)
     8   1461025   1466293   +0.4%
    16   2455629   2508001   +2.1%
    24   2707720   2823776   +4.3%

A custom random pread() test of the same setup, with this patch alone
against base over two boots of each kernel, shows +2.4% at 16 threads
and +5.9% at 24 threads, and no change at 8 threads or fewer. With
anonymous memory on zram, the refaults mostly hit the tier 0 counters
on a different line, and there is no measurable change.

The gain grows with the number of CPUs refaulting into the same lruvec
and needs a fast backing store; with real IO it is likely to be lost in
the noise. The patch is mostly meant to keep the lockless counters away
from the read-mostly fields regardless of where they end up.

Signed-off-by: Ridong Chen <chenridong@xxxxxxxxxx>
---
 include/linux/mmzone.h | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 6442e662aa4f..eb84af674e23 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -551,6 +551,14 @@ struct lru_gen_folio {
 	unsigned long min_seq[ANON_AND_FILE];
 	/* the birth time of each generation in jiffies */
 	unsigned long timestamps[MAX_NR_GENS];
+	/* whether the multi-gen LRU is enabled */
+	bool enabled;
+	/* the memcg generation this lru_gen_folio belongs to */
+	u8 gen;
+	/* the list segment this lru_gen_folio belongs to */
+	u8 seg;
+	/* per-node lru_gen_folio list for global reclaim */
+	struct hlist_nulls_node list;
 	/* the multi-gen LRU lists, lazily sorted on eviction */
 	struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* the multi-gen LRU sizes, eventually consistent */
@@ -562,16 +570,10 @@ struct lru_gen_folio {
 	/* can only be modified under the LRU lock */
 	unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	/* can be modified without holding the LRU lock */
+	__cacheline_group_begin_aligned(lrugen_stats);
 	atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
 	atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
-	/* whether the multi-gen LRU is enabled */
-	bool enabled;
-	/* the memcg generation this lru_gen_folio belongs to */
-	u8 gen;
-	/* the list segment this lru_gen_folio belongs to */
-	u8 seg;
-	/* per-node lru_gen_folio list for global reclaim */
-	struct hlist_nulls_node list;
+	__cacheline_group_end_aligned(lrugen_stats);
 };
 
 enum {
-- 
2.34.1