Re: [RFC PATCH 3/3] mm/mglru: move lrugen atomic stats off the read-mostly lines

[Barry Song]

On Thu, Nov 5, 2026 at 9:23 PM Ridong Chen <ridong.chen@xxxxxxxxx> wrote:
>
> From: Ridong Chen <chenridong@xxxxxxxxxx>
>
> lru_gen_eviction() and lru_gen_refault() update lrugen->evicted[] and
> lrugen->refaulted[] locklessly with atomic_long_add(). The arrays sit at
> the tail of struct lru_gen_folio, right before enabled/gen/seg/list:
> with LRU_GEN_STATS=n the file type counters share a cacheline with them.

So basically, this is causing false sharing, right?

I'd prefer a text diagram showing how the false sharing
occurs, specifically how freq read and freq write end up
in the same cacheline.

This will help clarify why rearranging the struct is the
right approach and how to rearrange it properly.

>
> Those fields are read far more often than they are written:
> lru_gen_add_folio() reads ->enabled on every LRU add, and shrink_many()
> walks ->list and reads ->gen/->seg under RCU. With frequent file refaults
> from several CPUs, every refault invalidates that line on the CPUs doing
> LRU adds.
>
> Move enabled/gen/seg/list next to max_seq/min_seq/timestamps, which are
> also mostly read, and put evicted[]/refaulted[] in their own cacheline
> group at the end of the struct. This grows struct lru_gen_folio from
> 1360 to 1408 bytes and struct mem_cgroup_per_node from 2176 to 2240

Why is the difference exactly 2240 - 2176 = 64 bytes,
which is one cacheline? I was expecting a value smaller
than one cacheline :-)

What I'm not convinced about is that nr_pages should be
updated even more frequently than evicted and refaulted.
Why aren't they causing a false sharing?

long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];

It would be better to have a text diagram showing the
potential false sharing, so we can consider the issue
more comprehensively.

> bytes on x86_64 (one more cacheline per memcg per node).
>
> Tested on an i7-13700HX (24 threads) with turbo disabled and the
> performance governor. 24 files of 128MiB on ext4 on brd, read with
> fio (psync, randread, bs=4k, fadvise_hint=random, buffered), all jobs
> in one memcg with memory.max=1G, so nearly every read refaults.
> IOPS, median of 5 runs:
>
>   jobs      base   patched
>      1    270537    274303   +1.4%  (sd 3.7%)
>      8   1461025   1466293   +0.4%
>     16   2455629   2508001   +2.1%
>     24   2707720   2823776   +4.3%
>
> A custom random pread() test of the same setup, with this patch alone
> against base over two boots of each kernel, shows +2.4% at 16 threads
> and +5.9% at 24 threads, and no change at 8 threads or fewer. With
> anonymous memory on zram, the refaults mostly hit the tier 0 counters
> on a different line, and there is no measurable change.
>
> The gain grows with the number of CPUs refaulting into the same lruvec
> and needs a fast backing store; with real IO it is likely to be lost in
> the noise. The patch is mostly meant to keep the lockless counters away
> from the read-mostly fields regardless of where they end up.
>
> Signed-off-by: Ridong Chen <chenridong@xxxxxxxxxx>
> ---
>  include/linux/mmzone.h | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
>

Thanks,
Barry