Re: [PATCH RFC 09/15] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
From: Kairui Song
Date: Tue Aug 18 2026 - 03:48:08 EST
On Tue, Aug 18, 2026 at 3:12 PM Baoquan He <baoquan.he@xxxxxxxxx> wrote:
>
> On 08/04/26 at 03:47am, Kairui Song via B4 Relay wrote:
> ...snip...
> > +int folio_inc_lru_refs(struct folio *folio, bool is_fault, bool is_exec)
> > +{
> > + int max_gen, min_gen;
> > + int type, refs, gen, new_gen;
> > + unsigned long new_flags, old_flags, max_seq;
> > + struct lru_gen_folio *lrugen;
> > + struct lruvec *lruvec;
> > +
> > + type = folio_is_file_lru(folio);
> > + lruvec = folio_lruvec_live_get(folio);
> > + lrugen = &lruvec->lrugen;
> > +
> > + old_flags = READ_ONCE(*folio_flags(folio, 0));
> > + do {
> > + new_flags = old_flags;
> > + gen = lru_gen_from_flags(old_flags);
> > + refs = lru_refs_from_flags(old_flags) + 1;
> > + new_gen = gen;
> > + if (!(old_flags & BIT(PG_lru)) || gen < 0)
> > + goto out;
> > +
> > + max_seq = READ_ONCE(lrugen->max_seq);
> > + max_gen = lru_gen_from_seq(max_seq);
> > + min_gen = lru_gen_from_seq(READ_ONCE(lrugen->min_seq[type]));
> > + if (gen == max_gen)
> > + goto out;
> > +
>
> I am a little confused about the new mechanism. In the current mglru, it
> does have the issue both mm walk and fd read set PG_referenced at the
> 1st access, this is a obvious drawback. Now with the change, the ref
> count is clearer, while the mm walk and fd read accessing is still mixed.
> Imagine the cases below:
> - one fd read; then mm walk; directly move to max_gen;
> - one mm walk; then several times fd read; promote to next gen;
>
> Can I understand the final effect as:
> 1) explicti ref count;
> 2) more drastically promote mm walk based on the mixing ref counting;
> - compared with the old behaviour: move to next gen when 2nd mm walk
Hi Baoquan, Thanks a lot for the review!
The old behavior is move to max_gen on walk access (unless it's the
first access of a folio), and only protect non-mapped folios with PID
refaults. Now, the non-mapped folios are also promoted after multiple
accesses.
The promotion is more proactive but not that drastic; it still
requires 8 accesses (2^3, which overflows tier 4) to force promote one
folio.
For anonymous pages, this also provides a more consistent and explicit
reference count mechanism, and you will see folios distributed fairly
among tiers now, as repeated page table access will increase the refs
above LRU_REFS_PROTECTED. Previously, all anon folios were stuck at
tiers 0, 1, and mostly 3, skipping tier 2 completely.
Also we reduced the bit usage by 1.
This new mechanism is supposed to work as a whole, This new mechanism
is supposed to work as a whole; it is documented and described in the
chunk of comments in mmzone.h (maybe not the best place but old
MGLRU's comments are there so I just updated it inplace). I tried to
split it into smaller parts, which doesn't seem doable.
And I think the three folio_inc_lru_refs* helper are also kind of self
explaining on this.
> I can only see one benefit and one significant change. Do I understand
> it correctly, and is it worth?
I think it's definitely worth it, if you look at the results in the
cover letter :), especially the zipf access test (1.2, 1.1, 0.9, 0.8
have similar results, zipf is commonly used standard to simulate
realworld access patterns) and a few other cases; 90% of the
performance gain comes from this design. I tweaked the refs &
promotion rule many times to double check and ensure it fits well for
different workloads.
And note that this is not an anon/file reclaim balance tradeoff issue,
both anon refaults, file refaults, and pgpgins are reduced, meaning
MGLRU is doing better at protecting both hotter anon and hotter files.
It's widely complaines that MGLRU was actually performing poor on
unmapped folio protection, worse than classical LRU in many workloads
(Not the LRU overhead, MGLRU had lower overhead, but the actual
ability to protect the hotter cache is not good, provable with the
zipf test in the cover letter). After this change, its performance is
obviously better than that of classical LRU.
Attachment: a typical tier layout of a workload before this change:
node 0
35 3910 37524 0x
0 238r 15596e 0p 0x
0x 0x
1 409r 6429e 6211p 0x
0x 0x
2 0r 0e 0p 0x
0x 0x
3 895r 39182e 8588p 0x
0x 0x
0x 0x 0x 0x
36 3244 84684 0
0 0x 0x 0x 2882r
6059e 0p
1 0x 0x 0x 448r
964e 0p
2 0x 0x 0x 298r
439e 0p
3 0x 0x 0x 416r
721e 0p
0x 0x 0x 0x
37 2858 221580 4269
0 0x 0x 0x 0x
0x 0x
1 0x 0x 0x 0x
0x 0x
2 0x 0x 0x 0x
0x 0x
3 0x 0x 0x 0x
0x 0x
0x 0x 0x 0x
38 2183 29167 242
0 12R 2499T 0x 2496R
4101T 0x
1 322R 29241T 0x 92R
156T 0x
2 0R 0T 0x 100R
144T 0x
3 150R 75764T 0x 741R
1161T 0x
20903930T 5439295Y 48822F 19595A
And after:
node 0
47 3227 197247 0x
0 967r 35231e 0p 0x
0x 0x
1 412r 6871e 1051p 0x
0x 0x
2 12r 639e 2319p 0x
0x 0x
3 0r 0e 49p 0x
0x 0x
0x 0x 0x 0x
48 1623 49290 0
0 0x 0x 0x 2065r
3586e 0p
1 0x 0x 0x 338r
836e 0p
2 0x 0x 0x 352r
817e 0p
3 0x 0x 0x 107r
139e 0p
0x 0x 0x 0x
49 1491 91927 4865
0 0x 0x 0x 0x
0x 0x
1 0x 0x 0x 0x
0x 0x
2 0x 0x 0x 0x
0x 0x
3 0x 0x 0x 0x
0x 0x
0x 0x 0x 0x
50 1305 10444 674
0 69R 6630T 0x 1082R
2586T 0x
1 21R 2509T 0x 55R
168T 0x
2 71R 2322T 0x 111R
288T 0x
3 0R 6T 0x 114R
619T 0x
31126090T 7780753Y 72142F 27409A
For file heavy workloads, the file folios are also now fairly
distributed amon gens instead of being stuck at tail gen. And it's not
just looks prettier, the performance is indeed better.