Re: [PATCH v4 04/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG)

[Kairui Song]

On Mon, Oct 19, 2026 at 6:10 PM Barry Song <baohua@xxxxxxxxxx> wrote:
>
> On Fri, Oct 16, 2026 at 2:15 AM Kairui Song via B4 Relay
> <devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
> >
> > From: Kairui Song <kasong@xxxxxxxxxxx>
> >
> > Complement MGLRU's eviction-time tier-PID protection with access-time
> > frequency-guided promotion.  Introduce a unified set of helpers built based
> > on referenced (access) count of a folio.
> >
> > Each access increments a folio's referenced count stored in folio flags
> > (refs), refs still maps to a logarithmic tier just like before, but with
> > more formal bit definitions, a few special thresholds are introduced:
> > LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> > and LRU_REFS_MAX (7). When refs reaches a certain threshold, the folio is
> > promoted proactively instead of waiting for the PID controller to kick in.
> >
> > Also simplify MGLRU's usage of PG_workingset and PG_referenced: they
> > become the low two bits of the refs count, with the higher bits
> > provided by LRU_REFS_MASK.  This reduces MGLRU's original refs count
> > bit usage by one, since only one extra bit is now needed to record a
> > max referenced count of 7, and makes MGLRU's refs accounting more
> > accurate.
> >
> > This doesn't affect classical LRU in any way, and it addresses several
> > shortcomings of MGLRU's old tier-only cache protection model:
> >
> > - Long feedback loop: protection only activated after enough re-faults,
> >   by which time the hot folios are already evicted, or no longer hot.
> >
> > - Limited tier resolution: once referenced count exceeded the bits
> >   limit (8 previously), MGLRU could no longer distinguish hotter folios as
> >   they are capped by the tier.  And what's worse, PG_workingset forces
> >   a folio to stay on tier 3.
> >
> > - Eviction hotness reversion: because PID protection activates upon
> >   eviction and always targets the LRU tail, it tends to protect cold tail
> >   folios at the expense of hotter head folios.  Once the tail folios
> >   consume the PID protection budget, head folios lose their protection.
> >
> > - Additionally, the PID cannot distinguish the access time of folios
> >   that share the same reference count, and there are only 4 tiers.
> >
> > To achieve a frequency-guided framework, this commit introduces and reworks
> > the LRU_REFS related helpers and definitions; most of the work is done by
> > the helpers below, and their inline comments describe the details.
> >
> > - folio_inc_lru_refs(): Called on any cache access (folio_mark_accessed)
> >   or page table access.  This is the main helper: it promotes folios
> >   according to their access frequency.
> >
> >   A first access only defers eviction for a folio at the tail of its
> >   LRU, i.e., in one of the MIN_NR_GENS oldest generations of its own
> >   type (see lru_gen_at_tail()), and a workingset folio accessed through
> >   a file descriptor is deferred from the same tail.  Those are the
> >   generations the eviction scans next, so the deferral moves the folio
> >   out of its way for one generation.
> >
> >   Promotion is lazy: the gen bits and size counters are updated eagerly,
> >   while the list move is deferred to the next isolation.  NOTE: For now,
> >   the lruvec lock is unconditionally taken on every on-list access to
> >   block concurrent aging; a lockless fast path will be implemented very
> >   soon in a following commit.
> >
> > - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging and
> >   by the batched look-around, both without the LRU lock, so max_seq may
> >   be stale.  A first page table access only advances a folio at the tail
> >   of its LRU, i.e., in one of the MIN_NR_GENS oldest generations of its
> >   own type (second chance), a second access promotes it to the newest
> >   generation; this also performs lazy promotion.
> >
> > - folio_inc_lru_refs_isolated(): Used by the rmap check before
> >   eviction.  The folio is isolated and hence this doesn't perform
> >   promotion by itself; the folio will be added back to the right gen
> >   upon return according to the access frequency.  This path also has a
> >   higher promotion bias.
> >
> > The eviction-time folio_inc_gen() still handles PID protection, but the
> > protection ratio is softer than before since proactive promotion is
> > mostly good enough already.  The PID gain factors are relaxed from
> > (2:3) to (1:2) and the setpoint now spans the cumulative mass of the
> > tiers below the candidate instead of tier 0 alone.  folio_inc_gen()
> > caps refs at WORKINGSET so the folio retains enough history to stay
> > above the cold tier.  Tier 1 is the fallback tier for PID, and tier 2
> > is the fallback tier for frequency-guided promotion.  The forced
> > protection for full-refs folios is removed, obsoleted by the proactive
> > promotion.
> >
> > Refaults are now activated purely according to access frequency:
> > the old fault bias applied in folio_add_lru() is simplified, since a
> > folio's access history is a more consistent signal than the context of
> > the faulting task.  Page table access is still considered a slightly
> > stronger signal.
> >
> > Consequently the placement hint for newly added file folios also uses
> > LRU_REFS_WORKINGSET instead of PG_referenced, so a folio accessed only
> > once now starts in the oldest generation rather than the second oldest.
> > The previous one-generation grace was an artifact of the fault bias which
> > is well described in the cover letter of this series, and it depended
> > on LRU drain timing.
> >
> > This also redefines PG_workingset and PG_referenced as the low two
> > bits of the refs count, eliminating the old restriction where
> > LRU_REFS_MASK was only valid when PG_referenced was set, and allows
> > all paths to use the same encoding consistently.  Following this idea,
> > a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET (2),
> > matching the active/inactive LRU's definition and giving in-kernel
> > consumers (PSI, readahead) consistent behavior on MGLRU, fixing the
> > longstanding issue that these users don't work well with MGLRU.
> >
> > PG_workingset and PG_referenced are no longer independent flags under
> > MGLRU.  Adjusting existing raw folio_test_*() callers to the new
> > semantics is left as follow-ups.
>
> So this is the main course, with the earlier ones as appetizers and
> the later ones as desserts?
> Overall, this dish looks very delicious to me.
>
> I ran kernel build on my machine, the result is quite similar with[1].
>
> W/o patch:
>
> eal    1m49.094s
> user    25m31.484s
> sys     3m40.959s
> pswpin: 662112
> pswpout: 2504415
> pgpgin: 4424336
> pgpgout: 10355273
> refault_file: 359626
> refault_anon: 670724
>
> W/ patch:
>
> real 1m48.944s
> user 25m30.987s
> sys 3m34.605s
> pswpin: 467952
> pswpout: 2319080
> pgpgin: 3481187
> pgpgout: 9603332
> swpout_zero: 602042
> swpin_zero: 6250
> refault_file: 310538
> refault_anon: 474154
>
> But it behaves in a much more sensible way.

Hi Barry,

Thanks for the review and testing. Glad to see it worked for you.

>
> I've had a preliminary look, but haven't reviewed it in detail yet.
> I have a few comments.
>
> [1] https://lore.kernel.org/linux-mm/20261016213115.70515-1-baohua@xxxxxxxxxx/

Right there could be some similarities, and some differences, e.g. no
more lru_gen_in_fault hanlding here, or difference between anon /
file. BTW did you test the whole series or just this patch? This patch
have a few workaround as intermeidate step (e.g. the forced LRU lock
in folio_mark_access, which is a pure burden but have to stay until
very end of the series).

> >  /**
> > @@ -212,13 +216,13 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
> >   */
> >  static inline int lru_get_refs_flags(unsigned long flags)
> >  {
> > -       if (!(flags & BIT(PG_referenced)))
> > -               return 0;
> > -       /*
> > -        * Return the total number of accesses including PG_referenced. Also see
> > -        * the comment on LRU_REFS_FLAGS.
> > -        */
> > -       return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
> > +       int refs;
> > +
> > +       /* Return the total number of accesses. See the comment above MAX_NR_TIERS. */
> > +       refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
> > +       refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
> > +       refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
> > +       return refs;
> >  }
>
> Could we use the following instead? Coming from a hardware register background,
> this looks more like register encoding to me:
>
> static inline int lru_get_refs_flags(unsigned long flags)
> {
>     int refs;
>
>     refs  = !!(flags & BIT(PG_referenced));
>     refs |= !!(flags & BIT(PG_workingset)) << 1;
>     refs |= ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
>     return refs;
> }

No problem; it means the same thing and looks better indeed. I think I
better change the return type to unsigned too, refs can never be
negative (gen could be though, as a reserved value).

>
> >
> >  static inline int folio_lru_refs(const struct folio *folio)
> > @@ -236,14 +240,29 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> >         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> >  }
> >
> > +void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
> > +
> >  static inline int folio_lru_gen(const struct folio *folio)
> >  {
> >         return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> >  }
> >
> > +static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
> > +{
> > +       unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
> > +
> > +       /*
> > +        * Consider a folio is at the tail of its LRU when it is within
> > +        * MIN_NR_GENS of the oldest generation of its own type
> > +        */
> > +       BUILD_BUG_ON(MIN_NR_GENS != 2);
> > +       return gen == lru_gen_from_seq(min_seq) ||
> > +              gen == lru_gen_from_seq(min_seq + 1);
> > +}
>
> If only two generations remain, should both be considered hot?
> If three generations remain, is only the oldest generation old?
>
> Also, the name is a bit confusing, as "tail" usually refers
> to a list. Could we rename it to one of the following?
>
> lru_gen_is_old()
> or
> lru_gen_is_near_oldest()

Right, I wanted to introduce an idea of something called tailing gens
(e.g. 8 tailing gens when there are 64 gens in total, folios in
tailing gens are promoted on each access instead of multiple access,
this works really well for fine grained hotness detection, and avoids
over protection too). But for now this helper is just meant to tell
the caller if the folio is close to be evicted or not, checking
min_seq only could easily fall into a pitfall where the oldest gen is
very short.

Maybe lru_gen_at_expiry? Or lru_gen_at_salvage?

BTW the whole picture involves three kinds of gens:
{tailing/expiry/salvage} gens (promote by one gen on each one FD
access), frequency gens (promote by one gen on multiple FD access),
times gens (promote to latest gen on PTE access); We now only have
expiry gens and frequency gens, timed gens are extensions for the
workingset report or pghot design. e.g we have 64 gens, then 8 tailing
gens are expiry gens, 16 frequency gens, and 48 newest gens are timed
gens.
https://lwn.net/Articles/976985/

I mentioned this in the cover letter that the "climbing of the file
cache" will stop at 8 gens or 16 gens, this is the whole story, there
is a lot of context and plan so I trimed them off to focus on the FG
improvement first..

>
> > +
> >  static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
> >  {
> > -       unsigned long max_seq = lruvec->lrugen.max_seq;
> > +       unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> >
> >         VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> >
> > @@ -300,23 +319,24 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
> >                                               bool reclaiming)
> >  {
> >         int gen;
> > +       int refs = folio_lru_refs(folio);
> >         int type = folio_is_file_lru(folio);
> >         const struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >
> >         /*
> > -        * +-----------------------------------+-----------------------------------+
> > -        * | Accessed through page tables and  | Accessed through file descriptors |
> > -        * | promoted by folio_update_gen()    | and protected by folio_inc_gen()  |
> > -        * +-----------------------------------+-----------------------------------+
> > -        * | PG_active (set while isolated)    |                                   |
> > -        * +-----------------+-----------------+-----------------+-----------------+
> > -        * |  PG_workingset  |  PG_referenced  |  PG_workingset  |  LRU_REFS_FLAGS |
> > -        * +-----------------------------------+-----------------------------------+
> > -        * |<---------- MIN_NR_GENS ---------->|                                   |
> > -        * |<---------------------------- MAX_NR_GENS ---------------------------->|
> > +        * +------------------------------------------+------------------------------------------+
> > +        * |     Accessed through page tables and     |     Accessed through file descriptors    |
> > +        * | promoted by folio_inc_lru_refs_walk()    | protected by folio_inc_lru_refs/inc_gen  |
> > +        * +------------------------------------------+------------------------------------------+
> > +        * | PG_active (set at isolation or refault)  |                                          |
> > +        * +--------------------+---------------------+--------------------+---------------------+
> > +        * |     LRU_REFS_MAX   | LRU_REFS_WORKINGSET |    LRU_REFS_MAX    | LRU_REFS_WORKINGSET |
> > +        * +------------------------------------------+------------------------------------------+
> > +        * |<-------------- MIN_NR_GENS ------------->|                                          |
> > +        * |<----------------------------------- MAX_NR_GENS ----------------------------------->|
> >          */
> >         if (folio_test_active(folio))
> > -               gen = MIN_NR_GENS - folio_test_workingset(folio);
> > +               gen = MIN_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
>
> Could we define the "workingset" variant earlier so we can reuse it here?
>
> workingset = refs >= LRU_REFS_WORKINGSET;

Looks good, will do.

> > +/*
> > + * Update the folio's lru refs indicator. The caller doesn't need to hold
> > + * the folio lock, isolate the folio, or hold the lruvec lock. Used by both
> > + * cache access (flags == 0) and page table access (LRU_REF_MAPPED,
> > + * optionally with LRU_REF_EXEC).
> > + */
> > +void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> > +{
> > +       int max_gen;
> > +       int type, refs, old_gen, gen;
> > +       unsigned long new_flags, old_flags, max_seq;
> > +       struct lru_gen_folio *lrugen;
> > +       struct lruvec *lruvec = NULL;
> > +
> > +       type = folio_is_file_lru(folio);
> > +       old_flags = READ_ONCE(*folio_flags(folio, 0));
> > +       do {
> > +               new_flags = old_flags;
> > +               old_gen = lru_get_gen_flags(old_flags);
> > +               refs = lru_get_refs_flags(old_flags) + 1;
> > +               gen = old_gen;
> > +               if (old_gen < 0)
> > +                       goto out;
> > +               /*
> > +                * Lock the lruvec if the folio is on-list. We are already
> > +                * doing lazy promotion so in theory we don't need this,
> > +                * but for now, concurrent aging would still corrupt the
> > +                * size counters.  This is a temporary limitation and
> > +                * will be lifted very soon, so the lock here is not a
> > +                * performance concern.
> > +                */
> > +               if (!lruvec) {
> > +                       lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
> > +                       lrugen = &lruvec->lrugen;
> >                 }
> > +               max_seq = READ_ONCE(lrugen->max_seq);
> > +               max_gen = lru_gen_from_seq(max_seq);
> > +               if (old_gen == max_gen)
> > +                       goto out;
> >
> > -               folio_set_lru_refs(folio, 1);
> > -               return false;
> > -       }
> > +               if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
> > +                       /* Promote second page table access or executable */
> > +                       if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
> > +                               gen = max_gen;
>
> This looks a bit odd to me. Could we handle LRU_REF_EXEC in a
> separate branch instead of mixing it with refs > LRU_REFS_REFERENCED?

Not sure how? They seem to want the same thing: promote the folio.

> > +                       /* First access only defers eviction of a tail gen */
> > +                       else if (lru_gen_at_tail(lruvec, type, old_gen))
> > +                               gen = (old_gen + 1) % MAX_NR_GENS;
>
> The logic makes a lot of sense to me. It should be able to
> differentiate readahead folios from those that are actually mapped
> and likely to be accessed more than once.
>
> If we allow two generations to remain, this could be a problem
> because old_gen + 1 would overflow. Maybe add a
> VM_WARN_ON() to guard against this case?

We just checked if (old_gen == max_gen), so this should be fine? I
mean overflow and the mod MAX_NR_GENS is also part of the sliding
window logic.

> > +                       refs = min(refs, LRU_REFS_PROTECTED);
> > +               } else if (refs > LRU_REFS_MAX) {
> > +                       /* LRU refs counting overflow, bump the gen */
> > +                       gen = (old_gen + 1) % MAX_NR_GENS;
> > +                       refs = LRU_REFS_PROTECTED;
> > +               } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {
>
> Sometimes you use >= LRU_REFS_WORKINGSET, while elsewhere
> you use refs > LRU_REFS_REFERENCED. Can we unify these
> checks or define a workingset variant?

No problem; I can use a macro for that, as they mean the same thing.

> > +/*
> > + * Update the lru refs indicator of an isolated folio, only used on
> > + * mapped folios upon the final eviction.
> > + *
> > + * Increments the refs count (capped at LRU_REFS_PROTECTED, evict_folios()
> > + * then caps it at LRU_REFS_WORKINGSET).  Returns true if the caller should
> > + * activate the folio (second access or executable), false to put it back
> > + * for a second chance.
> > + */
> > +static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
> > +{
> > +       unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> > +       int refs;
> > +
> > +       do {
> > +               new_flags = old_flags;
> > +               refs = lru_get_refs_flags(old_flags) + 1;
> > +               lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
>
> I'm having a hard time understanding why an activated folio
> still keeps LRU_REFS_PROTECTED, since it will be promoted
> to the active list anyway :-)

Right the FG idea is to always retain the frequency counts, instead of
resetting them to 0, so folio_inc_lru_refs can promote FD accessed
folios much more proactively, when a folio reaches LRU_REFS_MAX it
will be promoted by one gen.

I did a in depth analyze of runtime distribution, with this, we can
see very hot folio being promoted from max_seq - 1 to max_seq, and
after aging, the hotness info is still kept and workingset stay stable
and accurate (it's like the global aging of traditional LFU, which is
obsoluted in upstream due to overhead despite having better accuracy,
but now it comes with almost zero cost).

>
> Later, I see that you set it to LRU_REFS_WORKINGSET in
> evict_folios():
>
> folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));

Right that part is meant for aging the older folios and PID controller
though. LRU_REFS_WORKINGSET could means historical workingset, and
LRU_REFS_PROTECTED means folios being currently actively protected.

Also we are now giving the PID tiers a proper defination, tier 0 is
readahead / cold cache, tier 1 is folios used at least once, and
LRU_REFS_WORKINGSET is just the tier 1 fallback line, tier 2 is
LRU_REFS_PROTECTED which is the proactive promotion wokringset
/candidates, consistent with each other.

> So, is the purpose to distinguish between max_seq and
> max_seq - 1 even when we know the folio will be placed
> on the active list?
>
> My concern is that after the folio is moved to the
> appropriate generation, LRU_REFS_PROTECTED or
> LRU_REFS_WORKINGSET may still remain set. Are these
> flags cleared somewhere?

They are never cleared, only PID protection / eviction protection can
reset it to LRU_REFS_WORKINGSET, which is the idea of FG. And it
worked really well checking the zipf read data and especially for
multi tier DB benchmarks. Folios in

And actually for the current mainline MGLRU impelmentation,
PG_workinset is also never cleared and it will pin a folio at tier 3
causing over protection very easily, we will reset it to tier 1 here
and with a softer PID protection factor, it seems much better in real
workloads.