Re: [PATCH v5 15/17] mm/mglru: folio LRU refs based active/inactive number accounting
From: KunWu Chan
Date: Sat Nov 21 2026 - 08:48:05 EST
On Sat, Nov 21, 2026 at 2:59 AM Kairui Song <ryncsn@xxxxxxxxx> wrote:
>
> On Sat, Nov 21, 2026 at 12:30 AM KunWu Chan <kunwu.chan@xxxxxxxxx> wrote:
> >
> > On Fri, Nov 20, 2026 at 7:21 PM Kairui Song <ryncsn@xxxxxxxxx> wrote:
> > >
> > > On Fri, Nov 20, 2026 at 5:30 PM KunWu Chan <kunwu.chan@xxxxxxxxx> wrote:
> > > >
> > > > On Thu, Nov 19, 2026 at 5:04 AM Kairui Song via B4 Relay
> > > > <devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
> > > > >
> > > > > From: Kairui Song <kasong@xxxxxxxxxxx>
> > > > >
> > > > > Currently the active/inactive LRU sizes exported through /proc/vmstat
> > > > > and memory.stat are derived from the generation window: folios in the
> > > > > two newest generations are accounted as active, the rest as inactive.
> > > > > This causes several problems:
> > > > >
> > > > > - Unmapped file folios can stay in older gens despite repeated use,
> > > > > so active files can be under-reported, unlike classical LRU.
> > > > >
> > > > > - The reading is jumpy: on aging, the second-newest gen, previously
> > > > > active, becomes inactive all at once, causing a huge drop of active
> > > > > folios without any change in use.
> > > > >
> > > > > - The reading is reversed on swapless machines. When swapping is
> > > > > disabled or unavailable, MGLRU ages anon by moving the gens directly
> > > > > without walking or touching any folio, to reduce the overhead. So
> > > > > inactive anon folios can suddenly count as active, or vice versa,
> > > > > despite no change in their hotness.
> > > > >
> > > > > With frequency guided promotion, hotness is already tracked per folio by
> > > > > its referenced count, so we can do folio level accounting instead: a
> > > > > folio is active once its refs reach LRU_REFS_ACTIVATED (currently
> > > > > LRU_REFS_PROTECTED).
> > > > >
> > > > > On reaching this threshold, folios are accounted to the active part.
> > > > > On forced per-folio aging or PID protection, refs are capped at
> > > > > LRU_REFS_WORKINGSET, dropping active folios to the inactive part.
> > > > > Note that on swapless machines anon refs are not reset by aging (the
> > > > > shortcut skips the per-folio walk), but the counts no longer oscillate
> > > > > as the generation indices wrap.
> > > > >
> > > > > The accounting is now per folio and lruvec independent, so no size
> > > > > adjustment is needed when reparenting a dying memcg: the hierarchical
> > > > > stats keep the counts in the parent automatically.
> > > > >
> > > > > folio_migrate_lru_refs() also gains off-LRU VM_WARN_ON_ONCE checks and
> > > > > __folio_set_lru_refs() warns when the folio is still on an LRU list.
> > > > >
> > > > > The latter uses lru_get_gen_flags(), so that helper now has to move out of
> > > > > the CONFIG_LRU_GEN block and is built with the option off, where
> > > > > LRU_GEN_WIDTH is 0 and LRU_GEN_MASK is zero. Keep the decoded value in
> > > > > int so the subtraction of the one-off encoding cannot fold to ULONG_MAX
> > > > > before it happens: that is -Woverflow, and fatal with CONFIG_WERROR.
> > > > >
> > > > > Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> > > > > ---
> > > > > Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
> > > > > include/linux/mm_inline.h | 118 +++++++-------
> > > > > include/linux/mmzone.h | 23 ++-
> > > > > mm/folio.c | 39 +----
> > > > > mm/madvise.c | 2 +-
> > > > > mm/vmscan.c | 218 +++++++++++++++++---------
> > > > > mm/workingset.c | 2 +-
> > > > > 7 files changed, 230 insertions(+), 176 deletions(-)
> > > > >
> > > > > diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst
> > > > > index 9cb54b4ff5d9..808d2e97b3f3 100644
> > > > > --- a/Documentation/admin-guide/mm/multigen_lru.rst
> > > > > +++ b/Documentation/admin-guide/mm/multigen_lru.rst
> > > > > @@ -149,8 +149,8 @@ generations less than or equal to ``min_gen_nr``.
> > > > > ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
> > > > >
> > > > > ``min_gen_nr`` should be less than ``max_gen_nr-1``, since
> > > > > -``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
> > > > > -the active list) and therefore cannot be evicted. ``swappiness``
> > > > > +``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged and therefore
> > > > > +cannot be evicted. ``swappiness``
> > > > > overrides the default value in ``/proc/sys/vm/swappiness`` and the valid
> > > > > range is [0-200, max], with max being exclusively used for the reclamation
> > > > > of anonymous memory. ``nr_to_reclaim`` limits the number of pages to evict.
> > > > > diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> > > > > index 14aab2b1ed08..77f514143266 100644
> > > > > --- a/include/linux/mm_inline.h
> > > > > +++ b/include/linux/mm_inline.h
> > > > > @@ -105,6 +105,22 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
> > > > > return lru;
> > > > > }
> > > > >
> > > > > +/**
> > > > > + * lru_get_gen_flags - Return the LRU generation number from folio flags.
> > > > > + * @flags: folio flags
> > > > > + *
> > > > > + * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> > > > > + * -1 if the flags indicate the folio is off the list (e.g., isolated).
> > > > > + */
> > > > > +static inline int lru_get_gen_flags(unsigned long flags)
> > > > > +{
> > > > > + int gen = (int)((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
> > > > > +
> > > > > + /* The gen is stored offset by one, so zero means off-list */
> > > > > + VM_WARN_ON_ONCE(gen > MAX_NR_GENS);
> > > > > + return gen - 1;
> > > > > +}
> > > > > +
> > > > > #ifdef CONFIG_LRU_GEN
> > > > >
> > > > > static inline bool lru_gen_switching(void)
> > > > > @@ -168,22 +184,6 @@ static inline void lru_set_gen_flags(unsigned long *flags, int gen)
> > > > > *flags |= (gen + 1UL) << LRU_GEN_PGOFF;
> > > > > }
> > > > >
> > > > > -/**
> > > > > - * lru_get_gen_flags - Return the LRU generation number from folio flags.
> > > > > - * @flags: folio flags
> > > > > - *
> > > > > - * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> > > > > - * -1 if the flags indicate the folio is off the list (e.g., isolated).
> > > > > - */
> > > > > -static inline int lru_get_gen_flags(unsigned long flags)
> > > > > -{
> > > > > - int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> > > > > -
> > > > > - /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
> > > > > - VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
> > > > > - return gen;
> > > > > -}
> > > > > -
> > > > > static inline int folio_lru_gen(const struct folio *folio)
> > > > > {
> > > > > return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> > > > > @@ -247,12 +247,21 @@ static inline bool lru_refs_is_workingset(unsigned int refs)
> > > > > return refs >= LRU_REFS_WORKINGSET;
> > > > > }
> > > > >
> > > > > -static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> > > > > +/**
> > > > > + * __folio_set_lru_refs - Set a folio's LRU refs.
> > > > > + * @folio: the folio
> > > > > + * @refs: the new referenced count (0 .. LRU_REFS_MAX)
> > > > > + *
> > > > > + * Set the folio's LRU refs. The folio must be off the LRU list (e.g.,
> > > > > + * isolated), or use folio_inc_lru_refs or folio_reset_lru_refs instead.
> > > > > + */
> > > > > +static inline void __folio_set_lru_refs(struct folio *folio, unsigned int refs)
> > > > > {
> > > > > unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> > > > >
> > > > > do {
> > > > > new_flags = old_flags;
> > > > > + VM_WARN_ON_ONCE(lru_get_gen_flags(old_flags) != -1);
> > > > > lru_set_refs_flags(&new_flags, refs);
> > > > > if (new_flags == old_flags)
> > > > > break;
> > > > > @@ -261,12 +270,18 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> > > > >
> > > > > #ifdef CONFIG_LRU_GEN
> > > > > void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
> > > > > +bool folio_reset_lru_refs(struct folio *folio);
> > > > > #else
> > > > > static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> > > > > {
> > > > > /* Should not be called with !CONFIG_LRU_GEN */
> > > > > WARN_ON_ONCE(1);
> > > > > }
> > > > > +
> > > > > +static inline bool folio_reset_lru_refs(struct folio *folio)
> > > > > +{
> > > > > + return false;
> > > > > +}
> > > > > #endif
> > > > >
> > > > > /**
> > > > > @@ -280,10 +295,18 @@ static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> > > > > */
> > > > > static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
> > > > > {
> > > > > - folio_set_lru_refs(new, folio_lru_refs(old));
> > > > > + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(old), old);
> > > > > + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new), new);
> > > > > + __folio_set_lru_refs(new, folio_lru_refs(old));
> > > > > }
> > > > >
> > > > > #ifdef CONFIG_LRU_GEN
> > > > > +static inline bool lru_refs_is_active(unsigned int refs)
> > > > > +{
> > > > > + VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> > > > > + return refs >= LRU_REFS_ACTIVATED;
> > > > > +}
> > > > > +
> > > > > static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> > > > > {
> > > > > unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
> > > > > @@ -294,23 +317,12 @@ static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> > > > > gen == lru_gen_from_seq(min_seq + 1);
> > > > > }
> > > > >
> > > > > -static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
> > > > > -{
> > > > > - unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> > > > > -
> > > > > - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> > > > > -
> > > > > - /* see the comment on MIN_NR_GENS */
> > > > > - return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
> > > > > -}
> > > > > -
> > > > > static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
> > > > > int old_gen, int new_gen)
> > > > > {
> > > > > int type = folio_is_file_lru(folio);
> > > > > int zone = folio_zonenum(folio);
> > > > > int delta = folio_nr_pages(folio);
> > > > > - enum lru_list lru = type * LRU_INACTIVE_FILE;
> > > > > struct lru_gen_folio *lrugen = &lruvec->lrugen;
> > > > >
> > > > > VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
> > > > > @@ -321,19 +333,6 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
> > > > > atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
> > > > > if (new_gen >= 0)
> > > > > atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
> > > > > -
> > > > > - /* return now if not a promotion */
> > > > > - if (old_gen < 0 || new_gen < 0)
> > > > > - return;
> > > > > -
> > > > > - /* promotion */
> > > > > - if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
> > > > > - __update_lru_size(lruvec, lru, zone, -delta);
> > > > > - __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
> > > > > - }
> > > > > -
> > > > > - /* demotion requires isolation, e.g., lru_deactivate_fn() */
> > > > > - VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
> > > > > }
> > > > >
> > > > > static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
> > > > > @@ -376,7 +375,8 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > > > > {
> > > > > unsigned long seq;
> > > > > unsigned long flags;
> > > > > - int gen = folio_lru_gen(folio);
> > > > > + unsigned int refs;
> > > > > + int gen;
> > > > > int type = folio_is_file_lru(folio);
> > > > > int zone = folio_zonenum(folio);
> > > > > int delta = folio_nr_pages(folio);
> > > > > @@ -384,7 +384,7 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > > > > struct lru_gen_folio *lrugen = &lruvec->lrugen;
> > > > >
> > > > > BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
> > > > > - VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
> > > > > + VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
> > > > >
> > > > > if (folio_test_unevictable(folio) || !lrugen->enabled)
> > > > > return false;
> > > > > @@ -395,10 +395,12 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > > > > gen = lru_gen_from_seq(seq);
> > > > > flags = (gen + 1UL) << LRU_GEN_PGOFF;
> > > > > /* see the comment on MIN_NR_GENS about PG_active */
> > > > > - set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> > > > > + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> > > > > + /* use the refs from the atomic snapshot to avoid raced update */
> > > > > + refs = lru_get_refs_flags(flags);
> > > > >
> > > > > lru_gen_update_size(lruvec, folio, -1, gen);
> > > > > - if (lru_gen_is_active(lruvec, gen))
> > > > > + if (lru_refs_is_active(refs))
> > > > > lru += LRU_ACTIVE;
> > > > > __update_lru_size(lruvec, lru, zone, delta);
> > > > >
> > > > > @@ -425,24 +427,33 @@ static inline bool lru_gen_add_rejected_folio(struct lruvec *lruvec, struct foli
> > > > > static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
> > > > > {
> > > > > unsigned long flags;
> > > > > - int gen = folio_lru_gen(folio);
> > > > > + unsigned int refs;
> > > > > + int gen;
> > > > > int zone = folio_zonenum(folio);
> > > > > int delta = folio_nr_pages(folio);
> > > > > enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
> > > > > + unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> > > > >
> > > > > + /* Keep LRU_REFS_FLAGS: refs carry access history across isolation */
> > > > > + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
> > > > > + gen = lru_get_gen_flags(flags);
> > > > > + refs = lru_get_refs_flags(flags);
> > > > > if (gen < 0)
> > > > > return false;
> > > > >
> > > > > VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
> > > > > VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
> > > > >
> > > > > - /* for folio_migrate_flags() */
> > > > > - flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> > > > > - flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
> > > > > - gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> > > > > + /*
> > > > > + * See the comment in lru_gen_folio_seq. For migration, compaction,
> > > > > + * or any other isolation of a hot folio, try best to retain its gen
> > > > > + * info. Ideally we would keep the full gen info.
> > > > > + */
> > > > > + if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
> > > > > + folio_set_active(folio);
> > > > >
> > > > > lru_gen_update_size(lruvec, folio, gen, -1);
> > > > > - if (lru_gen_is_active(lruvec, gen))
> > > > > + if (lru_refs_is_active(refs))
> > > > > lru += LRU_ACTIVE;
> > > > > __update_lru_size(lruvec, lru, zone, -delta);
> > > > > list_del(&folio->lru);
> > > > > @@ -499,6 +510,7 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
> > > > > return;
> > > > > }
> > > > >
> > > > > + BUILD_BUG_ON(LRU_REFS_REFERENCED >= LRU_REFS_ACTIVATED);
> > > > > old_flags = READ_ONCE(*folio_flags(folio, 0));
> > > > > do {
> > > > > if (lru_get_refs_flags(old_flags))
> > > > > @@ -534,7 +546,7 @@ static inline bool folio_check_lru_refs(const struct folio *folio)
> > > > > static inline void folio_clear_lru_refs(struct folio *folio)
> > > > > {
> > > > > if (lru_gen_enabled())
> > > > > - folio_set_lru_refs(folio, 0);
> > > > > + folio_reset_lru_refs(folio);
> > > > > else
> > > > > folio_clear_referenced(folio);
> > > > > }
> > > >
> > > > Can clear_refs race with disabling MGLRU? Once the static key is
> > > > off, it can drop refs from 3 to 2 on a folio still on a gen list.
> > > > The drain then subtracts it from inactive, though it was counted
> > > > as active.
> > >
> > > Thanks for the very careful review!
> > >
> > > I think we are fine, folio_reset_lru_refs checks gen >= 0 before
> > > touching things, gen >= 0 means the folio is manged by MGLRU, right?
> > > The worst could happen is we clear PG_workingset for CLRU, which
> > > really doesn't matter, one metric off by one folio for a very short
> > > period and self healing, not a measurable thing even if you do that on
> > > purpose.
> >
> > Thanks, Kairui. The refs-based accounting makes sense to me.
> >
> > I meant the else branch in folio_clear_lru_refs(). The static key
> > can already be off while the folio is still on a gen list. If
> > clear_refs drops refs from 3 to 2 there, where do we subtract
> > those pages from the active count?
>
> Oh, that's a very good catch! I think we can do this (only changing
> the refs reset parts, a few other part need similar adaptation too,
> check gen <= 0 instead of lru_gen_enabled()):
>
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index a2442e49aca2..a97fa6b3b140 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -556,16 +556,14 @@ static inline bool folio_check_lru_refs(const
> struct folio *folio)
> static inline void folio_clear_lru_refs(struct folio *folio)
> {
> - if (lru_gen_enabled())
> + if (folio_lru_refs(folio))
> folio_reset_lru_refs(folio);
> - else
> - folio_clear_referenced(folio);
> }
>
> /*
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 4576a7a369e3..3c21463039da 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -965,7 +965,7 @@ void folio_inc_lru_refs(struct folio *folio,
> unsigned int flags)
> */
> bool folio_reset_lru_refs(struct folio *folio)
> {
> int type, gen;
> unsigned long seq, new_flags, old_flags;
> long nr_pages = folio_nr_pages(folio);
> @@ -980,7 +980,15 @@ bool folio_reset_lru_refs(struct folio *folio)
> gen = lru_get_gen_flags(old_flags);
> if (!refs)
> break;
> - lru_set_refs_flags(&new_flags, 0);
> + if (gen < 0)
> + refs = refs & ~LRU_REFS_REFERENCED;
> + else
> + refs = 0;
> + lru_set_refs_flags(&new_flags, new_refs);
> } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>
> if (gen >= 0) {
>
> ---
>
> In general we can check gen < 0 instead of lru_gen_enabled for a
> stable refs manipulation. Switching is a very rare thing, and I hope
> we can just delete that sometime later. Thanks for catching this!
Thanks, Kairui. Checking gen in the CAS loop covers the race I had
in mind. We just need to keep the old refs for accounting and still
clear PG_referenced with CONFIG_LRU_GEN=n.
Feel free to add my tag to 06/17 and 16/17, and to 15/17 with
those two details fixed.
Reviewed-by: Kunwu Chan <kunwu.chan@xxxxxxxxx>