Re: [PATCH v5 15/17] mm/mglru: folio LRU refs based active/inactive number accounting
From: Kairui Song
Date: Fri Nov 20 2026 - 06:22:02 EST
On Fri, Nov 20, 2026 at 5:30 PM KunWu Chan <kunwu.chan@xxxxxxxxx> wrote:
>
> On Thu, Nov 19, 2026 at 5:04 AM Kairui Song via B4 Relay
> <devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
> >
> > From: Kairui Song <kasong@xxxxxxxxxxx>
> >
> > Currently the active/inactive LRU sizes exported through /proc/vmstat
> > and memory.stat are derived from the generation window: folios in the
> > two newest generations are accounted as active, the rest as inactive.
> > This causes several problems:
> >
> > - Unmapped file folios can stay in older gens despite repeated use,
> > so active files can be under-reported, unlike classical LRU.
> >
> > - The reading is jumpy: on aging, the second-newest gen, previously
> > active, becomes inactive all at once, causing a huge drop of active
> > folios without any change in use.
> >
> > - The reading is reversed on swapless machines. When swapping is
> > disabled or unavailable, MGLRU ages anon by moving the gens directly
> > without walking or touching any folio, to reduce the overhead. So
> > inactive anon folios can suddenly count as active, or vice versa,
> > despite no change in their hotness.
> >
> > With frequency guided promotion, hotness is already tracked per folio by
> > its referenced count, so we can do folio level accounting instead: a
> > folio is active once its refs reach LRU_REFS_ACTIVATED (currently
> > LRU_REFS_PROTECTED).
> >
> > On reaching this threshold, folios are accounted to the active part.
> > On forced per-folio aging or PID protection, refs are capped at
> > LRU_REFS_WORKINGSET, dropping active folios to the inactive part.
> > Note that on swapless machines anon refs are not reset by aging (the
> > shortcut skips the per-folio walk), but the counts no longer oscillate
> > as the generation indices wrap.
> >
> > The accounting is now per folio and lruvec independent, so no size
> > adjustment is needed when reparenting a dying memcg: the hierarchical
> > stats keep the counts in the parent automatically.
> >
> > folio_migrate_lru_refs() also gains off-LRU VM_WARN_ON_ONCE checks and
> > __folio_set_lru_refs() warns when the folio is still on an LRU list.
> >
> > The latter uses lru_get_gen_flags(), so that helper now has to move out of
> > the CONFIG_LRU_GEN block and is built with the option off, where
> > LRU_GEN_WIDTH is 0 and LRU_GEN_MASK is zero. Keep the decoded value in
> > int so the subtraction of the one-off encoding cannot fold to ULONG_MAX
> > before it happens: that is -Woverflow, and fatal with CONFIG_WERROR.
> >
> > Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> > ---
> > Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
> > include/linux/mm_inline.h | 118 +++++++-------
> > include/linux/mmzone.h | 23 ++-
> > mm/folio.c | 39 +----
> > mm/madvise.c | 2 +-
> > mm/vmscan.c | 218 +++++++++++++++++---------
> > mm/workingset.c | 2 +-
> > 7 files changed, 230 insertions(+), 176 deletions(-)
> >
> > diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst
> > index 9cb54b4ff5d9..808d2e97b3f3 100644
> > --- a/Documentation/admin-guide/mm/multigen_lru.rst
> > +++ b/Documentation/admin-guide/mm/multigen_lru.rst
> > @@ -149,8 +149,8 @@ generations less than or equal to ``min_gen_nr``.
> > ``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
> >
> > ``min_gen_nr`` should be less than ``max_gen_nr-1``, since
> > -``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
> > -the active list) and therefore cannot be evicted. ``swappiness``
> > +``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged and therefore
> > +cannot be evicted. ``swappiness``
> > overrides the default value in ``/proc/sys/vm/swappiness`` and the valid
> > range is [0-200, max], with max being exclusively used for the reclamation
> > of anonymous memory. ``nr_to_reclaim`` limits the number of pages to evict.
> > diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> > index 14aab2b1ed08..77f514143266 100644
> > --- a/include/linux/mm_inline.h
> > +++ b/include/linux/mm_inline.h
> > @@ -105,6 +105,22 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
> > return lru;
> > }
> >
> > +/**
> > + * lru_get_gen_flags - Return the LRU generation number from folio flags.
> > + * @flags: folio flags
> > + *
> > + * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> > + * -1 if the flags indicate the folio is off the list (e.g., isolated).
> > + */
> > +static inline int lru_get_gen_flags(unsigned long flags)
> > +{
> > + int gen = (int)((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF);
> > +
> > + /* The gen is stored offset by one, so zero means off-list */
> > + VM_WARN_ON_ONCE(gen > MAX_NR_GENS);
> > + return gen - 1;
> > +}
> > +
> > #ifdef CONFIG_LRU_GEN
> >
> > static inline bool lru_gen_switching(void)
> > @@ -168,22 +184,6 @@ static inline void lru_set_gen_flags(unsigned long *flags, int gen)
> > *flags |= (gen + 1UL) << LRU_GEN_PGOFF;
> > }
> >
> > -/**
> > - * lru_get_gen_flags - Return the LRU generation number from folio flags.
> > - * @flags: folio flags
> > - *
> > - * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
> > - * -1 if the flags indicate the folio is off the list (e.g., isolated).
> > - */
> > -static inline int lru_get_gen_flags(unsigned long flags)
> > -{
> > - int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> > -
> > - /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
> > - VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
> > - return gen;
> > -}
> > -
> > static inline int folio_lru_gen(const struct folio *folio)
> > {
> > return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
> > @@ -247,12 +247,21 @@ static inline bool lru_refs_is_workingset(unsigned int refs)
> > return refs >= LRU_REFS_WORKINGSET;
> > }
> >
> > -static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> > +/**
> > + * __folio_set_lru_refs - Set a folio's LRU refs.
> > + * @folio: the folio
> > + * @refs: the new referenced count (0 .. LRU_REFS_MAX)
> > + *
> > + * Set the folio's LRU refs. The folio must be off the LRU list (e.g.,
> > + * isolated), or use folio_inc_lru_refs or folio_reset_lru_refs instead.
> > + */
> > +static inline void __folio_set_lru_refs(struct folio *folio, unsigned int refs)
> > {
> > unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> >
> > do {
> > new_flags = old_flags;
> > + VM_WARN_ON_ONCE(lru_get_gen_flags(old_flags) != -1);
> > lru_set_refs_flags(&new_flags, refs);
> > if (new_flags == old_flags)
> > break;
> > @@ -261,12 +270,18 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
> >
> > #ifdef CONFIG_LRU_GEN
> > void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
> > +bool folio_reset_lru_refs(struct folio *folio);
> > #else
> > static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> > {
> > /* Should not be called with !CONFIG_LRU_GEN */
> > WARN_ON_ONCE(1);
> > }
> > +
> > +static inline bool folio_reset_lru_refs(struct folio *folio)
> > +{
> > + return false;
> > +}
> > #endif
> >
> > /**
> > @@ -280,10 +295,18 @@ static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> > */
> > static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
> > {
> > - folio_set_lru_refs(new, folio_lru_refs(old));
> > + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(old), old);
> > + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new), new);
> > + __folio_set_lru_refs(new, folio_lru_refs(old));
> > }
> >
> > #ifdef CONFIG_LRU_GEN
> > +static inline bool lru_refs_is_active(unsigned int refs)
> > +{
> > + VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> > + return refs >= LRU_REFS_ACTIVATED;
> > +}
> > +
> > static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> > {
> > unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
> > @@ -294,23 +317,12 @@ static inline bool lru_gen_is_old(struct lruvec *lruvec, int type, int gen)
> > gen == lru_gen_from_seq(min_seq + 1);
> > }
> >
> > -static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
> > -{
> > - unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> > -
> > - VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
> > -
> > - /* see the comment on MIN_NR_GENS */
> > - return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
> > -}
> > -
> > static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
> > int old_gen, int new_gen)
> > {
> > int type = folio_is_file_lru(folio);
> > int zone = folio_zonenum(folio);
> > int delta = folio_nr_pages(folio);
> > - enum lru_list lru = type * LRU_INACTIVE_FILE;
> > struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >
> > VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
> > @@ -321,19 +333,6 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
> > atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
> > if (new_gen >= 0)
> > atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
> > -
> > - /* return now if not a promotion */
> > - if (old_gen < 0 || new_gen < 0)
> > - return;
> > -
> > - /* promotion */
> > - if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
> > - __update_lru_size(lruvec, lru, zone, -delta);
> > - __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
> > - }
> > -
> > - /* demotion requires isolation, e.g., lru_deactivate_fn() */
> > - VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
> > }
> >
> > static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
> > @@ -376,7 +375,8 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > {
> > unsigned long seq;
> > unsigned long flags;
> > - int gen = folio_lru_gen(folio);
> > + unsigned int refs;
> > + int gen;
> > int type = folio_is_file_lru(folio);
> > int zone = folio_zonenum(folio);
> > int delta = folio_nr_pages(folio);
> > @@ -384,7 +384,7 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >
> > BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
> > - VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
> > + VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
> >
> > if (folio_test_unevictable(folio) || !lrugen->enabled)
> > return false;
> > @@ -395,10 +395,12 @@ static inline bool __lru_gen_add_folio(struct lruvec *lruvec, struct folio *foli
> > gen = lru_gen_from_seq(seq);
> > flags = (gen + 1UL) << LRU_GEN_PGOFF;
> > /* see the comment on MIN_NR_GENS about PG_active */
> > - set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> > + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> > + /* use the refs from the atomic snapshot to avoid raced update */
> > + refs = lru_get_refs_flags(flags);
> >
> > lru_gen_update_size(lruvec, folio, -1, gen);
> > - if (lru_gen_is_active(lruvec, gen))
> > + if (lru_refs_is_active(refs))
> > lru += LRU_ACTIVE;
> > __update_lru_size(lruvec, lru, zone, delta);
> >
> > @@ -425,24 +427,33 @@ static inline bool lru_gen_add_rejected_folio(struct lruvec *lruvec, struct foli
> > static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
> > {
> > unsigned long flags;
> > - int gen = folio_lru_gen(folio);
> > + unsigned int refs;
> > + int gen;
> > int zone = folio_zonenum(folio);
> > int delta = folio_nr_pages(folio);
> > enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
> > + unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> >
> > + /* Keep LRU_REFS_FLAGS: refs carry access history across isolation */
> > + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
> > + gen = lru_get_gen_flags(flags);
> > + refs = lru_get_refs_flags(flags);
> > if (gen < 0)
> > return false;
> >
> > VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
> > VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
> >
> > - /* for folio_migrate_flags() */
> > - flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
> > - flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
> > - gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
> > + /*
> > + * See the comment in lru_gen_folio_seq. For migration, compaction,
> > + * or any other isolation of a hot folio, try best to retain its gen
> > + * info. Ideally we would keep the full gen info.
> > + */
> > + if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
> > + folio_set_active(folio);
> >
> > lru_gen_update_size(lruvec, folio, gen, -1);
> > - if (lru_gen_is_active(lruvec, gen))
> > + if (lru_refs_is_active(refs))
> > lru += LRU_ACTIVE;
> > __update_lru_size(lruvec, lru, zone, -delta);
> > list_del(&folio->lru);
> > @@ -499,6 +510,7 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
> > return;
> > }
> >
> > + BUILD_BUG_ON(LRU_REFS_REFERENCED >= LRU_REFS_ACTIVATED);
> > old_flags = READ_ONCE(*folio_flags(folio, 0));
> > do {
> > if (lru_get_refs_flags(old_flags))
> > @@ -534,7 +546,7 @@ static inline bool folio_check_lru_refs(const struct folio *folio)
> > static inline void folio_clear_lru_refs(struct folio *folio)
> > {
> > if (lru_gen_enabled())
> > - folio_set_lru_refs(folio, 0);
> > + folio_reset_lru_refs(folio);
> > else
> > folio_clear_referenced(folio);
> > }
>
> Can clear_refs race with disabling MGLRU? Once the static key is
> off, it can drop refs from 3 to 2 on a folio still on a gen list.
> The drain then subtracts it from inactive, though it was counted
> as active.
Thanks for the very careful review!
I think we are fine, folio_reset_lru_refs checks gen >= 0 before
touching things, gen >= 0 means the folio is manged by MGLRU, right?
The worst could happen is we clear PG_workingset for CLRU, which
really doesn't matter, one metric off by one folio for a very short
period and self healing, not a measurable thing even if you do that on
purpose.