Re: [PATCH v4 04/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG)

[Barry Song]

On Fri, Oct 16, 2026 at 2:15 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
>
> From: Kairui Song <kasong@xxxxxxxxxxx>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion.  Introduce a unified set of helpers built based
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still maps to a logarithmic tier just like before, but with
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX (7). When refs reaches a certain threshold, the folio is
> promoted proactively instead of waiting for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced: they
> become the low two bits of the refs count, with the higher bits
> provided by LRU_REFS_MASK.  This reduces MGLRU's original refs count
> bit usage by one, since only one extra bit is now needed to record a
> max referenced count of 7, and makes MGLRU's refs accounting more
> accurate.
>
> This doesn't affect classical LRU in any way, and it addresses several
> shortcomings of MGLRU's old tier-only cache protection model:
>
> - Long feedback loop: protection only activated after enough re-faults,
>   by which time the hot folios are already evicted, or no longer hot.
>
> - Limited tier resolution: once referenced count exceeded the bits
>   limit (8 previously), MGLRU could no longer distinguish hotter folios as
>   they are capped by the tier.  And what's worse, PG_workingset forces
>   a folio to stay on tier 3.
>
> - Eviction hotness reversion: because PID protection activates upon
>   eviction and always targets the LRU tail, it tends to protect cold tail
>   folios at the expense of hotter head folios.  Once the tail folios
>   consume the PID protection budget, head folios lose their protection.
>
> - Additionally, the PID cannot distinguish the access time of folios
>   that share the same reference count, and there are only 4 tiers.
>
> To achieve a frequency-guided framework, this commit introduces and reworks
> the LRU_REFS related helpers and definitions; most of the work is done by
> the helpers below, and their inline comments describe the details.
>
> - folio_inc_lru_refs(): Called on any cache access (folio_mark_accessed)
>   or page table access.  This is the main helper: it promotes folios
>   according to their access frequency.
>
>   A first access only defers eviction for a folio at the tail of its
>   LRU, i.e., in one of the MIN_NR_GENS oldest generations of its own
>   type (see lru_gen_at_tail()), and a workingset folio accessed through
>   a file descriptor is deferred from the same tail.  Those are the
>   generations the eviction scans next, so the deferral moves the folio
>   out of its way for one generation.
>
>   Promotion is lazy: the gen bits and size counters are updated eagerly,
>   while the list move is deferred to the next isolation.  NOTE: For now,
>   the lruvec lock is unconditionally taken on every on-list access to
>   block concurrent aging; a lockless fast path will be implemented very
>   soon in a following commit.
>
> - folio_inc_lru_refs_walk(): Used by the PTE walk path during aging and
>   by the batched look-around, both without the LRU lock, so max_seq may
>   be stale.  A first page table access only advances a folio at the tail
>   of its LRU, i.e., in one of the MIN_NR_GENS oldest generations of its
>   own type (second chance), a second access promotes it to the newest
>   generation; this also performs lazy promotion.
>
> - folio_inc_lru_refs_isolated(): Used by the rmap check before
>   eviction.  The folio is isolated and hence this doesn't perform
>   promotion by itself; the folio will be added back to the right gen
>   upon return according to the access frequency.  This path also has a
>   higher promotion bias.
>
> The eviction-time folio_inc_gen() still handles PID protection, but the
> protection ratio is softer than before since proactive promotion is
> mostly good enough already.  The PID gain factors are relaxed from
> (2:3) to (1:2) and the setpoint now spans the cumulative mass of the
> tiers below the candidate instead of tier 0 alone.  folio_inc_gen()
> caps refs at WORKINGSET so the folio retains enough history to stay
> above the cold tier.  Tier 1 is the fallback tier for PID, and tier 2
> is the fallback tier for frequency-guided promotion.  The forced
> protection for full-refs folios is removed, obsoleted by the proactive
> promotion.
>
> Refaults are now activated purely according to access frequency:
> the old fault bias applied in folio_add_lru() is simplified, since a
> folio's access history is a more consistent signal than the context of
> the faulting task.  Page table access is still considered a slightly
> stronger signal.
>
> Consequently the placement hint for newly added file folios also uses
> LRU_REFS_WORKINGSET instead of PG_referenced, so a folio accessed only
> once now starts in the oldest generation rather than the second oldest.
> The previous one-generation grace was an artifact of the fault bias which
> is well described in the cover letter of this series, and it depended
> on LRU drain timing.
>
> This also redefines PG_workingset and PG_referenced as the low two
> bits of the refs count, eliminating the old restriction where
> LRU_REFS_MASK was only valid when PG_referenced was set, and allows
> all paths to use the same encoding consistently.  Following this idea,
> a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET (2),
> matching the active/inactive LRU's definition and giving in-kernel
> consumers (PSI, readahead) consistent behavior on MGLRU, fixing the
> longstanding issue that these users don't work well with MGLRU.
>
> PG_workingset and PG_referenced are no longer independent flags under
> MGLRU.  Adjusting existing raw folio_test_*() callers to the new
> semantics is left as follow-ups.

So this is the main course, with the earlier ones as appetizers and
the later ones as desserts?
Overall, this dish looks very delicious to me.

I ran kernel build on my machine, the result is quite similar with[1].

W/o patch:

eal    1m49.094s
user    25m31.484s
sys     3m40.959s
pswpin: 662112
pswpout: 2504415
pgpgin: 4424336
pgpgout: 10355273
refault_file: 359626
refault_anon: 670724

W/ patch:

real 1m48.944s
user 25m30.987s
sys 3m34.605s
pswpin: 467952
pswpout: 2319080
pgpgin: 3481187
pgpgout: 9603332
swpout_zero: 602042
swpin_zero: 6250
refault_file: 310538
refault_anon: 474154

But it behaves in a much more sensible way.

I've had a preliminary look, but haven't reviewed it in detail yet.
I have a few comments.

[1] https://lore.kernel.org/linux-mm/20261016213115.70515-1-baohua@xxxxxxxxxx/

>
> Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> ---
>  include/linux/mm_inline.h |  96 ++++++++------
>  include/linux/mmzone.h    | 145 +++++++++++++++------
>  kernel/bounds.c           |   2 +-
>  mm/folio.c                |  50 +-------
>  mm/vmscan.c               | 320 +++++++++++++++++++++++++++++++---------------
>  mm/workingset.c           |  45 ++++---
>  6 files changed, 409 insertions(+), 249 deletions(-)
>
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index f52f02e8e5be..74d1b7a04ca1 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -144,12 +144,13 @@ static inline int lru_hist_from_seq(unsigned long seq)
>         return seq % NR_HIST_GENS;
>  }
>
> -static inline int lru_tier_from_refs(int refs, bool workingset)
> +static inline int lru_tier_from_refs(unsigned int refs)
>  {
> -       VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH));
> -
> -       /* see the comment on MAX_NR_TIERS */
> -       return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
> +       BUILD_BUG_ON(fls(LRU_REFS_MAX - 1) > MAX_NR_TIERS - 1);
> +       VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> +       if (refs < LRU_REFS_WORKINGSET)
> +               return 0;
> +       return fls(refs - 1);
>  }
>
>  /**
> @@ -187,21 +188,24 @@ static inline int lru_get_gen_flags(unsigned long flags)
>   * @flags: pointer to the folio flags
>   * @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
>   *
> - * For MGLRU, PG_referenced holds the first ref, and the extra bits hold the
> - * remaining refs. For classical LRU the extra bits are not used, so it can
> - * also be seen as the refs count never exceeds 1. In both cases, refs == 1
> - * means PG_referenced is set and the extra bits are zero, and refs == 0 means
> - * PG_referenced and the extra bits are all unset.
> + * For MGLRU, PG_referenced, PG_workingset are used as the lower two bits of
> + * refs counter, and extra bits hold the remaining higher bits. For classical
> + * LRU the extra bits are not used, and the two flags has no direct
> + * relationship with each other, but this helper can still be used to sync
> + * them. For both cases, refs == 0 means these two flags and the extra bits
> + * are all unset. And refs == 1 / 2 / 3 means PG_referenced and PG_workingset
> + * are set in an bit order way, which is more meaningful for MGLRU though.
>   */
>  static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
>  {
>         VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
> -       BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
> -
> +       BUILD_BUG_ON(LRU_REFS_MASK & (BIT(PG_referenced) | BIT(PG_workingset)));
>         *flags &= ~LRU_REFS_FLAGS;
> -       if (!refs)
> -               return;
> -       *flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
> +       if (refs & BIT(0))
> +               *flags |= BIT(PG_referenced);
> +       if (refs & BIT(1))
> +               *flags |= BIT(PG_workingset);
> +       *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
>  }
>
>  /**
> @@ -212,13 +216,13 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
>   */
>  static inline int lru_get_refs_flags(unsigned long flags)
>  {
> -       if (!(flags & BIT(PG_referenced)))
> -               return 0;
> -       /*
> -        * Return the total number of accesses including PG_referenced. Also see
> -        * the comment on LRU_REFS_FLAGS.
> -        */
> -       return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
> +       int refs;
> +
> +       /* Return the total number of accesses. See the comment above MAX_NR_TIERS. */
> +       refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
> +       refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
> +       refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
> +       return refs;
>  }

Could we use the following instead? Coming from a hardware register background,
this looks more like register encoding to me:

static inline int lru_get_refs_flags(unsigned long flags)
{
    int refs;

    refs  = !!(flags & BIT(PG_referenced));
    refs |= !!(flags & BIT(PG_workingset)) << 1;
    refs |= ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
    return refs;
}

>
>  static inline int folio_lru_refs(const struct folio *folio)
> @@ -236,14 +240,29 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
>         } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>  }
>
> +void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
> +
>  static inline int folio_lru_gen(const struct folio *folio)
>  {
>         return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
>  }
>
> +static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
> +{
> +       unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
> +
> +       /*
> +        * Consider a folio is at the tail of its LRU when it is within
> +        * MIN_NR_GENS of the oldest generation of its own type
> +        */
> +       BUILD_BUG_ON(MIN_NR_GENS != 2);
> +       return gen == lru_gen_from_seq(min_seq) ||
> +              gen == lru_gen_from_seq(min_seq + 1);
> +}

If only two generations remain, should both be considered hot?
If three generations remain, is only the oldest generation old?

Also, the name is a bit confusing, as "tail" usually refers
to a list. Could we rename it to one of the following?

lru_gen_is_old()
or
lru_gen_is_near_oldest()

> +
>  static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
>  {
> -       unsigned long max_seq = lruvec->lrugen.max_seq;
> +       unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
>
>         VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
>
> @@ -300,23 +319,24 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
>                                               bool reclaiming)
>  {
>         int gen;
> +       int refs = folio_lru_refs(folio);
>         int type = folio_is_file_lru(folio);
>         const struct lru_gen_folio *lrugen = &lruvec->lrugen;
>
>         /*
> -        * +-----------------------------------+-----------------------------------+
> -        * | Accessed through page tables and  | Accessed through file descriptors |
> -        * | promoted by folio_update_gen()    | and protected by folio_inc_gen()  |
> -        * +-----------------------------------+-----------------------------------+
> -        * | PG_active (set while isolated)    |                                   |
> -        * +-----------------+-----------------+-----------------+-----------------+
> -        * |  PG_workingset  |  PG_referenced  |  PG_workingset  |  LRU_REFS_FLAGS |
> -        * +-----------------------------------+-----------------------------------+
> -        * |<---------- MIN_NR_GENS ---------->|                                   |
> -        * |<---------------------------- MAX_NR_GENS ---------------------------->|
> +        * +------------------------------------------+------------------------------------------+
> +        * |     Accessed through page tables and     |     Accessed through file descriptors    |
> +        * | promoted by folio_inc_lru_refs_walk()    | protected by folio_inc_lru_refs/inc_gen  |
> +        * +------------------------------------------+------------------------------------------+
> +        * | PG_active (set at isolation or refault)  |                                          |
> +        * +--------------------+---------------------+--------------------+---------------------+
> +        * |     LRU_REFS_MAX   | LRU_REFS_WORKINGSET |    LRU_REFS_MAX    | LRU_REFS_WORKINGSET |
> +        * +------------------------------------------+------------------------------------------+
> +        * |<-------------- MIN_NR_GENS ------------->|                                          |
> +        * |<----------------------------------- MAX_NR_GENS ----------------------------------->|
>          */
>         if (folio_test_active(folio))
> -               gen = MIN_NR_GENS - folio_test_workingset(folio);
> +               gen = MIN_NR_GENS - (refs >= LRU_REFS_WORKINGSET);

Could we define the "workingset" variant earlier so we can reuse it here?

workingset = refs >= LRU_REFS_WORKINGSET;


>         else if (reclaiming)
>                 gen = MAX_NR_GENS;
>         else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
> @@ -324,7 +344,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
>                   (folio_test_dirty(folio) || folio_test_writeback(folio))))
>                 gen = MIN_NR_GENS;
>         else
> -               gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
> +               gen = MAX_NR_GENS - (refs >= LRU_REFS_WORKINGSET);

As above.

>
>         return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
>  }
> @@ -338,6 +358,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
>         int zone = folio_zonenum(folio);
>         struct lru_gen_folio *lrugen = &lruvec->lrugen;
>
> +       BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
>         VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
>
>         if (folio_test_unevictable(folio) || !lrugen->enabled)
> @@ -392,7 +413,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>   */
>  static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
>  {
> -       BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
>         folio_set_lru_refs(new, folio_lru_refs(old));
>  }
>  #else /* !CONFIG_LRU_GEN */
> @@ -422,6 +442,10 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
>         return false;
>  }
>
> +static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> +{
> +}
> +
>  static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
>  {
>         if (folio_test_referenced(old))
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 09ce82fa841a..b5a3769fdcab 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -453,57 +453,122 @@ enum lruvec_flags {
>  #define MAX_NR_GENS            4U
>
>  /*
> - * Each generation is divided into multiple tiers. A folio accessed N times
> - * through file descriptors is in tier order_base_2(N). A folio in the first
> - * tier (N=0,1) is marked by PG_referenced unless it was faulted in through page
> - * tables or read ahead. A folio in the last tier (MAX_NR_TIERS-1) is marked by
> - * PG_workingset. A folio in any other tier (1<N<5) between the first and last
> - * is marked by additional bits of LRU_REFS_WIDTH in folio->flags.
> + * Each generation is divided into multiple tiers. A folio's referenced
> + * count maps to a tier as shown below:
>   *
> - * In contrast to moving across generations which requires the LRU lock, moving
> - * across tiers only involves atomic operations on folio->flags and therefore
> - * has a negligible cost in the buffered access path. In the eviction path,
> - * comparisons of refaulted/(evicted+protected) from the first tier and the rest
> - * infer whether folios accessed multiple times through file descriptors are
> - * statistically hot and thus worth protecting.
> + * MGLRU (frequency guidance)
> + *  Refs  Tier  |- Refs: how many times (at least) a folio has been referenced.
> + *   0      0   |- Mostly cold pages, readahead, etc. [1]
> + *   1      0   |= LRU_REFS_REFERENCED: Used at least once. [2]
> + * -WORKINGSET-+|- Pages beyond are workingset and never fall below this floor. [3]
> + *   2      1<-+|= LRU_REFS_WORKINGSET: Classical workingset, accessed twice, protected. [4]
> + *   3      2   |- LRU_REFS_PROTECTED: Protected workingset, promoted pages capped at here. [5]
> + *   4      2   |
> + *   5      3   |- The tier here is MAX_NR_TIERS - 1
> + *   6      3   |
> + *   7      3   |= LRU_REFS_MAX: Promotion candidate. [6]
> + * -PROMOTION->-/
>   *
> - * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the
> - * number of categories of the active/inactive LRU when keeping track of
> - * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in
> - * folio->flags, masked by LRU_REFS_MASK.
> + * Ideally each tier holds folios of similar access patterns: lower tiers
> + * are less important and evicted faster.  A page's reference count and
> + * tier are capped when it changes generation, preventing it from
> + * dominating the new generation based on old-generation access history.
> + * Generation ordering already ensures a newer-gen page is hotter than an
> + * older-gen one regardless of tier.
> + *
> + * Refs tracks accesses from two sources: page table (lazily collected by
> + * the page table aging walk or rmap eviction lookup) and file descriptors
> + * (by folio_mark_accessed).  Page table accesses are weighted heavier
> + * because the accessed bit is sticky (undercounts repeated accesses),
> + * passively collected, and page faults are generally more important as
> + * userspace does not expect a memory access to block on reclaim.  Both
> + * access types increment refs by one; the result is capped at
> + * LRU_REFS_PROTECTED on promotion or deferral, or LRU_REFS_MAX otherwise.
> + *
> + * 1. Tier is fls(N-1) for N > 1, 0 otherwise.  Folios with zero
> + *    accesses (refs == 0) are generally cold, e.g. readahead folios.
> + *
> + *    Freshly allocated folios start with refs == 0; faulted and mapped
> + *    folios have their page table access bit set, so the first page table
> + *    access check always sets LRU_REFS_REFERENCED.  That is the second
> + *    chance described above MIN_NR_GENS: it moves the folio one
> + *    generation forward if it is at the tail of its LRU, i.e., in one
> + *    of the two oldest generations of its type.
> + *
> + * 2. Folios accessed once stay on tier 0: one-time usage does not
> + *    qualify for protection.  A second access advances the folio,
> + *    aligning with classical LRU's use-twice threshold.  A second page
> + *    table access promotes to the latest gen; file access only defers
> + *    eviction from the tail gens of its type.
> + *
> + * 3. Folios accessed at least twice are considered workingset.  This
> + *    mostly aligns with classical LRU: at least one I/O is saved by
> + *    keeping them in memory.  Folios at or above this level never fall
> + *    below tier 1 (the workingset floor), so tier 0 stays a clean tier
> + *    for cold cache while tier 1 serves as the fallback line for
> + *    actually reused or historically hot folios.
> + *
> + *    Folios refaulted through a page fault at refs 1 will enter the second
> + *    newest gen, so faulting will be protected better.
> + *
> + * 4. Starting from tier 1, PID protection sacrifices lower tiers to
> + *    protect higher tiers by comparing refault rates for long-term
> + *    accuracy, and caps higher refs to this value.  Since PID protection
> + *    bypasses page table lookup and clearing, when a further eviction
> + *    attempt occurs after PID loosens, the folio's page table access is
> + *    rechecked and a referenced folio is put back with refs capped to
> + *    this value as well.  This also gives folios a fair opportunity to be
> + *    promoted by file access again.
> + *
> + *    Folios refaulted through a page fault at tier 1 or above are activated
> + *    and enter the newest gen. Non fault page will enter second oldest gen,
> + *    driving aging and workingset shifting.
> + *
> + * 5. Pages beyond the ordinary workingset tier form new tiers for the
> + *    PID controller to protect differently.  Folios at or above this
> + *    level are capped at LRU_REFS_PROTECTED on promotion or deferral,
> + *    and at LRU_REFS_WORKINGSET under PID protection in the oldest
> + *    generation, where they represent a historical workingset.
> + *
> + * 6. Folios that reach LRU_REFS_MAX are advanced to the next generation
> + *    on further access, with refs capped to LRU_REFS_PROTECTED.  This
> + *    gives them a fair start for advancement to an even newer generation
> + *    while keeping hot folios distinguishable.
> + *
> + * Tiering uses PG_referenced and PG_workingset as the lower two bits,
> + * and the bits masked by LRU_REFS_MASK as the higher bits, so the refs
> + * count ranges from 0 to LRU_REFS_MAX.  A folio is on the workingset
> + * tier once accessed at least twice, which is more consistent with the
> + * classical LRU.
> + *
> + * A folio's referenced count never goes backwards except upon gen
> + * increase as described above, or when explicitly reset by
> + * lru_gen_clear_refs().  Refault of a reclaimed folio restores
> + * its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
> + * promotion.  Page table refaults of previous workingset folios send
> + * them to the latest gen, driving aging faster.
> + *
> + * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
> + * the number of categories of the active/inactive LRU.
>   */
>  #define MAX_NR_TIERS           4U
>  #define LRU_TIER_MIN           0U
>  #define LRU_TIER_MAX           (MAX_NR_TIERS - 1)
>
> +/* Access source flags for folio_inc_lru_refs() */
> +#define LRU_REF_MAPPED         0x1U
> +#define LRU_REF_EXEC           0x2U
> +
> +#define LRU_REFS_REFERENCED    0x1
> +#define LRU_REFS_WORKINGSET    0x2
> +#define LRU_REFS_PROTECTED     0x3
> +
>  #ifndef __GENERATING_BOUNDS_H
>
>  #define LRU_GEN_MASK           ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
>  #define LRU_REFS_MASK          ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
> -#define LRU_REFS_MAX           BIT(LRU_REFS_WIDTH)
> -
> -/*
> - * For folios accessed multiple times through file descriptors,
> - * lru_gen_inc_refs() sets additional bits of LRU_REFS_WIDTH in folio->flags
> - * after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its
> - * bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily
> - * promoted into the second oldest generation in the eviction path. And when
> - * folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that
> - * lru_gen_inc_refs() can start over. Note that for this case, LRU_REFS_MASK is
> - * only valid when PG_referenced is set.
> - *
> - * For folios accessed multiple times through page tables, folio_update_gen()
> - * from a page table walk or lru_gen_set_refs() from a rmap walk sets
> - * PG_referenced after the accessed bit is cleared for the first time.
> - * Thereafter, those two paths set PG_workingset and promote folios to the
> - * youngest generation. Like folio_inc_gen(), folio_update_gen() also clears
> - * PG_referenced. Note that for this case, LRU_REFS_MASK is not used.
> - *
> - * For both cases above, after PG_workingset is set on a folio, it remains until
> - * this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It
> - * can be set again if lru_gen_test_recent() returns true upon a refault.
> - */
> -#define LRU_REFS_FLAGS         (LRU_REFS_MASK | BIT(PG_referenced))
> +#define LRU_REFS_FLAGS         (LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_workingset))
> +#define LRU_REFS_MAX           (BIT(LRU_REFS_WIDTH + 2) - 1)
>
>  struct lruvec;
>  struct page_vma_mapped_walk;
> diff --git a/kernel/bounds.c b/kernel/bounds.c
> index 02b619eb6106..06a034713b5d 100644
> --- a/kernel/bounds.c
> +++ b/kernel/bounds.c
> @@ -25,7 +25,7 @@ int main(void)
>         DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
>  #ifdef CONFIG_LRU_GEN
>         DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1));
> -       DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2);
> +       DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 3);
>  #else
>         DEFINE(LRU_GEN_WIDTH, 0);
>         DEFINE(__LRU_REFS_WIDTH, 0);
> diff --git a/mm/folio.c b/mm/folio.c
> index 35e242b48870..e200efab814f 100644
> --- a/mm/folio.c
> +++ b/mm/folio.c
> @@ -273,7 +273,6 @@ static void lru_activate(struct lruvec *lruvec, struct folio *folio)
>         if (folio_test_active(folio) || folio_test_unevictable(folio))
>                 return;
>
> -
>         lruvec_del_folio(lruvec, folio);
>         folio_set_active(folio);
>         lruvec_add_folio(lruvec, folio);
> @@ -352,32 +351,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
>
>  #ifdef CONFIG_LRU_GEN
>
> -static void lru_gen_inc_refs(struct folio *folio)
> -{
> -       unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> -       int refs;
> -
> -       if (folio_test_unevictable(folio))
> -               return;
> -
> -       /* see the comment on LRU_REFS_FLAGS */
> -       if (!folio_lru_refs(folio)) {
> -               folio_set_lru_refs(folio, 1);
> -               return;
> -       }
> -
> -       do {
> -               new_flags = old_flags;
> -               refs = lru_get_refs_flags(old_flags);
> -               if (refs == LRU_REFS_MAX) {
> -                       if (!folio_test_workingset(folio))
> -                               folio_set_workingset(folio);
> -                       return;
> -               }
> -               lru_set_refs_flags(&new_flags, refs + 1);
> -       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> -}
> -
>  static bool lru_gen_clear_refs(struct folio *folio)
>  {
>         int gen = folio_lru_gen(folio);
> @@ -388,7 +361,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
>                 return true;
>
>         folio_set_lru_refs(folio, 0);
> -       folio_clear_workingset(folio);
>
>         rcu_read_lock();
>         seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
> @@ -399,10 +371,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
>
>  #else /* !CONFIG_LRU_GEN */
>
> -static void lru_gen_inc_refs(struct folio *folio)
> -{
> -}
> -
>  static bool lru_gen_clear_refs(struct folio *folio)
>  {
>         return false;
> @@ -428,7 +396,8 @@ void folio_mark_accessed(struct folio *folio)
>         if (folio_test_dropbehind(folio))
>                 return;
>         if (lru_gen_enabled()) {
> -               lru_gen_inc_refs(folio);
> +               if (!folio_test_unevictable(folio))
> +                       folio_inc_lru_refs(folio, 0);
>                 return;
>         }
>
> @@ -474,21 +443,6 @@ void folio_add_lru(struct folio *folio)
>                         folio_test_unevictable(folio), folio);
>         VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
>
> -       /*
> -        * For refaulted workingset folios, set PG_active so they
> -        * can be added to active generations.
> -        * For prefaulted file folios, folio_mark_accessed() sets
> -        * PG_referenced so lru_gen_folio_seq() places them into
> -        * the second oldest generation.
> -        */
> -       if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
> -           lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
> -               if (folio_test_workingset(folio))
> -                       folio_set_active(folio);
> -               else if (!folio_test_referenced(folio))
> -                       folio_mark_accessed(folio);
> -       }
> -
>         folio_batch_add_and_move(folio, lru_add);
>  }
>  EXPORT_SYMBOL(folio_add_lru);
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index cab3ba14d20c..d0f0e5a25d87 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -931,38 +931,186 @@ enum folio_references {
>  };
>
>  #ifdef CONFIG_LRU_GEN
> +/******************************************************************************
> + *                     Referenced count feedback
> + ******************************************************************************/
> +
>  /*
> - * Only used on a mapped folio in the eviction (rmap walk) path, where promotion
> - * needs to be done by taking the folio off the LRU list and then adding it back
> - * with PG_active set. In contrast, the aging (page table walk) path uses
> - * folio_update_gen().
> + * The folio_inc_lru_refs{_*} helpers below collect the referenced info
> + * (hotness) from other parts, including the page table walker, the rmap walk
> + * upon eviction, the rmap lookaround, and file descriptors
> + * (folio_mark_accessed).
> + *
> + * Page table accesses use refs from either source. The first reference
> + * advances tail-gen folios by one generation; a second promotes to the
> + * newest. Executable file folios promote on their first access to avoid
> + * I/O thrashing. Isolated rmap accesses use evict_folios() for putback.
> + *
> + * Promotion changes folio->flags; sort_folio() or inc_min_seq() moves the
> + * folio later. Young PTEs found during unmapping use folio_mark_accessed()
> + * and follow the file descriptor rules below.
> + *
> + * File descriptor accesses do not promote.  They only defer eviction from
> + * the tail gens of their type, and only once the folio is a workingset
> + * folio (LRU_REFS_WORKINGSET), leaving the rest to PID protection.  Page
> + * table accesses are treated more generously because the accessed bit is
> + * sticky (it under-counts repeated accesses) and because a page fault is
> + * more costly than file descriptor I/O.
> + *
> + * PID protection operates on tier > 0 folios.  The one proactive promotion
> + * outside of it and the page table path is the overflow case where the
> + * referenced count exceeds LRU_REFS_MAX, which means the folio is hotter
> + * than everything else in its generation.
> + *
> + * Whenever a folio changes generation here its referenced count is capped at
> + * LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
> + * of its old-generation access history.  PID protection (folio_inc_gen) caps
> + * at LRU_REFS_WORKINGSET independently.
>   */
> -static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
> -{
> -       /* see the comment on LRU_REFS_FLAGS */
> -       if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
> -               /* Activate file-backed executable folios after first usage. */
> -               if (is_exec_file_folio(folio, vma_flags)) {
> -                       folio_set_workingset(folio);
> -                       folio_set_lru_refs(folio, 0);
> -                       return true;
> +
> +/*
> + * Update the folio's lru refs indicator. The caller doesn't need to hold
> + * the folio lock, isolate the folio, or hold the lruvec lock. Used by both
> + * cache access (flags == 0) and page table access (LRU_REF_MAPPED,
> + * optionally with LRU_REF_EXEC).
> + */
> +void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
> +{
> +       int max_gen;
> +       int type, refs, old_gen, gen;
> +       unsigned long new_flags, old_flags, max_seq;
> +       struct lru_gen_folio *lrugen;
> +       struct lruvec *lruvec = NULL;
> +
> +       type = folio_is_file_lru(folio);
> +       old_flags = READ_ONCE(*folio_flags(folio, 0));
> +       do {
> +               new_flags = old_flags;
> +               old_gen = lru_get_gen_flags(old_flags);
> +               refs = lru_get_refs_flags(old_flags) + 1;
> +               gen = old_gen;
> +               if (old_gen < 0)
> +                       goto out;
> +               /*
> +                * Lock the lruvec if the folio is on-list. We are already
> +                * doing lazy promotion so in theory we don't need this,
> +                * but for now, concurrent aging would still corrupt the
> +                * size counters.  This is a temporary limitation and
> +                * will be lifted very soon, so the lock here is not a
> +                * performance concern.
> +                */
> +               if (!lruvec) {
> +                       lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
> +                       lrugen = &lruvec->lrugen;
>                 }
> +               max_seq = READ_ONCE(lrugen->max_seq);
> +               max_gen = lru_gen_from_seq(max_seq);
> +               if (old_gen == max_gen)
> +                       goto out;
>
> -               folio_set_lru_refs(folio, 1);
> -               return false;
> -       }
> +               if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
> +                       /* Promote second page table access or executable */
> +                       if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
> +                               gen = max_gen;

This looks a bit odd to me. Could we handle LRU_REF_EXEC in a
separate branch instead of mixing it with refs > LRU_REFS_REFERENCED?

> +                       /* First access only defers eviction of a tail gen */
> +                       else if (lru_gen_at_tail(lruvec, type, old_gen))
> +                               gen = (old_gen + 1) % MAX_NR_GENS;

The logic makes a lot of sense to me. It should be able to
differentiate readahead folios from those that are actually mapped
and likely to be accessed more than once.

If we allow two generations to remain, this could be a problem
because old_gen + 1 would overflow. Maybe add a
VM_WARN_ON() to guard against this case?


> +                       refs = min(refs, LRU_REFS_PROTECTED);
> +               } else if (refs > LRU_REFS_MAX) {
> +                       /* LRU refs counting overflow, bump the gen */
> +                       gen = (old_gen + 1) % MAX_NR_GENS;
> +                       refs = LRU_REFS_PROTECTED;
> +               } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {

Sometimes you use >= LRU_REFS_WORKINGSET, while elsewhere
you use refs > LRU_REFS_REFERENCED. Can we unify these
checks or define a workingset variant?

> +                       /* Defer eviction of just accessed workingset */
> +                       gen = (old_gen + 1) % MAX_NR_GENS;
> +                       refs = min(refs, LRU_REFS_PROTECTED);
> +               }
> +out:
> +               refs = min(refs, LRU_REFS_MAX);
> +               lru_set_refs_flags(&new_flags, refs);
> +               if (gen != old_gen)
> +                       lru_set_gen_flags(&new_flags, gen);
> +               if (new_flags == old_flags)
> +                       break;
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
>
> -       /* Promote on second access */
> -       if (folio_lru_refs(folio) > 1) {
> -               folio_set_workingset(folio);
> -               folio_set_lru_refs(folio, 0);
> -       } else {
> -               folio_mark_accessed(folio);
> -       }
> -       return true;
> +       if (gen != old_gen)
> +               lru_gen_update_size(lruvec, folio, old_gen, gen);
> +       if (lruvec)
> +               lruvec_unlock_irq(lruvec);
> +}
> +
> +/*
> + * Update the folio's lru refs indicator during a page table walk or the
> + * look-around. max_seq can be stale as neither holds the LRU lock.
> + *
> + * Returns the old generation and stores the new generation in @new_gen if
> + * the folio is on the LRU and not in the newest generation, or -1 otherwise.
> + */
> +static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
> +                                  const vma_flags_t *vma_flags,
> +                                  int *new_gen, int *type)
> +{
> +       unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +       int refs, gen, max_gen, ret;
> +       unsigned long max_seq;
> +
> +       do {
> +               max_seq = READ_ONCE(lruvec->lrugen.max_seq);
> +               max_gen = lru_gen_from_seq(max_seq);
> +               gen = lru_get_gen_flags(old_flags);
> +               refs = lru_get_refs_flags(old_flags) + 1;
> +               *type = folio_flags_is_file_lru(&old_flags);
> +               new_flags = old_flags;
> +
> +               if (gen >= 0 && gen != max_gen) {
> +                       ret = gen;
> +                       /* Promote second page table access or executable */
> +                       if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
> +                               *new_gen = max_gen;
> +                       /* First access only defers eviction of a tail gen */
> +                       else if (lru_gen_at_tail(lruvec, *type, gen))
> +                               *new_gen = (gen + 1) % MAX_NR_GENS;
> +                       else
> +                               *new_gen = gen;
> +                       lru_set_gen_flags(&new_flags, *new_gen);
> +                       lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
> +               } else {
> +                       ret = -1;
> +                       lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_MAX));
> +               }
> +               if (new_flags == old_flags)
> +                       break;
> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +
> +       return ret;
> +}
> +
> +/*
> + * Update the lru refs indicator of an isolated folio, only used on
> + * mapped folios upon the final eviction.
> + *
> + * Increments the refs count (capped at LRU_REFS_PROTECTED, evict_folios()
> + * then caps it at LRU_REFS_WORKINGSET).  Returns true if the caller should
> + * activate the folio (second access or executable), false to put it back
> + * for a second chance.
> + */
> +static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
> +{
> +       unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
> +       int refs;
> +
> +       do {
> +               new_flags = old_flags;
> +               refs = lru_get_refs_flags(old_flags) + 1;
> +               lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));

I'm having a hard time understanding why an activated folio
still keeps LRU_REFS_PROTECTED, since it will be promoted
to the active list anyway :-)

Later, I see that you set it to LRU_REFS_WORKINGSET in
evict_folios():

folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));

So, is the purpose to distinguish between max_seq and
max_seq - 1 even when we know the folio will be placed
on the active list?

My concern is that after the folio is moved to the
appropriate generation, LRU_REFS_PROTECTED or
LRU_REFS_WORKINGSET may still remain set. Are these
flags cleared somewhere?


> +       } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +
> +       /* Promote second page table access or executable */
> +       return refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags);
>  }
>  #else
> -static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
> +static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
>  {
>         return false;
>  }
> @@ -997,7 +1145,8 @@ static enum folio_references folio_check_references(struct folio *folio,
>                 if (!referenced_ptes)
>                         return FOLIOREF_RECLAIM;
>
> -               return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
> +               return folio_inc_lru_refs_isolated(folio, &vma_flags) ?
> +                      FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
>         }
>
>         referenced_folio = folio_test_clear_referenced(folio);

Thanks
Barry