Re: [PATCH v5 09/17] mm/mglru, gup: mark folios referenced via a fast helper

From: KunWu Chan

Date: Fri Nov 20 2026 - 03:20:58 EST


On Thu, Nov 19, 2026 at 5:03 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
>
> From: Kairui Song <kasong@xxxxxxxxxxx>
>
> Introduce folio_inc_lru_refs_fast(): raise refs from 0 to
> LRU_REFS_REFERENCED via cmpxchg, or just set PG_referenced for the
> classical LRU as before. No counter update or promotion is needed,
> nonzero refs are left unchanged.
>
> This helper only aims to be fast, it exists so the gup fast path can
> stay a fast path. The new helper doesn't account refs above
> LRU_REFS_REFERENCED but that's fine for now as GUP never used that
> or anything similar for CLRU either.
>
> Convert the gup fast-path sites. No behavior change for classical LRU.
> There isn't any regression for MGLRU either.
>
> Tested-by: Zhao Zhenzhuo <zhaozhengzhuo@xxxxxxxxxxxxx>
> Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> ---
> include/linux/mm_inline.h | 25 +++++++++++++++++++++++++
> mm/gup.c | 6 +++---
> 2 files changed, 28 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index b15c3618a1f2..3323a3c7336b 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
> @@ -483,6 +483,31 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
> }
> #endif /* CONFIG_LRU_GEN */
>
> +/**
> + * folio_inc_lru_refs_fast - Bump folio refs count without promotion.
> + * @folio: the folio
> + *
> + * Raise refs from 0 to LRU_REFS_REFERENCED and leave hotter folios
> + * untouched. For the classical LRU, a plain PG_referenced set.
> + */
> +static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
> +{
> + unsigned long new_flags, old_flags;
> +
> + if (!lru_gen_enabled()) {
> + folio_set_referenced(folio);
> + return;
> + }
> +
> + old_flags = READ_ONCE(*folio_flags(folio, 0));
> + do {
> + if (lru_get_refs_flags(old_flags))
> + break;
> + new_flags = old_flags;
> + lru_set_refs_flags(&new_flags, LRU_REFS_REFERENCED);
> + } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
> +}
> +
> static __always_inline
> void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
> {
> diff --git a/mm/gup.c b/mm/gup.c
> index 2b405cce2ca5..596e62954659 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -2911,7 +2911,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
> gup_put_folio(folio, 1, flags);
> goto pte_unmap;
> }
> - folio_set_referenced(folio);
> + folio_inc_lru_refs_fast(folio);
> pages[nr_pages] = page;
> nr_pages++;
> } while (ptep++, addr += PAGE_SIZE, addr != end);
> @@ -2980,7 +2980,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
>
> for (i = 0; i < nr_pages; i++)
> *(pages++) = page++;
> - folio_set_referenced(folio);
> + folio_inc_lru_refs_fast(folio);
> return nr_pages;
> }
>
> @@ -3022,7 +3022,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
>
> for (i = 0; i < nr_pages; i++)
> *(pages++) = page++;
> - folio_set_referenced(folio);
> + folio_inc_lru_refs_fast(folio);
> return nr_pages;
> }
>
>
> --
> 2.56.1
>
>

Reviewed-by: Kunwu Chan <kunwu.chan@xxxxxxxxx>

Thanks,
Kunwu