Re: [PATCH v4 15/17] mm/mglru: make folio_inc_lru_refs lruvec lockless
From: Barry Song
Date: Mon Nov 09 2026 - 04:51:32 EST
On Mon, Nov 9, 2026 at 4:08 PM Kairui Song <ryncsn@xxxxxxxxx> wrote:
>
> On Mon, Nov 9, 2026 at 5:48 AM Barry Song <baohua@xxxxxxxxxx> wrote:
> >
> > On Fri, Oct 16, 2026 at 2:16 AM Kairui Song via B4 Relay
> > <devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
> > >
> > > From: Kairui Song <kasong@xxxxxxxxxxx>
> > >
> > > The lruvec spinlock in folio_inc_lru_refs was only taken to keep
> > > concurrent aging from corrupting the size counters. This is no
> > > longer needed: the per-generation counters are atomic, and the
> > > active/inactive ABI counters are updated per folio and linearized by
> > > the CAS on the folio flags, so concurrent aging can no longer corrupt
> > > them. The same argument already allowed folio_reset_lru_refs() to
> > > drop the lock.
> > >
> > > Use folio_lruvec_live_get()/folio_lruvec_live_put() to hold the RCU
> > > read lock around the lruvec lookup, and drop the spinlock entirely.
> > >
> >
> > Yes, this is cool. We have two paths calling
> > folio_inc_lru_refs() or folio_inc_lru_refs_walk().
> >
> > 1. folio_mark_accessed() from syscall access
> > 2. walk_update_folio() from page table walker or look_around
> >
> > For the latter, we can't actually cross multiple
> > lruvecs/memcgs, because of get_pfn_folio() as shown
> > below:
> >
> > static struct folio *get_pfn_folio(unsigned long pfn, struct mem_cgroup *memcg,
> > struct pglist_data *pgdat)
> > {
> > ...
> > rcu_read_lock();
> > if (folio_memcg(folio) != memcg)
> > folio = NULL;
> > rcu_read_unlock();
> > ...
> > }
> >
> > Consider a common library file used by 100 memcgs, with
> > its folio residing in the root memcg.
> >
> > When walking the 99 non-root memcgs, even if the folio
> > has been accessed, we won't get a chance to call
> > folio_inc_lru_refs_walk() for it. As a result, this folio
> > may eventually reach the eviction rmap path. With a high
> > mapcount, folio_referenced() could become quite
> > expensive.
> >
> > That's what I was trying to point out here as well:
> >
> > https://lore.kernel.org/linux-mm/20261102213915.57805-1-baohua@xxxxxxxxxx/#t
>
> Hi Barry,
>
> Thanks for the review, I saw that too, very interesting approach! And
> I believe it will helpful.
>
> > For the former, you can get the lruvec with:
> >
> > lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
> >
> > I'm wondering whether we could eventually unify them
> > so that both support cross-memcg inc_refs.
>
> Definately, actually for that link you posted, combined with the code
> snip above, after FG, I think we can further simplify that, no need to
> check mem_cgroup_is_descendant, we just do:
>
> - folio_inc_lru_refs_walk: if the folio belongs to the lruvec being
> walked, the batch stays in walk->lruvec
> - folio_inc_lru_refs: if the folio is a remote one, per-folio, no batch
>
> non-descendant folio sharing across memcg are actually quite common
> e.g., libc and shared mapped memory, this could be more helpful? And
> it seems fast for the walker.
I agree. I mentioned mem_cgroup_is_descendant() because
folio_referenced() checks for descendants and ignores
folios belonging to non-descendant memcgs:
static bool invalid_folio_referenced_vma(struct vm_area_struct *vma, void *arg)
{
struct folio_referenced_arg *pra = arg;
struct mem_cgroup *memcg = pra->memcg;
...
/*
* If we are reclaiming on behalf of a cgroup, skip counting on behalf
* of references from different cgroups.
*/
if (memcg && !mm_match_cgroup(vma->vm_mm, memcg))
return true;
return false;
}
static inline bool mm_match_cgroup(struct mm_struct *mm,
const struct mem_cgroup *memcg)
{
struct mem_cgroup *task_memcg;
bool match = false;
rcu_read_lock();
task_memcg = mem_cgroup_from_task(rcu_dereference(mm->owner));
if (task_memcg)
match = mem_cgroup_is_descendant(task_memcg, memcg);
rcu_read_unlock();
return match;
}
I actually wonder whether this is really correct...
>
> folio_inc_lru_refs_walk follow the same rule as folio_inc_lru_refs
> (with MAPPED flag), just faster, skipping RCU and the speculative
> check afterwards as the walker owns the seq it is advancing, and
> caller handles the batch update of counter. While folio_inc_lru_refs
> only requires the caller to hold an reference to the folio it heavily
> relies on RCU / CAS conventions, not easy to batch things there, but
> it's still accetablily fast already (e.g. the fio read test showed
> almost no performance penalty).
Yes. One tricky aspect is that update_batch_size() is
designed for the current lruvec, not a remote one.
static void walk_update_folio(struct lru_gen_mm_walk *walk, struct
vm_area_struct *vma,
struct lruvec *lruvec, struct folio *folio, bool dirty)
{
...
if (walk) {
old_gen = folio_update_gen(folio, new_gen, &type, &vma->flags);
if (old_gen >= 0 && old_gen != new_gen)
update_batch_size(walk, folio, old_gen, new_gen, type);
} else if (lru_gen_set_refs(folio, &vma->flags)) {
...
}
}
>
> And I think the main reason MGLRU can't operate cross memcg was, lack
> of a protocol to stablilze the memcg's gen and holding a lock is too
> costly, now folio_inc_lru_refs handles that, and FG
> lives on top of that. So we can try that out right after the FG series.
Yes, absolutely. With MGLRU-FG, we lay the groundwork
for further cross-memcg optimizations.
On Android, we observed that folios with very large mapcounts
still reach the rmap path because walk and look-around cannot
promote them across memcgs. In more than 99% of cases,
folio_referenced() finds them PTE-young. As a result, these
folios incur the rmap overhead but are unlikely to be
reclaimed. It's essentially wasted rmap work.
With the cross-memcg optimizations, we are likely to
eliminate this overhead.
Thanks,
Barry