Re: [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up

[Kairui Song]

On Tue, Oct 20, 2026 at 5:29 PM Hui Zhu <hui.zhu@xxxxxxxxx> wrote:
> 2026/10/17 02:07, Kairui Song:
> > On Thu, Oct 15, 2026 at 4:02 PM Hui Zhu <hui.zhu@xxxxxxxxx> wrote:

...

> > Hmm, I know this could be noise but it looks like a regression?
> >
> >>       mm_vmscan_throttled  0               0 (all runs)
> >>       nr_isolated peak     0               0
> >>
> >>     (*) the aggregate is dominated by a preload-end residency lottery
> >>     (which shards' datasets happen to stay hot) and swings by up to 26%
> >>     between runs of the SAME kernel; with 4 runs per kernel the
> >>     difference between the means is within that noise (Welch t=1.95,
> >>     p~0.11). The throttle never fired in any patched run (the tracepoint
> >>     covers preload as well), so both kernels execute the same reclaim
> >>     path here, and the kernel-side counters agree across all 8 runs:
> >>     steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs
> >>     document the no-impact side: the check is dormant unless isolation
> >>     actually piles up.
> >>
> >> Signed-off-by: Hui Zhu <zhuhui@xxxxxxxxxx>
> > Hi Hui
> >
> > Looks like a good idea in general.
> >> ---
> >> Changelog:
> >> v7:
> >> Patch 1 was merged into mm-unstable, so this is resent as a single
> >> patch.
> >> According to the comments of Sashiko, Return instead of waiting when the
> >> lruvec has no evictable folios at all: allowed == 0 previously conflated
> >> "all empty" with "all over-isolated" and stalled even kswapd for 20ms.
> >> Move the v6 cover-letter test data into the commit message.
> >> v6:
> >> According to the comments of Barry, add real-world workload tests:
> >> a CI-style kernel build (repeated "make -j20 vmlinux") inside a
> >> memory-limited cgroup with its object tree on dm-delay-backed storage,
> >> measured at three pressure levels (1024/1536/2048MB), plus redis-based
> >> cache service runs (single instance and 8 shards sharing one cgroup)
> >> as no-impact data points. New and updated test scripts are in the
> >> gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
> >> current isolate_folios() type-selection context (is_single_type_
> >> reclaim()), with no semantic change. Re-run the tests and update the
> >> test logs.
> >> v5:
> >> According to the comments of Kairun and feedback from the test scripts,
> >> make the throttle check per lruvec (new nr_isolated counter) and skip
> >> empty types in the allowed mask.
> >> v4:
> >> According to the commens of Baolin and Barry, rework patch 2:
> >> drop the throttle_is_throttled() helper extracted
> >> from shrink_inactive_list() and leave the legacy path untouched.
> >> The new MGLRU-only throttle_evictable_types() only sleeps when all
> >> evictable types are over-isolated - v3 throttled as soon as any of
> >> them was, which unnecessarily blocked the reclaim of the other type
> >> and passes the mask of the remaining types to isolate_folios(),
> >> which restricts both its initial choice and its fallback, so that
> >> isolation never lands on a throttled type.  The v3 gate did not
> >> constrain the type actually isolated, so the fallback could still
> >> pick the over-isolated one.
> >> Re-run the tests and update the test log.
> >> v3:
> >> According to the commens of Baolin, remove the redundant nr_isolated
> >> check before restoring the NR_ISOLATED_* counters in evict_folios().
> >> rename the extracted helper to throttle_is_throttled() to avoid
> >> confusion with the existing wake_throttle_isolated() naming space.
> >> Use for_each_evictable_type() in the MGLRU throttle to check each
> >> evictable type's isolation instead of only the type returned by
> >> get_type_to_scan(), since isolate_folios() may fall back to the
> >> other type.
> >> Re-run the tests and update the test log.
> >> v2:
> >> According to the commens of Kairun, Rebased on mm-unstable.
> >> Split into two patches; patch 2 is new and adds the
> >> too_many_isolated() throttling to the MGLRU eviction path, which v1
> >> did not cover.
> >> Add test infomations.
> >>
> >>   include/linux/mmzone.h |   2 +
> >>   mm/vmscan.c            | 184 +++++++++++++++++++++++++++++++++++++----
> >>   2 files changed, 172 insertions(+), 14 deletions(-)
> >>
> >> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> >> index 3b96d6c7123b..7759a53326ba 100644
> >> --- a/include/linux/mmzone.h
> >> +++ b/include/linux/mmzone.h
> >> @@ -564,6 +564,8 @@ struct lru_gen_folio {
> >>          /* can be modified without holding the LRU lock */
> >>          atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> >>          atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> >> +       /* the number of folios isolated from this lruvec and being reclaimed */
> >> +       atomic_long_t nr_isolated[ANON_AND_FILE];
> >>          /* whether the multi-gen LRU is enabled */
> >>          bool enabled;
> >>          /* the memcg generation this lru_gen_folio belongs to */
> >> diff --git a/mm/vmscan.c b/mm/vmscan.c
> >> index daa7e37c2236..36887117d76a 100644
> >> --- a/mm/vmscan.c
> >> +++ b/mm/vmscan.c
> >> @@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
> >>          sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
> >>   }
> >>
> >> -static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> >> +static unsigned long lruvec_type_evictable_size(struct lruvec *lruvec, int type)
> >>   {
> >> -       int gen, type, zone;
> >> +       int gen, zone;
> >>          unsigned long seq, total = 0;
> >>          struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >>          DEFINE_MAX_SEQ(lruvec);
> >>          DEFINE_MIN_SEQ(lruvec);
> >>
> >> -       for_each_evictable_type(type, swappiness) {
> >> -               for (seq = min_seq[type]; seq <= max_seq; seq++) {
> >> -                       gen = lru_gen_from_seq(seq);
> >> -                       for (zone = 0; zone < MAX_NR_ZONES; zone++)
> >> -                               total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> >> -               }
> >> +       for (seq = min_seq[type]; seq <= max_seq; seq++) {
> >> +               gen = lru_gen_from_seq(seq);
> >> +               for (zone = 0; zone < MAX_NR_ZONES; zone++)
> >> +                       total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> >>          }
> >>
> >>          return total;
> >>   }
> >>
> >> +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> >> +{
> >> +       unsigned long total = 0;
> >> +       int type;
> >> +
> >> +       for_each_evictable_type(type, swappiness)
> >> +               total += lruvec_type_evictable_size(lruvec, type);
> >> +
> >> +       return total;
> >> +}
> >> +
> > The helper split looks clean; it could be a stand alone commit I think.
> >
> >>   static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
> >>   {
> >>          unsigned long total;
> >> @@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness)
> >>                 swappiness == SWAPPINESS_ANON_ONLY;
> >>   }
> >>
> >> +/*
> >> + * The MGLRU counterpart to too_many_isolated().
> >> + *
> >> + * too_many_isolated() compares node-wide isolated counts against
> >> + * node-wide inactive sizes. It predates per-memcg LRU lists, which
> >> + * moved isolation to per-lruvec granularity. MGLRU reclaimers contend
> >> + * per lruvec, so this check is per lruvec as well, and isolation in
> >> + * one memcg does not throttle reclaim in another.
> >> + *
> >> + * The legacy threshold, the size of the inactive list, is not usable
> >> + * here either: in MGLRU the inactive counters are only a compatibility
> >> + * shim - they jump around when swap runs out or a memcg hits its swap
> >> + * limit, and proactive aging can inflate them. Compare against the
> >> + * total number of evictable pages of the type divided by MIN_NR_GENS,
> >> + * approximating the size of one generation - the same unit the
> >> + * original MGLRU aging heuristics used.
> >> + */
> >> +static bool lru_gen_too_many_isolated(struct lruvec *lruvec, int type,
> >> +                                     struct scan_control *sc)
> >> +{
> >> +       unsigned long isolated, evictable;
> >> +
> >> +       if (current_is_kswapd())
> >> +               return false;
> >> +
> >> +       if (!writeback_throttling_sane(sc))
> >> +               return false;
> >> +
> >> +       isolated = atomic_long_read(&lruvec->lrugen.nr_isolated[type]);
> >> +       evictable = lruvec_type_evictable_size(lruvec, type);
> >> +
> >> +       /*
> >> +        * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so
> >> +        * they won't be blocked by normal direct-reclaimers, forming a
> >> +        * circular deadlock.
> >> +        */
> >> +       if (gfp_has_io_fs(sc->gfp_mask))
> >> +               evictable >>= 3;
> >> +
> >> +       return isolated > evictable / MIN_NR_GENS;
> >> +}
> > The magic number of >>=3 looks a bit odd, others are fine to me.
> >
> >> +
> >> +/*
> >> + * Unlike the legacy path, where the LRU list to isolate from is known
> >> + * before isolation, isolate_folios() picks the type from the refault
> >> + * feedback and may fall back to the other one. Therefore, instead of
> >> + * throttling on a single type, collect the evictable types that do not
> >> + * have too many isolated folios, and only sleep when all of them do.
> >> + *
> >> + * Returns the mask of the types isolate_folios() may isolate from, or
> >> + * 0 if reclaim should stop. Also sets @fatal to tell the caller that
> >> + * the task received a fatal signal while waiting.
> >> + */
> >> +static unsigned int throttle_evictable_types(struct lruvec *lruvec,
> >> +                                            int swappiness,
> >> +                                            struct scan_control *sc,
> >> +                                            bool *fatal)
> >> +{
> >> +       unsigned int allowed;
> >> +       bool stalled = false;
> >> +       bool has_evictable;
> >> +       struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> >> +       int i;
> >> +
> >> +       *fatal = false;
> >> +
> >> +       for (;;) {
> >> +               allowed = 0;
> >> +               has_evictable = false;
> >> +               for_each_evictable_type(i, swappiness) {
> >> +                       /*
> >> +                        * A type with no evictable folios has nothing to
> >> +                        * isolate. Don't allow it, otherwise reclaim can be
> >> +                        * redirected to it and spin making no progress, e.g.
> >> +                        * when the only type with folios is over-isolated.
> >> +                        */
> >> +                       if (!lruvec_type_evictable_size(lruvec, i))
> >> +                               continue;
> >> +
> >> +                       has_evictable = true;
> >> +
> >> +                       if (!lru_gen_too_many_isolated(lruvec, i, sc))
> >> +                               allowed |= BIT(i);
> >> +               }
> >> +
> >> +               /*
> >> +                * Nothing evictable at all is not over-isolation. Stop
> >> +                * reclaim instead of waiting, so that nobody - including
> >> +                * kswapd, which is exempt from the check below - stalls on
> >> +                * an empty lruvec.
> >> +                */
> >> +               if (!has_evictable)
> >> +                       return 0;
> >> +
> >> +               if (allowed) {
> >> +                       /* Wake up reclaimers waiting on the isolation to go down. */
> >> +                       wake_throttle_isolated(pgdat);
> >> +                       return allowed;
> > This seems counterintuitive: we reach here because a reclaimer is
> > starting, and we wake up even more reclaimer?
> >
> >> +               }
> >> +
> >> +               /*
> >> +                * All evictable types are over-isolated. Like the legacy
> >> +                * path, wait once for concurrent reclaimers to put their
> >> +                * isolated folios back; give up if that makes no progress.
> >> +                */
> >> +               if (stalled)
> >> +                       return 0;
> >> +
> >> +               stalled = true;
> >> +               reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
> >> +
> >> +               /* We are about to die and free our memory. Return now. */
> >> +               if (fatal_signal_pending(current)) {
> >> +                       *fatal = true;
> >> +                       return 0;
> >> +               }
> >> +       }
> >> +}
> >> +
> > Hmm... this part looks a bit too complex, see below.
> >
> >>   static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >>                            struct scan_control *sc, int swappiness,
> >> -                         struct list_head *list, int *isolated,
> >> -                         int *isolate_type, int *isolate_scanned)
> >> +                         unsigned int allowed, struct list_head *list,
> >> +                         int *isolated, int *isolate_type, int *isolate_scanned)
> >>   {
> >>          bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
> >>          int type = get_type_to_scan(lruvec, swappiness);
> >>          int total_scanned = 0, scanned, tier;
> >>          bool tried = false;
> >>
> >> +       /*
> >> +        * The preferred type may have been excluded by
> >> +        * throttle_evictable_types(); start from the other one.
> >> +        */
> >> +       if (!(allowed & BIT(type)))
> >> +               type = !type;
> >> +
> >>   retry:
> >>          tier = get_tier_idx(lruvec, type);
> >>          scanned = scan_folios(nr_to_scan, lruvec, sc,
> >> @@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >>
> >>          /*
> >>           * We are running out of the current reclaim type. Fall back to
> >> -        * the other type if allowed.
> >> +        * the other type if allowed. Only fall back to a type that is
> >> +        * not throttled.
> >>           */
> >> -       if (!scanned && !tried && type_fallback_allowed) {
> >> +       if (!scanned && !tried && type_fallback_allowed &&
> >> +           (allowed & BIT(!type))) {
> >>                  type = !type;
> >>                  tried = true;
> >>                  type_fallback_allowed = false;
> >> @@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >>          bool skip_retry = false;
> >>          struct mem_cgroup *memcg = lruvec_memcg(lruvec);
> >>          struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> >> +       unsigned int allowed;
> >> +       bool fatal;
> >> +
> >> +       allowed = throttle_evictable_types(lruvec, swappiness, sc, &fatal);
> >> +       if (!allowed) {
> >> +               /*
> >> +                * We are about to die and free our memory. Like the legacy
> >> +                * path, pretend some pages were reclaimed so reclaim
> >> +                * unwinds quickly instead of looping back into the
> >> +                * throttle.
> >> +                */
> >> +               if (fatal)
> >> +                       sc->nr_reclaimed += SWAP_CLUSTER_MAX;
> > And reporting a fake nr_reclaimed looks a bad thing to do in the first place.
> >
> > It seems a lot of trouble is caused by the design of the reclaim loop,
> > so will it work better if we calculate the number & type to reclaim
> > first like CLRU?
> >
> > There is a demonstration from Barry:
> > https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
> >
> > And instead of throttle at the reclaim loop, balance the scan budge
> > based on (size - isolation) number, and throttle a while LRU simply
> > when the total isolated number is larger than total size /
> > MIN_NR_GENS.
> >
> > Just an idea for reference, seems could help get rid of a lot of complex logic.
>
> Hi Kairui,
>
> Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and
> I am quite interested in it.
>
> I think that branch still needs an over-isolation guard. The scan budget
> answers "which type and how much", but it is computed once per
> try_to_shrink_lruvec() and does not see what other reclaimers are
> holding. Each reclaimer's in-flight isolation is bounded by the batch
> size rather than by the budget, so N reclaimers entering together all
> see the same state and all proceed.

Yeah, exactly.

> On the other hand, that branch would also make the guard much simpler
> and clearer. Since evict_folios() already takes the type from its
> caller, the allowed mask, the type rewrite and the fallback restriction
> in isolate_folios() all go away, and the throttle reduces to a single
> check on the totals in the shrink loop, which is basically the second
> half of what you suggested.
>
> So I would like to ask what the development plan for this branch is,
> and whether you think it is worth moving the MGLRU too_many_isolated()
> work over to it.

Thanks for looking into that!

I think the isolation protection is definitely needed; really
appreciate your work!

However, many things are currently happening in parallel for MGLRU.
Will the code be cleaner if worked on based on that branch? And
MGLRU's budge will be more similiar with CLRU, so I hope maybe it can
even be unified with CLRU for this isolation protection, just with a
different gating (size / MIN_NR_GEN vs size of inactive), will be much
lower LOC.