Re: [PATCH v4 00/17] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup
From: Kairui Song
Date: Fri Nov 13 2026 - 16:02:47 EST
On Fri, Oct 16, 2026 at 02:15:49AM +0800, Kairui Song via B4 Relay wrote:
> Hi all, this cover letter is going to be very long, as it includes a
> systematic test and analysis. And moving this series out of RFC phase
> as almost all readings are positive, tested on at least five different
> platforms (desktop, large server, Android, x86_64 and arm64 boxes)
> covering typical benchmarks, database, real applications, and has
> good test reports from other individual testers. It's also pretty clear
> about the next steps, and this blocks or breaks nothing AFAIK, which
> should be a pure gain, improving the performance or reducing IO of many
> workloads by ~10-40%.
A little update on the testing result: I've been running a week-long,
four-arm test matrix on this, and the tests took 2 weeks to finish
because it's a very large matrix based on this V4, on top of the
partial reversion of 6cbdd9726fb5:
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index b3d85fe91385..d39061238fab 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -337,7 +337,7 @@ static inline unsigned long
lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
- gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+ gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_lru_refs(folio) > 1);
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..a2600d1b6d6c 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -317,15 +317,14 @@ static void lru_gen_refault(struct folio *folio,
void *shadow)
refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + 1;
tier = lru_tier_from_refs(refs, workingset);
+ if (lru_gen_in_fault()) {
+ mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
+ folio_set_active(folio);
+ }
+
atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
if (workingset) {
- /*
- * see folio_add_lru(), where folio_set_active() is
- * called for workingset folios
- */
- if (lru_gen_in_fault())
- mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
The four arms are:
- ORIG: mm-unstable 2 weeks ago
- FIXED: mm-unstable 2 weeks ago with the fix above
- FG: MGLRU-FG on top of FIXED
- RD: Refault distance on top of FG
- The refault distance algorithm is from:
- https://lore.kernel.org/linux-mm/20260502-mglru-fg-v1-0-913619b014d9@xxxxxxxxxxx/
- The refault distance part is still being tweaked frequently, so RD
results are just for reference.
I didn't include CLRU this time or it will be too long, all tests are done
for 12 interleave boots, could be noise but given the size of the matrix I
think it's quite reliable.
Test spec:
Zipf is widely used here, as it is commonly recognized as a good way
to represent different kinds of workloads covering server, desktop
interacrtive workloads, CDN, router, etc, depending on the theta
factor and memory limit size:
None of the test involves MADV_FREE / MADV_COLD, MADV_RANDOM is also bypassed.
### fio
# Test is done with a NVME drive, memory limitation using memcg, THP off,
# swap off. 40 GiB working set (16 × 2560 MiB files) Common cmdline part:
numactl --membind=0 --cpunodebind=0 \
fio --name=fg --numjobs=16 --nrfiles=1 \
--filename_format=/var/lib/kmultibench/realdk.$jobnum.img \
--size=2560M --rw=randread --bs=4k --time_based \
--random_distribution=zipf:${θ} \
--ramp_time=120s --runtime=600s --group_reporting
# fio fd reading:
--buffered=1 --ioengine=sync
# fio mmap reading (bypassing MADV_RANDOM at the moment):
--ioengine=mmap --fadvise_hint=0
### YCSB
# MongoDB 7 container, --network host, --memory 12G --memory-swap 12G (no swap),
# workload {a,b,c}, mem 12G, threads 16, iops limited to 5000.
# Also wiredTigerCacheSizeGB 0.25 (forces the hot set into the OS page cache)
# LOAD (untimed): 18,000,000 records x 10 fields x 100 B ~= 18 GB
ycsb load mongodb -s -p mongodb.url=mongodb://localhost:27017/ycsb \
-p recordcount=18000000 -p fieldcount=10 -p fieldlength=100 \
-threads 16 -P workloads/workloada
# RUN (timed): 100% zipfian read, 27,000,000 ops (1.5 x recordcount), 16 threads
ycsb run mongodb -s -p mongodb.url=mongodb://localhost:27017/ycsb \
-p recordcount=18000000 -p fieldcount=10 -p fieldlength=100 -threads 16 \
-p requestdistribution=zipfian -p readproportion=1 -p updateproportion=0 \
-P workloads/workloada -p operationcount=27000000 -t
And the results are:
fio FD READ: Avg IOPS (higher is better)
memcg/θ ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
-----------------------------------------------------------------------------------------
16G/1.4 13,993,868| 14,153,014 ( +1.1%) | 14,619,834 ( +4.5%) | 14,492,871 ( +3.6%) |
16G/1.2 4,557,469| 4,564,882 ( +0.2%) | 5,188,226 ( +13.8%) | 5,066,064 ( +11.2%) |
16G/1.1 2,167,267| 2,158,549 ( -0.4%) | 2,453,911 ( +13.2%) | 2,417,420 ( +11.5%) |
16G/0.9 610,129| 617,994 ( +1.3%) | 718,869 ( +17.8%) | 708,871 ( +16.2%) |
16G/0.8 401,362| 406,178 ( +1.2%) | 465,164 ( +15.9%) | 454,454 ( +13.2%) |
fio FD READ: refault per read (lower is better)
memcg/θ ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
-----------------------------------------------------------------------------------------
16G/1.4 0.00182| 0.00169 ( -7.0%) | 0.00131 ( -28.1%) | 0.00135 ( -25.7%) |
16G/1.2 0.01557| 0.01551 ( -0.4%) | 0.01323 ( -15.0%) | 0.01342 ( -13.8%) |
16G/1.1 0.03785| 0.03782 ( -0.1%) | 0.03261 ( -13.8%) | 0.03313 ( -12.5%) |
16G/0.9 0.13992| 0.13978 ( -0.1%) | 0.12301 ( -12.1%) | 0.12495 ( -10.7%) |
16G/0.8 0.21259| 0.21256 ( -0.0%) | 0.18853 ( -11.3%) | 0.19146 ( -9.9%) |
fio MMAP: Avg IOPS (higher is better)
memcg/θ ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
-----------------------------------------------------------------------------------------
16G/1.4 20,670,288| 18,601,708 ( -10.0%) | 21,428,715 ( +3.7%) | 21,604,461 ( +4.5%) |
16G/1.2 4,739,384| 4,959,906 ( +4.7%) | 5,126,105 ( +8.2%) | 5,270,792 ( +11.2%) |
16G/1.1 2,145,716| 2,422,244 ( +12.9%) | 2,560,802 ( +19.3%)*| 3,554,448 ( +65.7%)*|
16G/0.9 3,887,814| 4,259,544 ( +9.6%) | 4,372,044 ( +12.5%) | 4,431,153 ( +14.0%) |
16G/0.8 3,566,541| 3,907,269 ( +9.6%) | 4,085,982 ( +14.6%) | 4,040,826 ( +13.3%) |
8G/1.4 13,471,704| 14,102,398 ( +4.7%) | 14,524,256 ( +7.8%) | 15,528,982 ( +15.3%) |
4G/1.4 9,222,754| 9,540,640 ( +3.4%) | 9,973,791 ( +8.1%) | 11,066,377 ( +20.0%) |
12G/1.2 3,565,213| 3,593,112 ( +0.8%) | 3,826,527 ( +7.3%) | 4,019,067 ( +12.7%) |
8G/1.2 2,504,381| 2,649,853 ( +5.8%) | 2,776,443 ( +10.9%) | 2,990,251 ( +19.4%) |
4G/1.2 1,676,208| 1,728,339 ( +3.1%) | 1,825,272 ( +8.9%) | 2,064,706 ( +23.2%) |
12G/1.1 1,592,245| 1,579,824 ( -0.8%) | 1,726,557 ( +8.4%) | 1,775,451 ( +11.5%) |
8G/1.1 1,117,784| 1,169,936 ( +4.7%) | 1,229,181 ( +10.0%) | 1,325,119 ( +18.5%) |
6G/1.1 939,984| 980,984 ( +4.4%) | 996,724 ( +6.0%) | 1,134,737 ( +20.7%) |
fio MMAP: refault per read (lower is better)
memcg/θ ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
-----------------------------------------------------------------------------------------
16G/1.4 0.00202| 0.00317 ( +57.0%) | 0.00187 ( -7.7%) | 0.00179 ( -11.3%) |
16G/1.2 0.01671| 0.01550 ( -7.2%) | 0.01529 ( -8.5%) | 0.01462 ( -12.5%) |
16G/1.1 0.03965| 0.03394 ( -14.4%) | 0.03318 ( -16.3%)*| 0.02312 ( -41.7%)*|
16G/0.9 0.01982| 0.01791 ( -9.6%) | 0.01771 ( -10.6%) | 0.01735 ( -12.4%) |
16G/0.8 0.02149| 0.01957 ( -8.9%) | 0.01889 ( -12.1%) | 0.01923 ( -10.5%) |
8G/1.4 0.00446| 0.00408 ( -8.5%) | 0.00394 ( -11.7%) | 0.00359 ( -19.4%) |
4G/1.4 0.00745| 0.00700 ( -6.1%) | 0.00668 ( -10.3%) | 0.00583 ( -21.7%) |
12G/1.2 0.02338| 0.02252 ( -3.7%) | 0.02131 ( -8.8%) | 0.02016 ( -13.8%) |
8G/1.2 0.03311| 0.03076 ( -7.1%) | 0.02984 ( -9.9%) | 0.02766 ( -16.5%) |
4G/1.2 0.05043| 0.04746 ( -5.9%) | 0.04574 ( -9.3%) | 0.04071 ( -19.3%) |
12G/1.1 0.05536| 0.05371 ( -3.0%) | 0.05099 ( -7.9%) | 0.04847 ( -12.4%) |
8G/1.1 0.07682| 0.07110 ( -7.4%) | 0.06966 ( -9.3%) | 0.06507 ( -15.3%) |
6G/1.1 0.09150| 0.08489 ( -7.2%) | 0.08313 ( -9.1%) | 0.07650 ( -16.4%) |
YCSB: throughput ops/sec (higher is better), n=3
workload ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
----------------------------------------------------------------------------------------
workloada 14,657| 14,284 ( -2.5%) | 16,788 ( +14.5%) | 17,836 ( +21.7%) |
workloadb 14,309| 14,304 ( -0.0%) | 16,504 ( +15.3%) | 17,957 ( +25.5%) |
workloadc 14,458| 13,777 ( -4.7%) | 16,806 ( +16.2%) | 17,868 ( +23.6%) |
YCSB READ p99 latency (lower is better):
workload ORIG | FIXED (vs ORIG) | FG (vs ORIG) | RD (vs ORIG) |
----------------------------------------------------------------------------------------
10-wla 61535.0| 65375.0 ( +6.2%) | 3474.3 ( -94.4%) | 2888.3 ( -95.3%) |
20-wlb 65641.7| 64265.7 ( -2.1%) | 3901.0 ( -94.1%) | 2757.0 ( -95.8%) |
30-wlc 63348.3| 71188.3 ( +12.4%) | 3251.0 ( -94.9%) | 2724.3 ( -95.7%) |
So in conclusion:
1. The FIXED is helpful for the MMAP read by ~5-10%, harmless in most
cases and with certain regressions in one or two cases, it may
correspond to the LevelDB regression I described a while ago:
https://lore.kernel.org/linux-mm/CAMgjq7AtuK22FsuqvGRLVZ2eWrp-N4qQ_MHoXsXDkVBprekdDA@xxxxxxxxxxxxxx/
Which I think it's fine to have as the previous change that FIXED is
trying to patch is causing other DB regression too.
2. MGLRU-FG is suitable across all cases covered above and
outperformance everything we have in upstream so far by ~9-20% in
these generic tests, and a huge improvement for lower latency (-90%).
3. Refault Distance will greatly help with cases where working set
sits at the cgroup boundary on top of FG, offering up ~65% vs ORIG
(~38% over FG) performance gain for certain cases. I think Refault
distance will also cover the workingset shift case, with a slight
regression for buffered read which could be the counter update
overhead, I will explore that further after FG.
Posting these here so I can reference them later and try make the test
result less bloated for later versions, and will be rebasing for V5
soon, maybe with a shorter test process.