Re: [PATCH v6 00/14] mm/mglru: improve reclaim loop and dirty folio handling
From: Barry Song
Date: Fri Apr 24 2026 - 06:38:44 EST
On Fri, Apr 24, 2026 at 1:43 AM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@xxxxxxxxxx> wrote:
>
> From: Kairui Song <kasong@xxxxxxxxxxx>
>
> This series cleans up and slightly improves MGLRU's reclaim loop and
> dirty writeback handling. As a result, we can see an up to ~30% increase
> in some workloads like MongoDB with YCSB and a huge decrease in file
> refault, no swap involved. Other common benchmarks have no regression,
> and LOC is reduced, with less unexpected OOM, too.
>
> Some of the problems were found in our production environment, and
> others were mostly exposed while stress testing during the development
> of the LSM/MM/BPF topic on improving MGLRU [1]. This series cleans up
> the code base and fixes several performance issues, preparing for
> further work.
>
> MGLRU's reclaim loop is a bit complex, and hence these problems are
> somehow related to each other. The aging, scan number calculation, and
> reclaim loop are coupled together, and the dirty folio handling logic is
> quite different, making the reclaim loop hard to follow and the dirty
> flush ineffective.
>
> This series slightly cleans up and improves these issues using a scan
> budget by calculating the number of folios to scan at the beginning of
> the loop, and decouples aging from the reclaim calculation helpers.
> Then, move the dirty flush logic inside the reclaim loop so it can kick
> in more effectively. These issues are somehow related, and this series
> handles them and improves MGLRU reclaim in many ways.
>
> Test results: All tests are done on a 48c96t NUMA machine with 2 nodes
> and a 128G memory machine using NVME as storage.
>
> MongoDB
> =======
> Running YCSB workloadb [2] (recordcount:20000000 operationcount:6000000,
> threads:32), which does 95% read and 5% update to generate mixed read
> and dirty writeback. MongoDB is set up in a 10G cgroup using Docker, and
> the WiredTiger cache size is set to 4.5G, using NVME as storage.
>
> Not using SWAP.
>
> Before:
> Throughput(ops/sec): 62485.02962831822
> AverageLatency(us): 500.9746963330107
> pgpgin 159347462
> pgpgout 5413332
> workingset_refault_anon 0
> workingset_refault_file 34522071
>
> After:
> Throughput(ops/sec): 79760.71784646061 (+27.6%, higher is better)
> AverageLatency(us): 391.25169970043726 (-21.9%, lower is better)
> pgpgin 111093923 (-30.3%, lower is better)
> pgpgout 5437456
> workingset_refault_anon 0
> workingset_refault_file 19566366 (-43.3%, lower is better)
>
> We can see a significant performance improvement after this series.
> The test is done on NVME and the performance gap would be even larger
> for slow devices, such as HDD or network storage. We observed over
> 100% gain for some workloads with slow IO.
>
> Chrome & Node.js [3]
> ====================
> Using Yu Zhao's test script [3], testing on a x86_64 NUMA machine with 2
> nodes and 128G memory, using 256G ZRAM as swap and spawn 32 memcg 64
> workers:
>
> Before:
> Total requests: 79915
> Per-worker 95% CI (mean): [1233.9, 1263.5]
> Per-worker stdev: 59.2
> Jain's fairness: 0.997795 (1.0 = perfectly fair)
> Latency:
> Bucket Count Pct Cumul
> [0,1)s 26859 33.61% 33.61%
> [1,2)s 7818 9.78% 43.39%
> [2,4)s 5532 6.92% 50.31%
> [4,8)s 39706 49.69% 100.00%
>
> After:
> Total requests: 81382
> Per-worker 95% CI (mean): [1241.9, 1301.3]
> Per-worker stdev: 118.8
> Jain's fairness: 0.991480 (1.0 = perfectly fair)
> Latency:
> Bucket Count Pct Cumul
> [0,1)s 26696 32.80% 32.80%
> [1,2)s 8745 10.75% 43.55%
> [2,4)s 6865 8.44% 51.98%
> [4,8)s 39076 48.02% 100.00%
>
> Reclaim is still fair and effective, total requests number seems
> slightly better.
>
> OOM issue with aging and throttling
> ===================================
> For the throttling OOM issue, it can be easily reproduced using dd and
> cgroup limit as demonstrated in patch 14, and fixed by this series.
>
> The aging OOM is a bit tricky, a specific reproducer can be used to
> simulate what we encountered in production environment [4]:
> Spawns multiple workers that keep reading the given file using mmap,
> and pauses for 120ms after one file read batch. It also spawns another
> set of workers that keep allocating and freeing a given size of
> anonymous memory. The total memory size exceeds the memory limit
> (eg. 14G anon + 8G file, which is 22G vs a 16G memcg limit).
>
> - MGLRU disabled:
> Finished 128 iterations.
>
> - MGLRU enabled:
> OOM with following info after about ~10-20 iterations:
> [ 62.624130] file_anon_mix_p invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
> [ 62.624999] memory: usage 16777216kB, limit 16777216kB, failcnt 24460
> [ 62.640200] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> [ 62.640823] Memory cgroup stats for /demo:
> [ 62.641017] anon 10604879872
> [ 62.641941] file 6574858240
>
> OOM occurs despite there being still evictable file folios.
>
> - MGLRU enabled after this series:
> Finished 128 iterations.
>
> Worth noting there is another OOM related issue reported in V1 of
> this series, which is tested and looking OK now [5].
>
> MySQL:
> ======
>
> Testing with innodb_buffer_pool_size=26106127360, in a 2G memcg, using
> ZRAM as swap and test command:
>
> sysbench /usr/share/sysbench/oltp_read_only.lua --mysql-db=sb \
> --tables=48 --table-size=2000000 --threads=48 --time=600 run
>
> Before: 17260.781429 tps
> After this series: 17266.842857 tps
>
> MySQL is anon folios heavy, involves writeback and file and still
> looking good. Seems only noise level changes, no regression.
>
> FIO:
> ====
> Testing with the following command, where /mnt/ramdisk is a
> 64G EXT4 ramdisk, each test file is 3G, in a 10G memcg,
> 6 test run each:
>
> fio --directory=/mnt/ramdisk --filename_format='test.$jobnum.img' \
> --name=cached --numjobs=16 --size=3072M --buffered=1 --ioengine=mmap \
> --rw=randread --norandommap --time_based \
> --ramp_time=1m --runtime=5m --group_reporting
>
> Before: 9196.481429 MB/s
> After this series: 9256.105000 MB/s
>
> Also seem only noise level changes and no regression or slightly better.
>
> Build kernel:
> =============
> Build kernel test using ZRAM as swap, on top of tmpfs, in a 3G memcg
> using make -j96 and defconfig, measuring system time, 12 test run each.
>
> Before: 2589.63s
> After this series: 2543.58s
>
> Also seem only noise level changes, no regression or very slightly better.
>
> Android:
> ========
> Xinyu reported a performance gain on Android, too, with this series. The
> test consisted of cold-starting multiple applications sequentially under
> moderate system load. [6]
>
> Before:
> Launch Time Summary (all apps, all runs)
> Mean 868.0ms
> P50 888.0ms
> P90 1274.2ms
> P95 1399.0ms
>
> After:
> Launch Time Summary (all apps, all runs)
> Mean 850.5ms (-2.07%)
> P50 861.5ms (-3.04%)
> P90 1179.0ms (-8.05%)
> P95 1228.0ms (-12.2%)
>
> Link: https://lore.kernel.org/linux-mm/CAMgjq7BoekNjg-Ra3C8M7=8=75su38w=HD782T5E_cxyeCeH_g@xxxxxxxxxxxxxx/ [1]
> Link: https://github.com/brianfrankcooper/YCSB/blob/master/workloads/workloadb [2]
> Link: https://lore.kernel.org/all/20221220214923.1229538-1-yuzhao@xxxxxxxxxx/ [3]
> Link: https://github.com/ryncsn/emm-test-project/tree/master/file-anon-mix-pressure [4]
> Link: https://lore.kernel.org/linux-mm/acgNCzRDVmSbXrOE@KASONG-MC4/ [5]
> Link: https://lore.kernel.org/linux-mm/20260417025123.2971253-1-wxy2009nrrr@xxxxxxx/ [6]
>
> Signed-off-by: Kairui Song <kasong@xxxxxxxxxxx>
> ---
Hi Kairui,
I haven't identified the exact commit, but this patchset seems to
make MGLRU's swappiness behavior more erratic.
In mainline, MGLRU does not show as strong an effect as the
active/inactive LRU, but it still behaves roughly linearly: higher
swappiness leads to more swap activity and fewer file refaults.
With this patchset, however, the behavior becomes non-monotonic as
swappiness increases. I observed clear up-and-down fluctuations.
I reproduced this by running a kernel build in a memcg limited to
1GB, with swappiness set to 35, 70, 105, 140, and 175.
this is mainline using MGLRU:
*** Executing round 1 ***
set swappiness to 35
real 1m49.247s
user 25m30.484s
sys 3m37.203s
pswpin: 933731
pswpout: 3365968
pgpgin: 5649320
pgpgout: 13786572
swpout_zero: 794960
swpin_zero: 10594
refault_file: 354998
refault_anon: 944323
*** Executing round 2 ***
set swappiness to 70
real 1m49.313s
user 25m31.643s
sys 3m40.661s
pswpin: 1049052
pswpout: 3565887
pgpgin: 5694288
pgpgout: 14582200
swpout_zero: 840947
swpin_zero: 12029
refault_file: 242973
refault_anon: 1061033
*** Executing round 3 ***
set swappiness to 105
real 1m48.611s
user 25m32.198s
sys 3m37.210s
pswpin: 981095
pswpout: 3396069
pgpgin: 5283940
pgpgout: 13898988
swpout_zero: 795932
swpin_zero: 11249
refault_file: 202432
refault_anon: 992295
*** Executing round 4 ***
set swappiness to 140
real 1m49.398s
user 25m35.650s
sys 3m50.656s
pswpin: 1222881
pswpout: 3935186
pgpgin: 6165024
pgpgout: 16056664
swpout_zero: 913808
swpin_zero: 13251
refault_file: 191564
refault_anon: 1236083
*** Executing round 5 ***
set swappiness to 175
real 1m49.513s
user 25m35.442s
sys 3m55.869s
pswpin: 1343139
pswpout: 4256014
pgpgin: 6557152
pgpgout: 17341452
swpout_zero: 998107
swpin_zero: 15692
refault_file: 175795
refault_anon: 1358782
this is mm-new using MGLRU:
*** Executing round 1 ***
set swappiness to 35
real 1m51.804s
user 25m38.070s
sys 4m16.301s
pswpin: 1587728
pswpout: 4932011
pgpgin: 8788688
pgpgout: 20062761
swpout_zero: 1129975
swpin_zero: 17944
refault_file: 487923
refault_anon: 1605670
*** Executing round 2 ***
set swappiness to 70
real 1m51.503s
user 25m37.581s
sys 4m18.161s
pswpin: 1743890
pswpout: 5214587
pgpgin: 8676728
pgpgout: 21178716
swpout_zero: 1185453
swpin_zero: 20016
refault_file: 317993
refault_anon: 1763904
*** Executing round 3 ***
set swappiness to 105
real 1m51.154s
user 25m37.956s
sys 4m15.017s
pswpin: 1687517
pswpout: 5073825
pgpgin: 8173036
pgpgout: 20608932
swpout_zero: 1161806
swpin_zero: 20069
refault_file: 249769
refault_anon: 1707538
*** Executing round 4 ***
set swappiness to 140
real 1m50.732s
user 25m37.686s
sys 4m16.066s
pswpin: 1671678
pswpout: 5118895
pgpgin: 7929960
pgpgout: 20790468
swpout_zero: 1171029
swpin_zero: 19596
refault_file: 193421
refault_anon: 1691228
*** Executing round 5 ***
set swappiness to 175
real 1m49.518s
user 25m37.653s
sys 4m12.619s
pswpin: 1506888
pswpout: 4789793
pgpgin: 7270448
pgpgout: 19479188
swpout_zero: 1119251
swpin_zero: 16699
refault_file: 187304
refault_anon: 1523585
The final one is classic active/inactive LRU:
*** Executing round 1 ***
set swappiness to 35
real 1m50.038s
user 25m21.911s
sys 3m42.798s
pswpin: 476994
pswpout: 2258185
pgpgin: 5247280
pgpgout: 9354640
swpout_zero: 684759
swpin_zero: 6387
refault_file: 750021
refault_anon: 483334
*** Executing round 2 ***
set swappiness to 70
real 1m48.781s
user 25m25.682s
sys 3m37.854s
pswpin: 515470
pswpout: 2306901
pgpgin: 4265500
pgpgout: 9547436
swpout_zero: 706437
swpin_zero: 6960
refault_file: 459740
refault_anon: 522381
*** Executing round 3 ***
set swappiness to 105
real 1m48.233s
user 25m26.623s
sys 3m38.843s
pswpin: 519540
pswpout: 2343897
pgpgin: 3628788
pgpgout: 9696500
swpout_zero: 743576
swpin_zero: 7782
refault_file: 303701
refault_anon: 527273
*** Executing round 4 ***
set swappiness to 140
real 1m48.800s
user 25m32.067s
sys 3m50.751s
pswpin: 605537
pswpout: 2615227
pgpgin: 3470540
pgpgout: 10776312
swpout_zero: 825446
swpin_zero: 9055
refault_file: 173236
refault_anon: 614544
*** Executing round 5 ***
set swappiness to 175
real 1m52.356s
user 25m29.727s
sys 3m55.664s
pswpin: 698228
pswpout: 2908292
pgpgin: 3602884
pgpgout: 11945332
swpout_zero: 912127
swpin_zero: 10298
refault_file: 117625
refault_anon: 708478
The build script is available here if you want to have a try:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/diff/tools/mm/build-kernel-with-increasing-swappiness.sh?h=zram-async-gc&id=d47888e9
I am also debugging this. One possibility is that placing
dirty pages in the youngest generation may have affected
lruvec_evictable_size()?
Thanks
Barry