Re: [PATCH RFC v2 00/15] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup

[Xiang Liu]

Hi Kairui,

I tested the complete MGLRU-FG RFC v2 series on an x86_64 machine with
a SATA SSD. I found that the LevelDB Scan/Get result changes with the
relative intensity of GET and SCAN.

Setup: base commit 9958290885035431e2494159f555e332c3c14906, about
28 GiB LevelDB, 2.5 GiB memcg, swap off, Zipf 0.99, SCAN length 10000,
240 seconds warm-up plus 240 seconds measurement. Each untraced ABBA
sample used a fresh database copy, page-cache drop, reboot and cooldown.

Untraced ABBA results
=====================

Workload                  GET/SCAN   Baseline       FG     Delta
1 SCAN + 3 GET                92.5    4688.62  4425.82    -5.61%
4 mixed workers, 99.5/0.5    195.2    6200.60  6385.57    +2.98%
1 SCAN + 7 GET               237.2    9574.97  9831.70    +2.68%

The first workload is the original four-thread layout: one continuous
SCAN thread and three GET threads. The second is a custom workload in
which every worker chooses 99.5% GET or 0.5% SCAN per request. The third
keeps one continuous SCAN thread and increases GET threads to seven.

In the original workload, FG changed GET throughput by -5.74% and SCAN
throughput by +6.91%. The two ABBA position pairs were -6.21% and
-5.00%. A separate 5 GiB memcg repeat was also negative: -9.07% and
-7.67%.

Traced results
==============

I ran one fresh-boot baseline/FG pair for each workload. To limit probe
overhead, I selected about 1/1024 of file offsets using a stable hash of
inode and page index.

Initial placement of newly loaded GET folios
--------------------------------------------

We tracked new file pages loaded by GET requests and recorded where
MGLRU initially placed each page: the oldest generation or the
second-oldest generation.

Workload / kernel       Sampled   Oldest        Second-oldest
1S+3G / baseline           1452    186 (12.8%)   1266 (87.2%)
1S+3G / FG                 1371   1052 (76.7%)    319 (23.3%)
Mixed / baseline           1782    199 (11.2%)   1583 (88.8%)
Mixed / FG                 1662   1312 (78.9%)    350 (21.1%)
1S+7G / baseline           2719    469 (17.2%)   2250 (82.8%)
1S+7G / FG                 2677   2107 (78.7%)    570 (21.3%)

For example, the original 1-SCAN/3-GET baseline run sampled 1452 newly
loaded GET folios. Of these, 1266 entered the second-oldest generation
and 186 entered the oldest. With FG, 1052 of 1371 sampled folios entered
the oldest generation.

The same change appears in all three workloads: mainline places most
new GET folios in the second-oldest generation, whereas FG places most
of them in the oldest generation.

FG promotion was active in all three workloads:

Workload              Later GET       Moved to the promoted generation
                      changed gen     during reclaim scan
1 SCAN + 3 GET              38              548
4 mixed workers             32              558
1 SCAN + 7 GET              70              936

However, FG also removed more sampled GET folios before another GET
reused the same in-memory folio:

                         Removed after one GET      Later reloaded
1 SCAN + 3 GET            81.5% -> 88.4%           44.7% -> 48.6%
4 mixed workers           73.4% -> 90.2%           48.3% -> 52.7%
1 SCAN + 7 GET            84.5% -> 88.2%           56.2% -> 61.1%

The arrows show baseline -> FG. "Later reloaded" means a later GET
demanded the same file offset after it had been removed from page cache.

The request-level counters explain why the final results differ:

FG change per request       GET cache miss   GET I/O wait   SCAN cache miss
1 SCAN + 3 GET                    +2.29%          +6.87%           -15.97%
4 mixed workers                   -5.68%          -4.19%            -6.93%
1 SCAN + 7 GET                    -2.89%          -0.32%            -8.26%

FG improved the SCAN side in every workload. In the original workload,
the GET miss and I/O increases were larger than the SCAN gain, producing
the 5.61% total regression. When GET was denser relative to SCAN, GET
misses also decreased and FG became a net win.

My interpretation is that FG introduces a cold-start window for a new
GET folio:

  first GET -> oldest generation
            -> a later GET promotes it, if it is still present
            -> otherwise reclaim removes it first

The outcome depends on whether GET reuse arrives before the continuous
SCAN stream consumes the oldest generation. The trace shows that both
promotion and early removal occur in practice.

This machine uses SATA, while the positive results in the cover letter
used NVMe. Storage latency may also change the timing among page-cache
insertion, LRU drain, the next GET and reclaim.

Is this cold-start window an expected trade-off of the current design?
Would limited initial protection for refs=1 folios, or a decision based
on observed refault/reuse distance, be worth testing?

I have preserved the workload source, ABBA results, kernel configs,
probe programs and raw traces and can share them if useful.

Thanks,
Xiang Liu
liuxiang.277@xxxxxxxxxxxxx