Re: [RFC PATCH v3 0/4] mm: enable lru cache for smaller large folios

[Garg, Shivank]

On Wed, 2026-08-19 at 06:59 +0800, Barry Song (Xiaomi) wrote:
> This patchset enables the per-CPU LRU cache for large folios with fewer
> than `FOLIO_BATCH_SIZE` (31) pages. It also limits each per-CPU LRU cache
> to at most `FOLIO_BATCH_SIZE` pages to avoid negatively affecting
> accounting and memory reclamation pressure.
> 
> This is particularly beneficial on systems that use relatively small
> large folios. For larger folios, the benefit is likely to be smaller
> because far fewer folios are expected to contend for the LRU cache.
> 
> * Use the following microbenchmark:
> 
>  #include <pthread.h>
>  #include <sys/mman.h>
>  #include <string.h>
> 
>  #define NUM_THREADS     20
>  #define MEM_SIZE        (16 * 1024 * 1024)
>  #define LOOP_COUNT      1000
> 
>  void* thread_worker(void* arg) {
>         void *addr = mmap(NULL, MEM_SIZE, PROT_READ | PROT_WRITE,
>                         MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> 
>         for (int i = 0; i < LOOP_COUNT; i++) {
>                 for(int j = 0; j < MEM_SIZE; j += 4096)
>                         *(unsigned char *)(addr + j) = 0x55;
>                 madvise(addr, MEM_SIZE, MADV_DONTNEED);
>         }
> 
>         munmap(addr, MEM_SIZE);
>         pthread_exit(NULL);
>  }
> 
>  int main() {
>         pthread_t threads[NUM_THREADS];
> 
>         for (long t = 0; t < NUM_THREADS; t++) {
>                 pthread_create(&threads[t], NULL, thread_worker, (void*)t);
>         }
> 
>         for (int t = 0; t < NUM_THREADS; t++) {
>                 pthread_join(threads[t], NULL);
>         }
> 
>         return 0;
>  }
> 
> w/o patch:
> 
> $ time ./a.out 
> 
> real	0m17.760s
> user	0m8.207s
> sys	5m46.067s
> 
> perf lock report:
>                 Name   acquired  contended     avg wait   total wait     max wait     min wait 
> 
>                        21414857   21414857     30.07 us     10.73 m     101.94 us       880 ns 
>                           42890      42890     11.29 us    484.21 ms    123.08 us       923 ns 
>                            1678       1678      1.57 us      2.63 ms      5.77 us       985 ns 
>                              52         52      1.46 us     75.98 us      2.69 us      1.07 us 
>                              18         18     17.82 us    320.69 us     58.92 us      1.62 us 
>                              18         18      2.36 ms     42.52 ms      5.31 ms      1.47 us 
>            rcu_state         12         12      1.98 us     23.78 us      2.62 us      1.47 us 
>            rcu_state          9          9      1.72 us     15.49 us      2.27 us      1.34 us 
>                               2          2      2.42 us      4.83 us      2.48 us      2.36 us 
> 
> w/ patch:
> 
> $ time ./a.out 
> 
> real	0m16.292s
> user	0m8.587s
> sys	5m13.787s
> 
> perf lock report
> 
>                 Name   acquired  contended     avg wait   total wait     max wait     min wait 
> 
>                         2641286    2641286     46.87 us      2.06 m     107.92 us      1.01 us 
>                          235275     235275     10.57 us      2.49 s     115.32 us      1.02 us 
>            rcu_state       1982       1982      7.58 us     15.02 ms     31.72 us      1.26 us 
>            rcu_state       1929       1929      7.42 us     14.32 ms     30.50 us      1.43 us 
>                              86         86      3.61 us    310.19 us     13.89 us      1.15 us 
>                              20         20     19.70 us    394.03 us    108.07 us      2.15 us 
>        tasklist_lock          1          1      2.13 us      2.13 us      2.13 us      2.13 us 
> 
> 
> * Build the kernel in a 1 GiB memcg by -j20 with zRAM configured as swap:
> 
> w/o patch:
> 
> Perf lock report:
> 
>                 Name   acquired  contended     avg wait   total wait     max wait     min wait 
> 
>                          782337     782337     17.61 us     13.78 s     402.22 us       896 ns 
>                           55459      55459     19.48 us      1.08 s     117.59 us      1.01 us 
>                            7826       7826      8.01 us     62.68 ms     19.03 us       887 ns 
>                            5324       5324      7.83 us     41.68 ms     37.59 us      1.11 us 
>            rcu_state       5144       5144      6.27 us     32.23 ms     25.17 us      1.58 us 
>            rcu_state       5142       5142      6.35 us     32.67 ms     30.55 us      1.48 us 
>                            3855       3855      8.74 us     33.68 ms     42.32 us       996 ns 
>                            2770       2770      9.22 us     25.55 ms     27.83 us       914 ns 
>                            2342       2342      5.88 us     13.77 ms    318.75 us      1.00 us 
> time:
> 
> *** Executing round 0 ***
> 
> real    1m46.847s
> user    25m10.848s
> sys     2m57.282s
> 
> *** Executing round 1 ***
> 
> real    1m46.423s
> user    25m10.072s
> sys     2m54.348s
> 
> *** Executing round 2 ***
> 
> real    1m46.308s
> user    25m13.800s
> sys     2m58.963s
> 
> *** Executing round 3 ***
> 
> real    1m46.155s
> user    25m18.079s
> sys     2m59.721s
> 
> *** Executing round 4 ***
> 
> real    1m45.980s
> user    25m15.493s
> sys     2m56.959s
> 
> w/ patch:
> 
> perf lock report
>                 Name   acquired  contended     avg wait   total wait     max wait     min wait 
> 
>                          202647     202647     34.27 us      6.94 s     467.82 us      1.18 us 
>                           51819      51819     16.26 us    842.46 ms    245.55 us       885 ns 
>      inode_hash_lock      15169      15169      8.58 us    130.21 ms     23.23 us      1.04 us 
>                            5306       5306      7.54 us     40.03 ms     31.53 us      1.17 us 
>            rcu_state       4945       4945      6.97 us     34.47 ms     30.00 us      1.76 us 
>            rcu_state       4899       4899      7.03 us     34.42 ms     24.65 us      1.42 us 
>                            3923       3923      8.56 us     33.57 ms     27.51 us      1.08 us 
>                            2212       2212      5.23 us     11.56 ms    222.79 us       965 ns 
>                            1412       1412      6.35 us      8.97 ms     23.09 us      1.77 us 
> 
> time:
> 
> *** Executing round 0 ***
> 
> real	1m46.463s
> user	25m17.448s
> sys	2m49.524s
> 
> *** Executing round 1 ***
> 
> real	1m46.274s
> user	25m12.178s
> sys	2m53.522s
> 
> *** Executing round 2 ***
> 
> real	1m46.362s
> user	25m13.115s
> sys	2m53.005s
> 
> *** Executing round 3 ***
> 
> real	1m46.036s
> user	25m17.627s
> sys	2m53.477s
> 
> *** Executing round 4 ***
> 
> real	1m46.329s
> user	25m15.130s
> sys	2m51.508s
> 
> -RFC v3:
>   * Rather than hard-coding `< COSTLY_ORDER` to enable the lru_cache,
>     allow the lru_cache for larger orders as long as the folio contains
>     fewer than `FOLIO_BATCH_LRU` pages. Also limit the total number of
>     pages held in the lru_cache. Hugh may prefer this approach as well.
>   * Clean up the comments and `if` conditions in "mm: improve large folio
>     reuse for LRU-cached folios" based on David's feedback. Thanks!
>   * Properly drain the lru_cache when splitting folios. Thanks to Sashiko
>     and David!
> 
> -RFC v2:
>  * Make __wp_can_reuse_large_anon_folio() aware of LRU-cached large
>    folios. As David pointed out, it currently does not account for
>    large folios residing in the per-CPU LRU cache.
>    https://lore.kernel.org/all/20260709081536.82768-1-baohua@xxxxxxxxxx/
> 
> Barry Song (Xiaomi) (4):
>   mm: allow smaller large folios to use lru_cache
>   mm: improve large folio reuse for LRU-cached folios
>   mm: drain LRU cache if necessary for splitting large folios
>   mm: batch lru_cache draining in deferred_split_scan
> 
>  include/linux/folio_batch.h | 25 +++++++++++++++++++++++++
>  mm/folio.c                  | 10 +++++++++-
>  mm/huge_memory.c            | 23 +++++++++++++++++++----
>  mm/internal.h               |  4 ++--
>  mm/memory.c                 |  6 ++++++
>  5 files changed, 61 insertions(+), 7 deletions(-)


Thanks for this series.
I tested this against a Graph500 regression with 16K mTHP.
The series removes the regression and reduces folio_lruvec_lock_irqsave contention to approximately the 4K rate.


Tested-by: Shivank Garg <shivankg@xxxxxxx>


System and Workload
===================

AMD EPYC Zen 5 System, 2 sockets, 160 cores/socket,
SMT on -> 320 cores / 640 threads
2 NUMA nodes, ~512 GB/node
node 0 CPUs: 0-159,320-479

Base: v7.3-rc2+ (50d05c7c76c9)
Runtime: performance governor, preempt=full (lazy)

Workload: Graph500, Scale 27, edgefactor 16
Metric: median_time in ms (lower is better)

Bind to node 0 with numactl -C 0-159,320-479 -m 0


Performance
===========

                  Base       Patched    delta
4K                109.60     109.11     -0.4%
mTHP-16K          144.10      99.00    -31.3%
mTHP-64K          104.75     104.25     -0.5%

mTHP-16K was ~31% slower than THP-never, due to lock contentions.
With your series, mTHP-16 performs ~9% faster than THP-never.


The dominant folio_lruvec_lock_irqsave entries were:

                   Base                  Patched
               contended  wait         contended  wait
4K               834603   2.41 m         844193   2.61 m
mTHP-16K        7008834   28.88 m        840695   4.87 m
mTHP-64K         188144   752 ms         40874    93.46 ms


Thanks,
Shivank