[RFC PATCH 0/3] mm/mglru: cacheline follow-ups on the reclaim hot paths

[Ridong Chen]

From: Ridong Chen <chenridong@xxxxxxxxxx>

Joanne's series to reduce cacheline contention around struct
mem_cgroup_per_node [1], now in linux-next, made me look at the MGLRU
side of the same structures, and at the stores MGLRU does on hot paths,
to see whether any of them write a line for no reason. This RFC has the
three candidates that showed a measurable difference. They are
independent of each other.

1. lru_gen_use_mm() writes mm->lru_gen.bitmap on every context switch
   to a user task, although the bitmap is almost always already set.
   For threads of one process spread over several CPUs, the line
   bounces between them. Read first and skip the store: +2-3% context
   switches per CPU for a shared mm, no change for private mms.

2. folio_update_gen() and folio_set_lru_refs() always end with a lock
   cmpxchg, even when the flags do not change. During aging, a folio
   mapped by N processes is visited N times and only the first visit
   changes anything. Skip the no-op cmpxchg: an aging pass over a 2GiB
   file shared by 8-64 processes gets 18-22% faster.

3. lrugen->evicted[] and ->refaulted[] are updated locklessly on
   every eviction and refault, and the file type counters share a line
   with lrugen->enabled/gen/seg/list, which are read on every LRU add
   and by shrink_many(). Move the read-mostly fields up and give the
   counters their own lines: +4.3% fio IOPS with 24 jobs refaulting
   file pages in one memcg, no change at 8 jobs or fewer. This one
   costs a cacheline per memcg per node, and its gain needs many CPUs
   refaulting from a fast backing store, so I am not sure it is worth
   it on its own; feedback welcome.

All numbers are from an i7-13700HX (8P+8E, 24 threads, one socket) with
turbo disabled and the performance governor, on next-20261023 with the
same config for base and patched kernels. Details are in each patch.
The series is rebased onto next-20261104, which includes [1], without
code changes; the struct sizes in patch 3 are updated for it.

Limitations of the testing, where input would be welcome:

- Patch 1 does not show up in will-it-scale context_switch1 or
  sched_yield (explained in the patch), and perf bench sched messaging
  and schbench are too noisy here to resolve 2-3%. The numbers come
  from a small test that pins two sched_yield()ing threads per CPU,
  with a one-process-per-CPU control.

- Patch 2 has no standard benchmark either. In fio with many mmap()
  readers under memcg pressure, reclaim time goes to rmap walks on the
  eviction side and aging barely runs. The numbers come from triggering
  aging through debugfs with force_scan, which is the upper bound. On
  real systems the gain scales with how many processes in the same
  memcg share the hot file pages, e.g. preforked servers or Android
  without per-app memcgs.

- Patch 3 depends on the layout of the containing structs, which
  depends on the config (LRU_GEN_STATS changes the array sizes). I
  checked LRU_GEN_STATS=n only.

The test programs are small C files; I can post them if that helps.

[1] https://lore.kernel.org/20261023195417.1391216-1-joannelkoong@xxxxxxxxx


Ridong Chen (3):
  mm/mglru: skip redundant bitmap store in lru_gen_use_mm()
  mm/mglru: skip the flags cmpxchg when nothing changes
  mm/mglru: move lrugen atomic stats off the read-mostly lines

 include/linux/mm_inline.h |  2 ++
 include/linux/mm_types.h  |  3 ++-
 include/linux/mmzone.h    | 18 ++++++++++--------
 mm/vmscan.c               |  2 ++
 4 files changed, 16 insertions(+), 9 deletions(-)


base-commit: 8b7179210b22f8e48a82c5ceacb9d56a9c4cfec4
-- 
2.34.1