[PATCH v2 0/2] mm/shrinker: per-memcg shrinker units in an xarray, allocated on demand

[Konstantin Khorenko]

Every memory cgroup allocates, per NUMA node, one shrinker_info_unit
per 64 memcg-aware shrinker ids up to shrinker_nr_max, whether or not
it ever charges an object to those shrinkers, plus a pointer array
sized for shrinker_nr_max. Every superblock registers such a shrinker,
and shrinker_nr_max never comes down, so a host that mounts many
filesystems and runs many cgroups pays for every superblock in every
cgroup, live or dying, and keeps paying for new cgroups after the
mounts are gone.

Patch 1 keeps the units in an xarray per memcg and node instead of the
pointer array. A new unit is inserted where it belongs: nothing is
reallocated, copied or published under RCU when shrinker_nr_max grows,
and the readers find a unit with xa_load() or walk the present ones
with xa_for_each(). The units are still allocated up front, so this
patch changes no behaviour.

Patch 2 allocates a unit only when the cgroup gets a list on that
shrinker's list_lru, which is where map bits are set: a cgroup pays for
the shrinkers it holds objects of, not for every superblock on the host.
The install is an xarray insert with the gfp of the list allocation;
there is no array to race with, so no lock beyond the xarray's. The root
memcg keeps its units for the deferred work of global reclaim and is the
only memcg a shrinker registration still visits.

Why an xarray and not the on-demand array
-----------------------------------------

Allocating the units on demand removes most of their memory with either
container. The pointer array keeps three costs that scale with
shrinker_nr_max x cgroups x nodes instead of with what a cgroup uses:

- the array itself, per cgroup and node, also in cgroups that hold
  nothing;
- a registration that crosses a unit boundary reallocates and copies
  the array of every cgroup on every node under shrinker_mutex, which
  every other mount and umount waits for;
- shrink_slab_memcg() walks shrinker_nr_max / 64 indexes per cgroup
  and node on every reclaim pass, whether the cgroup has objects or
  not.

The xarray holds only the units a cgroup has, so none of the three
grows with the number of mounts. It is also less code: mm/shrinker.c
ends up shorter than before the series.

Measured on a 48-CPU, 4-node host with 50000 tmpfs mounts
(shrinker_nr_max 50112) and 20000 live, empty memory cgroups. base is
v7.3-rc7 with the kmem_cache patch, where every cgroup has every unit;
both on-demand variants sit on top of it. "old array on demand" is
patch 2/2 of "mm/shrinker: make per-memcg shrinker units cheaper" v2:

+--------------------------------------+-----------+-----------+--------+
|                                      | base      | old array | xarray |
|                                      |           | on demand |        |
+--------------------------------------+-----------+-----------+--------+
| unreclaimable slab per empty cgroup  | 1.6 MB    | 65 KB     | 34 KB  |
|   of it, the per-cgroup array        | 32 KB     | 32 KB     | 0      |
+--------------------------------------+-----------+-----------+--------+
| 640 mounts crossing 10 unit          |           |           |        |
| boundaries, 20000 cgroups alive      | 1.41 s    | 1.31 s    | 0.10 s |
|   the same 640 mounts, no cgroups    | 0.08 s    | 0.08 s    | 0.07 s |
+--------------------------------------+-----------+-----------+--------+
| echo 2 > drop_caches: shrink_slab()  |           |           |        |
| for every online cgroup, the walk    |           |           |        |
| kswapd does on every pass            | 23 s      | 11.3 s    | 1.3 s  |
|   of it, shrink_slab_memcg() per     |           |           |        |
|   cgroup and node                    | 22.8 us   | 10.7 us   | 0.5 us |
+--------------------------------------+-----------+-----------+--------+

This v2 on v7.4-rc1 gives the same xarray numbers on the same host.

Assisted-by: LLM
---
This replaces patch 2/2 of "mm/shrinker: make per-memcg shrinker units
cheaper" v2 with an xarray variant. Patch 1/2 of that series is in
v7.4-rc1 as commit faaf8b406c517 ("mm/shrinker: allocate
shrinker_info_unit from a dedicated kmem_cache").

Changes in v2:
- Rebased on v7.4-rc1, which has the kmem_cache patch this series was
  built on.
- Patch 1: memcg_struct_check() asserts shrinker_units in the
  read-mostly group, where shrinker_info was.
- Patch 2: free_shrinker_info() no longer takes shrinker_mutex, as a
  registration only visits the root memcg.
- Patch 2: __list_lru_init() keeps the SHRINKER_MEMCG_AWARE check of
  commit 6a100811c7c5f ("mm/list_lru: don't copy stale shrinker id from
  non-memcg-aware shrinkers") next to its own check for a per-memcg
  list.
- Cover letter: why an xarray and not the on-demand array, measured on
  a 48-CPU host, from the v1 thread (Harry Yoo, Andrew Morton).
- Move Assisted-by from the patches to the cover letter, after commit
  1e035297ffdad ("docs: remove Assisted-by references").
- Link to v1: https://patch.msgid.link/20261019-b4-prep-vstor-144717-per-memcg-shrinker-units-on-demand-xarray-v1-0-64ed1f7957c9@xxxxxxxxxxxxx

---
Konstantin Khorenko (2):
      mm/shrinker: keep the per-memcg shrinker units in an xarray
      mm/shrinker: allocate the per-memcg shrinker units on demand

 include/linux/memcontrol.h |   7 +-
 include/linux/shrinker.h   |   6 -
 mm/list_lru.c              |  20 ++-
 mm/memcontrol.c            |   8 +-
 mm/shrinker.c              | 377 +++++++++++++++++----------------------------
 5 files changed, 164 insertions(+), 254 deletions(-)
---
base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
change-id: 20261019-b4-prep-vstor-144717-per-memcg-shrinker-units-on-demand-xarray-fbfd27a8cd69

Best regards,
--  
Konstantin Khorenko,
Virtuozzo Linux Kernel Team