Re: [PATCH v7 0/4] mm: swap: introduce swap tier infrastructure

[Nhat Pham]

On Tue, May 26, 2026 at 11:23 PM Youngjun Park <youngjun.park@xxxxxxx> wrote:
>
> This is v7 of the swap tier series addressing review feedback.
> The cover letter has been simplified.
>
> I revisited the design (see Design Rationale). Since our use case
> fits best with a memcg-based model, the implementation remains
> within memcg and preserves its resource accounting semantics.
>
> Alternatives considered:
>
> 1. A separate sysfs interface under swap. (Workable. But, it would still
>    need to reference memcg paths, and fully decoupling it would add
>    swap-layer logic to manage memcgs, making it secondary option.)
>
> 2. Making the feature non-default.
>
> Other interfaces were also reviewed. Aside from sysfs and BPF,
> the options involve trade-offs and are largely design choices.
> BPF was excluded due to possible disablement on our embedded
> platform, though future extension remains possible.
>
> Overview
> ========
>
> Swap Tiers group swap devices into performance classes (e.g. NVMe,
> HDD, Network) and allow per-memcg selection of which tiers to use.
> This mechanism was suggested by Chris Li.
>
> Design Rationale
> ================
>
> Swap tier selection is attached to memcg. A child cgroup may select a
> subset of the parent's allowed tiers.
>
> This
> - Preserves cgroup inheritance semantics (boundary at parent,
>   refinement at child).
> - Reuses memcg, which already groups processes and enforces
>   hierarchical memory limits.
> - Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
> - Avoids introducing a parallel swap control hierarchy.
>
> Placing tier control outside memcg (e.g., via BPF, syscalls, or
> madvise) would allow swap preference to diverge from the memcg
> hierarchy. Integrating it into memcg keeps the swap policy
> consistent with existing memory ownership semantics. There are
> also real use cases built around memcg.
>
> In the future, this can be extended to other interfaces to cover
> additional use cases.
>
> I believe a memcg-based swap control is a good starting point
> before such extensions.
>
> Use Cases
> =========
>
> #1: Latency separation (our primary deployment scenario)
>   [ / ]
>      |
>      +-- latency-sensitive workload  (fast tier)
>      +-- background workload         (slow tier)
>
> The parent defines the memory boundary.
> Each workload selects a swap tier via memory.swap.tiers according to
> latency requirements.
>
> This prevents latency-sensitive workloads from being swapped to
> slow devices used by background workloads.
>
> #2: Per-VM swap selection (Chris Li's deployment scenario)
>   [ / ]
>      |
>      +-- [ Job on VM ]              (tiers: zswap, SSD)
>             |
>             +-- [ VMM guest memory ]  (tiers: SSD)
>
> The parent (job) has access to both zswap and SSD tiers.
> The child (VMM guest memory) selects SSD as its swap tier via
> memory.swap.tiers. In this deployment, swap device selection
> happens at the child level from the parent's available set.
>
> #3: Tier isolation for reduced contention (hypothetical)
>   [ / ]                    (tiers: A, B)
>      |
>      +-- workload X        (tiers: A)
>      +-- workload Y        (tiers: B)
>
> Each child uses a different tier. Since swap paths are separated
> per tier, synchronization overhead between the two workloads is
> reduced.
>
> Future extension
> ================
>
> #1: Intra-tier distribution policy:
>   Currently, swap devices with the same priority are allocated in a
>   round-robin fashion. Per-tier policy files under
>   /sys/kernel/mm/swap/tiers/ can control how devices within a tier
>   are selected (e.g. round-robin, weighted).
>
> #2: Inter-tier promotion and demotion:
>   Promotion and demotion apply between tiers, not within a single
>   tier. The current interface defines only tier assignment; it does
>   not yet define when or how pages move between tiers. Two triggering
>   models are possible:
>
>   (a) User-triggered: userspace explicitly initiates migration between
>       tiers (e.g. via a new interface or existing move_pages semantics).
>   (b) Kernel-triggered: the kernel moves pages between tiers at
>       appropriate points such as reclaim or refault.
>
> #3: Per-VMA, per-process swap and BPF:
>   Not just for memcg based swap, possible to extend Per-VMA or per-process swap.
>   Or we can use it as BPF program.
>
> Experimentation
> ===============
>
> Tested on our internal platform using NBD as a separate swap tier.
> Our first production's simple usecase.
>
> Without tiers:
> - No selective control over flash wear
> - Cannot selectively assign NBD to specific applications
>
> Cold launch improvement (preloaded vs. baseline):
> - App A: 13.17s -> 4.18s (68%)
> - App B: 5.60s -> 1.12s (80%)
> - App C: 10.25s -> 2.00s (80%)
>
> Performance impact with no tiers configured:
> <1% regression in kernel build and vm-scalability benchmarks
>

Bit late to the party - working on my review backlog right now :)

I see some parallels with this and memory tiering work being done. One
future line of work could be considering how to ensure fairness when
multiple cgroups share same tiers:

https://lwn.net/Articles/1073400/

I can imagine a scenario where one noisy neighbor eagerly swaps first
and occupy all the space in the faster tier(s), pushing the other
colocated tenants to the slower tier(s). We might need to figure out a
way to ensure fairness here (while letting cgroups occupy fast swap
backends opportunistically if there is no resources scarcity).