Re: Path forward for Virtualized Swap?

[Kairui Song]

On Sat, Sep 5, 2026 at 5:14 AM Nhat Pham <nphamcs@xxxxxxxxx> wrote:
>
> To Kairui, Johannes, Yosry, Chris, David, Andrew, and other (z)swap folks -
> I need some help here.

Hello Nhat,

>
> I'd like to ask for a decision, or failing that, the specific thing that
> would get us to one. I'd much rather work against a concrete objection than
> keep respinning into silence.
>
> Quick recap: today a swapped-out page holds a disk slot even when the data
> never reaches disk, like zswap or zero-filled pages. My patch series
> (latest version at [10]) puts a virtual entry in the page table instead and
> resolves the backing separately, so zswap stops reserving disk it never
> uses and can work on machines with no storage swap space at all.
>
> I've been working on this since April 2025, going through 10 revisions (6
> on the original design ([1] through [6]) and 4 on the swap-table-based
> rewrite ([7] through [10])). I also gave a talk on it at LPC [11].

Thanks for the hard work! And I mean that. Upstream work at this scale
does take a long time and it isn't a reflection on the series: the swap
cache optimization idea also took me over 2 years and changed direction
multiple times and split multiple times before it landed as
the swap table series, and the readahead unification part is
still in flight after nearly 3 years. It's frustrating and I don't think
anyone enjoys it. I mentioned this once with a longer history:

https://lore.kernel.org/linux-mm/CAMgjq7CF9RgnZCAS-+Gv0LAvkzzHk4jiok+_6-KOFw-o+s8E_g@xxxxxxxxxxxxxx/

There is also a quote from LSF/MM/BPF saying it's for improvements to
"find their way into the mainline kernel and Linux distributions in the
next 24-48 months.", suffering souls everywhere :P

>
> Throughout the process, I've been very receptive to reviewers - including
> parties whose use cases are very different from my own, like zram users and
> the ghost-swapfile-style deployments that want zswap with no backing disk
> at all - and the current design reflects that:
>
> * Kairui asked that this be optional, not a replacement [12]. It is now:
>   vswap is opt-in behind the vswap= cmdline parameter, and with vswap=off
>   the existing swap paths aren't touched at all.
>
> * Kairui also pointed out that if I kept shrinking the metadata I'd "end up
>   reimplementing the swap table format" [12]. I took that feedback, and
>   reworked the entire series on top of swap table.
>
> * Per Chris' requests, I've figured out ways to minimize memory overhead
>   [13]. On a zswap-only setup there's almost 0 overhead now, because the
>   vtable is merged with zswap's xarray rather than sitting on top of it.
>   The indirection is paid for by the tree we delete.
>
> * Johannes pointed out that a physical cluster backing only vswap entries
>   still had to allocate a memcg_table, which was an overhead regression
>   against my older design on the zswap writeback case [14] (since a swap
>   entry occupies multiple levels). I've also fixed this.
>
> * I've spent a lot of time chasing regressions on setups we don't even run
>   at Meta - zram in particular [6]. And if anyone's still worried after all
>   that, vswap=off keeps the old swap code available.

Acknowledged, specifically reworking the whole series on top of swap
table, the memcg table fix, and chasing zram regressions you don't even
run were all real work on cases that don't benefit your fleet. Thank
you!

Roughly 80% of my own time also goes to things that don't run on my
fleet either. Desktop and Android (I do use them though), lowend and even
32bits. Upstream covers every usage case as the baseline and it's tiring,
I know what it costs. :)

> Numbers are in the v4 cover letter [10]. In terms of performance, vswap is
> on par or better than baseline on zswap across memhog, usemem and kernel
> build, and vswap=off tracks baseline within noise. I've even included an
> A/B experiment on a real production workload (Instagram/django): throughput
> on par, latency slightly better, and the swapfile footprint for every zswap
> user on the host goes away.
>
> Now, on xswap. Baoquan's working on a series [15] that covers some of the
> same ground, and the VM_SPARSE cluster_info idea in it is genuinely good.

My read is that xswap and vswap differ mostly in the interface, and that
both could sit on VM_SPARSE for cluster/metadata management.
Simpler and faster. There are some other issues though, see below.

> I've been reviewing that lineage since July [16] and I'd like whatever
> lands to end up with the best parts of both. From my perspective the
> differences are:
>
> 1. Userspace knobs. xswap asks the admin for a size (a percent of RAM) plus
>    a per-device limit to tune afterwards. I'm not aware of any use case
>    that needs those, and I don't think users have a good way to answer the
>    question anyway - sizing swap for compressed memory depends on memory
>    size, workload, and compression ratio all at once. That's precisely the
>    provisioning problem vswap exists to remove. The kernel should be as
>    transparent and dynamic as possible here, and not add knobs unless
>    there's a use case for them.

I asked Baoquan the same thing, see the "Why not make both args
optional" part:

https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/

Where I've ended up is that unbounded growth is a real concern. On a
host with no memcg limit (root cgroup, and most desktop and embedded
setups), an unlimited pool means usage can keep growing, with no
admin visible ceiling at all. I'm not attached to xswap's percent of RAM
knob specifically, but I do think some kind of bound makes sense.

> 2. Writeback support. Writeback is core functionality for zswap, not an
>    add-on, and a design needs to account for it from the start. This came
>    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
>    a solution here to be acceptable, it has to work with the primary
>    usecase and support disk writeback. Without it, whatever zswap won't
>    take (incompressible pages especially) has nowhere to go, and cold
>    compressed data can never leave RAM.

Agreed completely, and I think this one is settled: whatever lands
has to support disk writeback in the end. Baoquan mentioned that
what he sent is just a foundation though. I know talking is cheap
but still, could it be combined with the writeback support from the
other side? Or am I missing anything here?

> 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
>    swap usage counter. It's fundamentally a different resource from
>    physical swapfile space, and memory.swap.* should read 0 when nothing is
>    on disk [18]. I made the longer argument for this in [19].

I personally don't have a strong opinion on this. But I think this
needs an explicit decision from the group rather than being
inherited from whichever series merges first. Skipping the charge
when a device has a special flag isn't hard to do if the code
abstraction is done well, so this is more of a design decision
than a technical problem.

> 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
>    is already on par with or beating baseline. I like the sparse array
>    idea, but why are we landing an optimization before the feature itself,
>    without any A/B data showing the difference matters?

This is where I'd push back a little. To be clear, my concern isn't the
size of the win, it's that VM_SPARSE isn't purely an optimization. It
changes the cluster/metadata layout and avoids RCU conventions,
could help avoid some future sleep while atomic limitations and
it's tangled with the initial size issue too. So bolting it on afterwards
could mean rewriting the parts of vswap that touch it and could be
costly some conflict in ideas. That's why I'd rather we agree on
it now, not that it has to land first.

Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
yet, but with it we can use clusters natively in the same way. It shows
about 20% latency cut and 10% throughput gain, which seems a major
improvement, and that's serial performance, not even mentioning
scalability yet:
https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@xxxxxxxxx/

And it matters more than most micro-optimizations: under heavy
pressure many workloads spend less than half of system time in
compression itself, the rest is mm/swap. Swap has had a terrible
reputation for good reasons, and even with zswap/zram users
have been scared of the performance.

After the recent refactor and optimization we already noticed many
users can now run much heavier workloads using compressed memory only
or fast storages, which is a real breakthrough. Every chunk of
overhead we remove there unblocks another class of user from using
compressed memory as a "secondary memory".

>
> One thing I do want to be clear about: I'm glad other people care about
> this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> after the same set of problems, and that's a good sign. It means this is
> real and shared, not something only Meta runs into.
>
> What's been harder is the shape of the engagement. Alternatives keep
> getting posted and pushed that don't cover all the requirements, while this
> series sits without review. I don't think I'm owed anyone's interest in the
> problems I care about. But I do think working code, with benchmarks and
> production exposure behind it, deserves a fair hearing next to in-progress
> proposals.

I'm sorry it has felt that way, and I don't want to hand wave it. For
what it's worth, I did review it, and I did think it could be merged
with some adjustments:

https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@xxxxxxxxxxxxxx/#t

We were also discussing the limitation of the global cluster cache in
that thread, so I paused my other work for a while and posted the swap
queue series first. That one is orthogonal on purpose, and helps both
swap tier and vswap, it reduces the overhead enough to cancel out
the layering cost due to missing caching for either:
https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@xxxxxxxxx/
(this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Then xswap appeared with the VM_SPARSE idea and exploring it seems
necessary because getting the cluster foundation right affects
everything built above where it went. Not because the vswap series
wasn't worth reviewing.

> So what I'm asking for: I'd like us to converge rather than keep two series
> in flight. My preference is that we land vswap first, then build Baoquan's
> sparse array on top of it as an optimization. That gets the feature in, and
> by then we'd have the A/B data to show whether the sparse array actually
> beats the xarray.
>
> If you think that's the wrong order, I'd genuinely like to understand why -
> after 17 months and 10 revisions I still don't have a clear picture of the
> objection.

I fully agree we must converge. At the moment I don't think one can
can settle that alone as there are several of us reviewing both. I'll try
to be more concrete about what I think has to be agreed first.

The two interfaces aren't equivalent, xswap has its limitations, but a
couple of xswap's properties look genuinely better to me. Runtime
on/off rather than a boot parameter is better, more flexibility and
doesn't conflict with what vswap is trying to do. Some other of these
differences involve design decisions rather than just surface knobs,
and those are the parts we're stuck with once they ship.

So a few current interface difference of vswap and xswap I've noticed are:

 - default size: somehow also tangled with VM_SPARSE, which
needs an initial virtual value. It's super cheap if not zero cost
(the only cost is occupying some vmalloc area, which is fine
I guess?). But we still have to pick a number, could be large
enough to satisfy everyone?
 - boot vs runtime toggle: I believe a runtime toggle is better. It
can still be tuned on boot via cmdline, but it offers the extra
ability to turn it on without modifying a base VM boot image or
kernel.
 - growth / limits: as talked above.
 - charging semantics
 - there could be other longterm design decisions on top of these as
we talked in above.

Once these settle, I think the picture will be clean. Your writeback
logic and swap table integration are solid, while Baoquan's VM_SPARSE,
previous preparation with swap ops give us a structural foundation. A
clean long-term architecture means getting the core structure right,
let's try not treating this as a race between series.