Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Sun, Sep 6, 2026 at 10:52 PM Kairui Song <ryncsn@xxxxxxxxx> wrote:

Hi Kairui,

Thank you for the kind words and encouragement :) It means a lot
coming from you.

My main concern so far has been - it seems like every time I satisfied
one of your guys' requirements, another pre-requisite is added to the
list. I mean the performance regression and memory overhead concerns
make a lot of sense, especially when vswap is non-optional. But now it
just seems like we're delaying real use case for the fabled perfect
swap system. It gets very exhausting trying to keep expanding the
scope of this.

>
> I asked Baoquan the same thing, see the "Why not make both args
> optional" part:
>
> https://lore.kernel.org/linux-mm/apaw_XmqnEcY8sv7@MiWiFi-R3L-srv/
>
> Where I've ended up is that unbounded growth is a real concern. On a
> host with no memcg limit (root cgroup, and most desktop and embedded
> setups), an unlimited pool means usage can keep growing, with no
> admin visible ceiling at all. I'm not attached to xswap's percent of RAM
> knob specifically, but I do think some kind of bound makes sense.

Technically speaking, virtual swap *is* bounded, i.e 8PB (xarray
induced a limit to the number of clusters it can allocate). But more
importantly, it's effectively capped by the backend it can take:
zswap, swap cache, disk swap, etc.

I'm not entirely certain this will be problem in practice...? Besides,
we can always add new knobs if needs arise (especially since the old
design still exists if unlimited swapping is somehow real), but if we
started out with some knobs that turns out to be unnecessary or
undesirable, removing them and/or educating users to be careful with
using them is harder, in my opinion and experience deploying zswap
across the fleet in the past 4 years.

>
> > 2. Writeback support. Writeback is core functionality for zswap, not an
> >    add-on, and a design needs to account for it from the start. This came
> >    up before, in the discussion around Chris' ghost swapfile RFC [17]: for
> >    a solution here to be acceptable, it has to work with the primary
> >    usecase and support disk writeback. Without it, whatever zswap won't
> >    take (incompressible pages especially) has nowhere to go, and cold
> >    compressed data can never leave RAM.
>
> Agreed completely, and I think this one is settled: whatever lands
> has to support disk writeback in the end. Baoquan mentioned that
> what he sent is just a foundation though. I know talking is cheap
> but still, could it be combined with the writeback support from the
> other side? Or am I missing anything here?

I'm just very nervous about any design that doesn't quite take that
into account and handwave it away into a follow-up :)

>
> > 3. Cgroup charging behavior. vswap/xswap shouldn't be charged against the
> >    swap usage counter. It's fundamentally a different resource from
> >    physical swapfile space, and memory.swap.* should read 0 when nothing is
> >    on disk [18]. I made the longer argument for this in [19].
>
> I personally don't have a strong opinion on this. But I think this
> needs an explicit decision from the group rather than being
> inherited from whichever series merges first. Skipping the charge
> when a device has a special flag isn't hard to do if the code
> abstraction is done well, so this is more of a design decision
> than a technical problem.

I'm fine with discussing about this. I just want to point out that
this is not something I snuck in :) This has been the behavior since
probably the first or second RFC of the first design though, and in
most of them, it is a front-and-centre patch of its own, not hushed
away in one big patch. Folks have had 1.5 years to comment on it (and
still not too late - nothing is merged yet!)

I'd love to hear opinions from other zswap users regarding this :)

>
> > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> >    is already on par with or beating baseline. I like the sparse array
> >    idea, but why are we landing an optimization before the feature itself,
> >    without any A/B data showing the difference matters?
>
> This is where I'd push back a little. To be clear, my concern isn't the
> size of the win, it's that VM_SPARSE isn't purely an optimization. It
> changes the cluster/metadata layout and avoids RCU conventions,
> could help avoid some future sleep while atomic limitations and
> it's tangled with the initial size issue too. So bolting it on afterwards
> could mean rewriting the parts of vswap that touch it and could be
> costly some conflict in ideas. That's why I'd rather we agree on
> it now, not that it has to land first.

I don't think the current code differs significantly from the existing
model, no?

For the most part, RCU played quite nicely with the current swap code.
Practically, it's just:

1. If you have a reference to the swap entry, you can assume entry and
the cluster owning it is safe.

2. If you don't, then do not assume it's safe once you exit the rcu
read section.

What conflicts specifically do you have in mind here :)

>
> Using VM_SPARSE does outsmart my earlier xarray suggestion :) There
> is an interesting early benchmark from Baoquan, VM_SPARSE wasn't there
> yet, but with it we can use clusters natively in the same way. It shows
> about 20% latency cut and 10% throughput gain, which seems a major
> improvement, and that's serial performance, not even mentioning
> scalability yet:
> https://lore.kernel.org/linux-mm/20260707073215.72183-1-baoquan.he@xxxxxxxxx/

I think a lot of these wins come from the fact that the zswap's xarray
specifically is *really* bad in certain paths. For instance, in the
free path (swap_range_free() -> zswap_invalidate()), there is no
batching behavior, so we walk the tree from scratch for each entry
rather than using an xas iterator for the entire batch. That goes away
in the current vswap model, because the xarray manages cluster rather
than entries, so you just do one tree walk + one flat array walk.

You can see some performance numbers in my cover letter - it's
promising on both micro benchmarks, and a production workload that I
throw vswap on :) I've also done an ablative study, where I still let
zswap and swapfile coupled, but only remove the xarray and replace it
with the flat array in swapfile's cluster struct. I observe the same
win, basically (especially on the usemem benchmark).

>
> And it matters more than most micro-optimizations: under heavy
> pressure many workloads spend less than half of system time in
> compression itself, the rest is mm/swap. Swap has had a terrible
> reputation for good reasons, and even with zswap/zram users
> have been scared of the performance.
>
> After the recent refactor and optimization we already noticed many
> users can now run much heavier workloads using compressed memory only
> or fast storages, which is a real breakthrough. Every chunk of
> overhead we remove there unblocks another class of user from using
> compressed memory as a "secondary memory".

I don't necessarily disagree with any of the points you made here.
FWIW, I like the vmalloc array idea too. I have actually put in work
to review Baoquan's patch series, because I'm hopeful that it will be
superior in the end (we're using this infra too so it helps us too :))

I just want to re-iterate that:

1. Vswap is already decently performant.

2. We're not removing the old code, so no one will regress with vswap
(they can just disable it for now). Optimization can be done on top of
it after we land the use cases (which also unblock zswap usage for
many parties who are capped by the storage limitations).

>
> >
> > One thing I do want to be clear about: I'm glad other people care about
> > this problem. Chris' ghost swapfile and Baoquan's xswap are both going
> > after the same set of problems, and that's a good sign. It means this is
> > real and shared, not something only Meta runs into.
> >
> > What's been harder is the shape of the engagement. Alternatives keep
> > getting posted and pushed that don't cover all the requirements, while this
> > series sits without review. I don't think I'm owed anyone's interest in the
> > problems I care about. But I do think working code, with benchmarks and
> > production exposure behind it, deserves a fair hearing next to in-progress
> > proposals.
>
> I'm sorry it has felt that way, and I don't want to hand wave it. For
> what it's worth, I did review it, and I did think it could be merged
> with some adjustments:
>
> https://lore.kernel.org/linux-mm/CAMgjq7D4XsAD4NGDL7FC2kaYAQAP8PDJdn4bpzGZwXYtjEpJ6w@xxxxxxxxxxxxxx/#t
>
> We were also discussing the limitation of the global cluster cache in
> that thread, so I paused my other work for a while and posted the swap
> queue series first. That one is orthogonal on purpose, and helps both
> swap tier and vswap, it reduces the overhead enough to cancel out
> the layering cost due to missing caching for either:
> https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@xxxxxxxxx/
> (this is V2, V1 is earlier - thanks to Lian for keeping this going!)

Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P

I'll spend some time thinking about the design and reviewing it. In
the long run, it will get things more simple, but another thing that
should not be a blocker, correct?

>
> Then xswap appeared with the VM_SPARSE idea and exploring it seems
> necessary because getting the cluster foundation right affects
> everything built above where it went. Not because the vswap series
> wasn't worth reviewing.
>
> > So what I'm asking for: I'd like us to converge rather than keep two series
> > in flight. My preference is that we land vswap first, then build Baoquan's
> > sparse array on top of it as an optimization. That gets the feature in, and
> > by then we'd have the A/B data to show whether the sparse array actually
> > beats the xarray.
> >
> > If you think that's the wrong order, I'd genuinely like to understand why -
> > after 17 months and 10 revisions I still don't have a clear picture of the
> > objection.
>
> I fully agree we must converge. At the moment I don't think one can
> can settle that alone as there are several of us reviewing both. I'll try
> to be more concrete about what I think has to be agreed first.
>
> The two interfaces aren't equivalent, xswap has its limitations, but a
> couple of xswap's properties look genuinely better to me. Runtime
> on/off rather than a boot parameter is better, more flexibility and

I would also like to point out - I *do* have a runtime parameter in an
older version of vswap (v3). There is also not a lot of engagement
from folks in that version - no one expressed any interest in that
runtime knob in particular. Only Johannes made the case that a boot
parameter is superior, so I switched over. I think his reasoning makes
sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
(which in turns remove a bunch of useless stubs - saving 100 LoCs), so
I figure folks will prefer less code and less userspace decisions :)

v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@xxxxxxxxx/

Regardless - do you have a specific use case in mind where a runtime
knob would be useful?

> doesn't conflict with what vswap is trying to do. Some other of these
> differences involve design decisions rather than just surface knobs,
> and those are the parts we're stuck with once they ship.
>
> So a few current interface difference of vswap and xswap I've noticed are:
>
>  - default size: somehow also tangled with VM_SPARSE, which
> needs an initial virtual value. It's super cheap if not zero cost
> (the only cost is occupying some vmalloc area, which is fine
> I guess?). But we still have to pick a number, could be large
> enough to satisfy everyone?

8PB? ;)

I think choosing a meaningful number that covers from O(dozen of GB)
to O(terrabytes) scale machine, with different workload shapes, access
patterns, memory compressibility, etc. is an unsolvable problem. If
it's cheap, why don't we just max out?

>  - boot vs runtime toggle: I believe a runtime toggle is better. It
> can still be tuned on boot via cmdline, but it offers the extra
> ability to turn it on without modifying a base VM boot image or
> kernel.
>  - growth / limits: as talked above.
>  - charging semantics
>  - there could be other longterm design decisions on top of these as
> we talked in above.
>
> Once these settle, I think the picture will be clean. Your writeback
> logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> previous preparation with swap ops give us a structural foundation. A
> clean long-term architecture means getting the core structure right,
> let's try not treating this as a race between series.

I've reached out to Baoquan as well, but I figured there's value in
both discussions in big and small groups, and especially since there's
a chance some of us might not be coming to LPC. Let's see what we come
up with.