Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Fri, Sep 11, 2026 at 9:10 AM Kairui Song <ryncsn@xxxxxxxxx> wrote:
>
>> >
> > >
> > > > 4. Data structure (xarray vs sparse vmalloc array). Even with xarray, vswap
> > > >    is already on par with or beating baseline. I like the sparse array
> > > >    idea, but why are we landing an optimization before the feature itself,
> > > >    without any A/B data showing the difference matters?
> > >
> > > This is where I'd push back a little. To be clear, my concern isn't the
> > > size of the win, it's that VM_SPARSE isn't purely an optimization. It
> > > changes the cluster/metadata layout and avoids RCU conventions,
> > > could help avoid some future sleep while atomic limitations and
> > > it's tangled with the initial size issue too. So bolting it on afterwards
> > > could mean rewriting the parts of vswap that touch it and could be
> > > costly some conflict in ideas. That's why I'd rather we agree on
> > > it now, not that it has to land first.
> >
> > I don't think the current code differs significantly from the existing
> > model, no?
> >
> > For the most part, RCU played quite nicely with the current swap code.
> > Practically, it's just:
> >
> > 1. If you have a reference to the swap entry, you can assume entry and
> > the cluster owning it is safe.
> >
> > 2. If you don't, then do not assume it's safe once you exit the rcu
> > read section.
> >
> > What conflicts specifically do you have in mind here :)
>
> Yeah you're right. I'm just a bit worried if that RCU period will be
> extended, and it's always better to make the convention as simple as
> possible. I'm already trying to get rid of the local lock, in that
> swap queue series, that is not so painful and still a bit troublesome
> for a few new use cases.

Completely understood. Your inquiry made me give it a double take - I
took Baoquan's code and hacked it on top of my vswap patch series, and
it seems reasonably simple :)

https://lore.kernel.org/all/20260910232704.3364879-1-nphamcs@xxxxxxxxx/

Seems to slot in quite nicely. I mean, we're converging to the *same*
architecture, just different data structure. We can switch down the
line - it just would require a lot more testing, benchmarking,
evaluating, etc. to adopt this.

> >
> > I think a lot of these wins come from the fact that the zswap's xarray
> > specifically is *really* bad in certain paths. For instance, in the
> > free path (swap_range_free() -> zswap_invalidate()), there is no
> > batching behavior, so we walk the tree from scratch for each entry
> > rather than using an xas iterator for the entire batch. That goes away
> > in the current vswap model, because the xarray manages cluster rather
> > than entries, so you just do one tree walk + one flat array walk.
>
> Right but with vswap replacing zswap, isn't the performance is still
> basically unchanged according to your test (or just a little bit
> faster)? My test still doesn't show a measurable improvement.

Yeah in my test it also improves in some cases but not others. But
FWIW, the xarray-less approach does not outperform the xarray-version
yet :) I tested with the old coupled design, only replacing xarray
with a per-cluster flat array - so basically Baoquan's design but
without the vmalloc overhead. It performs basically on par with the
xarray (where it wins, it matches xarray's wins, etc.).

And I tried my best to keep the old code intact (all the
swap_is_vswap() that Baoquan disliked was for that too haha). That
make sure physical-swapfile-only users, and those who still disable
vswap at boottime, has no overhead.

>
> > zswap and swapfile coupled, but only remove the xarray and replace it
> > with the flat array in swapfile's cluster struct. I observe the same
> > win, basically (especially on the usemem benchmark).
>
> Yeah, merging the zswap xarray into cluster should buy a lot of
> performance gain, the only concern is the cluster it self is in a
> xarray tree so if your vswap space gets huge, could that somehow
> become an issue? Zswap used a split xarray, and swap also used to have
> a 64M partition for the swap cache xarray, which is gone after the
> swap table.

Hmm not in my testing so far, both in production and synth-benchs, and
we have some pretty big swap users.

As long as the xarray does not lock us into any model, we can always
optimize further IMHO. There are several axes of scaling - another one
is raising the size of the cluster.

>
> ...
>
> > > We were also discussing the limitation of the global cluster cache in
> > > that thread, so I paused my other work for a while and posted the swap
> > > queue series first. That one is orthogonal on purpose, and helps both
> > > swap tier and vswap, it reduces the overhead enough to cancel out
> > > the layering cost due to missing caching for either:
> > > https://lore.kernel.org/linux-mm/20260829-swap-pcp-priq-v2-resend-0-68d3d925578c@xxxxxxxxx/
> > > (this is V2, V1 is earlier - thanks to Lian for keeping this going!)
> >
> > Ah that! I sort of tabled that as a nice-idea-but-still-optimization :P
> >
> > I'll spend some time thinking about the design and reviewing it. In
> > the long run, it will get things more simple, but another thing that
> > should not be a blocker, correct?
>
> Right, definitely not a blocker. I think the swap queue could be
> merged after tiering and other stuff, as an optimization, just to
> demonstrate that we don't need another layer of cache; we can just use
> the devices directly without any cache above them. It will also
> deprecate plist; with the only plist rotate user is gone, we can get
> rid of a large chunk of common plist code.

Look forward to seeing more of this! It was a personal request from
me, so I appreciate that you look into this.

>
> ...
>
> > > The two interfaces aren't equivalent, xswap has its limitations, but a
> > > couple of xswap's properties look genuinely better to me. Runtime
> > > on/off rather than a boot parameter is better, more flexibility and
> >
> > I would also like to point out - I *do* have a runtime parameter in an
> > older version of vswap (v3). There is also not a lot of engagement
> > from folks in that version - no one expressed any interest in that
> > runtime knob in particular. Only Johannes made the case that a boot
> > parameter is superior, so I switched over. I think his reasoning makes
> > sense at the time FWIW, and it also removes the need for CONFIG_VSWAP
> > (which in turns remove a bunch of useless stubs - saving 100 LoCs), so
> > I figure folks will prefer less code and less userspace decisions :)
> >
> > v3: https://lore.kernel.org/all/20260806184254.3790858-1-nphamcs@xxxxxxxxx/
>
> Ah, thanks for the reminder.
>
> > Regardless - do you have a specific use case in mind where a runtime
> > knob would be useful?
>
> A boot time bool will cause real trouble for us, at least. We have
> many VMs with fixed boot images and fixed cmdlines. However, the VM
> itself is used by different customers, some are ZRAM only users, and
> quite a lot don't want any kind of compression (e.g. their data is
> dense compression is just waste of CPU) and prefer plain swap. So I do
> hope we can have a runtime tunable instead of modifying the cmdline
> and rebooting or rebuilding the base image for different cases, and we
> have already seen that a runtime toggle isn't hard at all.

Ah yeah I see. But before moving back to runtime parameter, do you
think we can come up with a heuristics for this that have does not
require userspace input? In v4, I'm doing this at folio_alloc_swap()
time:

a. If the cgroup enables zswap, then always go vswap.

b. If zswap is not available for the cgroup, then bypass vswap, and go
straight for physical swap entries. My intuition is that we are not
buying anything for these zram users, as we need a physical swap slot
anyway. Thanks to your design, both can co-exist in the system now -
that's something I actually did not envision possible in the original
version of vswap. Pretty cool eh :)

Would something like this fix it for you? ZRAM users will not get
vswap indirection overhead at all, because it would bypass vswap :)
Down the line we can revisit this decision - for e.g, if there is a
use case for vswap-on-top-of-swapfile. There might be other interface
that makes more sense.

>
> > > doesn't conflict with what vswap is trying to do. Some other of these
> > > differences involve design decisions rather than just surface knobs,
> > > and those are the parts we're stuck with once they ship.
> > >
> > > So a few current interface difference of vswap and xswap I've noticed are:
> > >
> > >  - default size: somehow also tangled with VM_SPARSE, which
> > > needs an initial virtual value. It's super cheap if not zero cost
> > > (the only cost is occupying some vmalloc area, which is fine
> > > I guess?). But we still have to pick a number, could be large
> > > enough to satisfy everyone?
> >
> > 8PB? ;)
>
> Right I saw that at V1 :).
>
> I'm not sure if this is the reason that is in conflict with VM_SPARSE
> in your mind? Techically, at least for 64 bit, we can just apply the
> same limit to VM_SPARSE, for 64 bit machines we have a lot of virtual
> space to consume and that should be fine (VM_SPARSE is just the
> cluster map, which is much smaller compared to swap space).
>
> And another thing is do we really need that much in any case? Will
> things like, e.g. 64 times the physical memory (max pfn) be enough? I
> know Baoquan suggest just use the physical memory size. So that could
> be tunable, it's really easy to tune, just a number.

Potentially, but we have many users at Meta. There's a huge diversity
of machine types, workingset size, access patterns (both frequency and
file:anon split), compressibility, etc.

Tuning even just one number, but for a combinatorial explosions of
these factors, cost a lot of time and effort. This is a per-host
property, which often means big hosts with multiple services
co-located cannot decide alone.

And even if sysadmin lets users decide this value, it's very likely
they'll get that wrong (improper tuning or just copy the config from
another place), which in turns cost engineering hours to go chase that
down.

So I always lean towards as plug-and-play as possible. And as I said
earlier, it's easier to add knobs than to remove them (or ask users to
stop using them).

>
> For things like an extremely high amount of zero folio swap, I think
> remapping the zero folio it self is definately a better idea instead
> of wasting a lot of SWAP entries and metadata in the long term
>

I don't disagree here.

We tried doing that in the past, but it was slightly buggy. It might
be solvable, we just don't have the manpower to deal with this yet :(


> > >  - boot vs runtime toggle: I believe a runtime toggle is better. It
> > > can still be tuned on boot via cmdline, but it offers the extra
> > > ability to turn it on without modifying a base VM boot image or
> > > kernel.
> > >  - growth / limits: as talked above.
> > >  - charging semantics
> > >  - there could be other longterm design decisions on top of these as
> > > we talked in above.
> > >
> > > Once these settle, I think the picture will be clean. Your writeback
> > > logic and swap table integration are solid, while Baoquan's VM_SPARSE,
> > > previous preparation with swap ops give us a structural foundation. A
> > > clean long-term architecture means getting the core structure right,
> > > let's try not treating this as a race between series.
> >
> > I've reached out to Baoquan as well, but I figured there's value in
> > both discussions in big and small groups, and especially since there's
> > a chance some of us might not be coming to LPC. Let's see what we come
> > up with.
>
> I'll be there this year if nothing goes wrong :), I hope we can come
> up with something.

I'm excited to see you and the rest of the swap cabal too :)