Re: Path forward for Virtualized Swap?

[Nhat Pham]

On Sat, Sep 12, 2026 at 1:48 AM Kairui Song <ryncsn@xxxxxxxxx> wrote:
>
> On Sat, Sep 12, 2026 at 3:04 AM Nhat Pham <nphamcs@xxxxxxxxx> wrote:
> >
> > On Fri, Sep 11, 2026 at 11:14 AM Kairui Song <ryncsn@xxxxxxxxx> wrote:
> > >
> > > > Would something like this fix it for you? ZRAM users will not get
> > > > vswap indirection overhead at all, because it would bypass vswap :)
> > > > Down the line we can revisit this decision - for e.g, if there is a
> > > > use case for vswap-on-top-of-swapfile. There might be other interface
> > > > that makes more sense.
> > >
> > > Hmm, this interface looks confusing though. you mean tangles vswap
> > > with zswap through the cgroup's zswap limit? I originally expected
> > > this to be a problem solved by tiering, skipping certain tiers seems
> > > much more intuitive. Maybe Youngjun have some idea here?
> >
> > It wouldn't be confusing if this is something users don't have to
> > think about at all :) My point is that we don't have a use case where
> > we need userspace input on vswap-enablement on a per-cgroup basis yet,
>
> Ha, you really can't assume that for everyone :), we even have
> workloads working at different compression levels, and the tiering
> series shows different cgroups working on different devices, there has
> been pathes enabling per-cgroup swap device a decade ago.

I still don't think you're getting my points.

Compression levels have nothing to do with vswap selection. It has to
do with *zswap* (i.e compressed swapping) selection (or the algorithm
being selected). Vswap is just the transparent mechanism underneath.

Once you HAVE selected zswap, then the question of "do you want vswap
or not" is easier to answer:

1. Overhead: rather minimal with vswap (at least based on the testing
I have done so far).

2. Usability: vswap buys you the decoupling of swap space from
physical swapfile.

I struggle to see why you would not enable vswap if you enable zswap.
And more accurately, a scenario where you want vswap in some cgroups
with zswap, but not others.

Taking a step back from all of this - we're arguing about the
participation of vswap/xswap in an interface that has not even been
merged yet. None of us can say for sure how it will look, or if it
will be merged in the first place. I think that making sure vswap can
participate in that interface is already more than reasonable. Once
that interface is in, AND the usecase that necessitates per-cgroup
vswap selection presents itself, then we can expose vswap later.

>
> > > > Potentially, but we have many users at Meta. There's a huge diversity
> > > > of machine types, workingset size, access patterns (both frequency and
> > > > file:anon split), compressibility, etc.
> > >
> > > You can just set the number as 8PB? :)
> >
> > Sure, but with the vmalloc-array approach, there is more metadata
> > overhead even if we have not allocated the backing page of the
> > clusters yet. This will be annoying on the smaller size machines
> > (O(dozen of GB)) to also pay the overhead of 8PB-swap space metadata
> > reservation.
>
> Not al all? With VM_SPARSE, you literally have zero overhead, unless
> you really use these slots.
>
> And even if you really use these slots, the overhead in both CPU and
> memory is lower than Xarray, if I did the math right. I just ran a
> quick test with Baoquan's Xswap, forcing it to start with 8PB on boot;
> it was smooth as batter on a 1GB box and still worked fine after heavy
> anon memory pressure.
>
> See below (8PB xswap device with almost no metadata overhead, the only
> overhead comes from zswap so I reduced the tree number but that can be
> easily merged into clusters as we discussed before):
>
> ### Before XSWAP
> # free -m
>                total        used        free      shared  buff/cache   available
> Mem:            1464         405         756           1         544        1058
> Swap:              0           0           0
>
> ### Create a XSWAP device, I hardcoded the max size in code to 8PB,
> ### and use fewer ZSWAP trees:
> # echo 0 > /sys/kernel/mm/xswap/create
> [  140.289280] xswap: adding extendable swap type 0 (prio -1, init 733
> clusters = 375295 pages, max 2199023255552 pages)
> # free -m
>                total        used        free      shared  buff/cache   available
> Mem:            1464         407         755           1         545        1057
> Swap:           1465           0        1465
>
> ### Lift the ceiling to 8PB:
> # echo 4294967296 > /sys/kernel/debug/xswap/type0_cluster_limit
> # free -m
>                total        used        free      shared  buff/cache   available
> Mem:            1464         403         756           1         547        1060
> Swap:     8589934591           0  8589934591
>
> ### Run memhog:
> # time memhog 12G
> ...
> ________________________________________________________
> Executed in    4.36 secs    fish           external
>    usr time    0.46 secs    0.00 millis    0.46 secs
>    sys time    3.77 secs    3.80 millis    3.76 secs
>
> ### After memhog:
> # free -m
>                total        used        free      shared  buff/cache   available
> Mem:            1464         336        1136           0         222        1127
> Swap:     8589934591          31  8589934560
>
> The only different from vswap now is that you just hides everything :)
> (writeback is not hard to add)

Ah I see hmmm. I probably messed up my napkin math.

Regardless, I think we have strayed far from my original ask. It's a
good idea, but with significantly less exposure to correctness and
performance testing. Baoquan himself said it:

"Throughput is within ~2-3% of plain swap+zswap on a 64G/64-thread
swapout, so it is dominated by zswap compression, not the cluster table."

I've even tried to show that that if we are to want it, replacing the
xarray with vmalloc is simple, conceptually.

Why are we throwing away all of the testing, validation, design work?

Let's recap the arguments so far:

1. We began with: it's more performant/less overhead. Which
intuitively makes sense, but the numbers have not played out yet. And
as I have said, we still keep the old code in tact. Optimizations can
be done on top after the main use cases have been landed - it in fact
gives us a testing grounds for optimizations.

2. Now we have moved on to "it's simpler". My apologies, but I have to
push back on this, because the comparison so far is so unfair it's
farcical.

With xswap, we're rolling a custom-written extensible array. It
requires a 1000+LoC patch series to implement.

With vswap, we're using a data structure that work out-of-the-box
(xarray), with well-understood API and concurrency model. Yes, it
involves more null checks and interactions with RCU, but with vmalloc
we're adding a workqueue to support only tail-shrink, and have to
invoke vmalloc API, which includes nasty surprises such as sleepable
allocations, which can cause deadlocks in lock section.

If we're comparing LoC, the first 2 patches of vswap, which enables
writeback-less swap virtualization, is "+440 / -37" and "+456 / -29",
respectively (including places where I have to set things up for
writeback, detailed documentation of the encodings, etc.).

I don't think vmalloc is significantly simpler. It *looks* simpler,
because the interface looks like plain array access, but unlike the
xarray we're in charge of maintaining the entire data structure
itself, not just using it, so we have to take into account the data
structure's implementation in the complexity equations.

3. Userspace API - there's more to argue here, but I will note that
vswap can be runtime-enabled and sized too. I just don't think there's
a good use case for it yet (but we're discussing this in other
threads).

>
> > https://lore.kernel.org/all/CAKEwX=NYoMH8pTKeCvA=XHFmmNVMeDz3mxbhp5q6-PxSTJ5wOg@xxxxxxxxxxxxxx/
>
> Right I saw that, but 64 bit of private id will make shadows really
> awkward, not for SWAP but for file folios and file mappings, swap
> table is actully fine since memcg_table is standaone, and we can
> workaround the hieratical refault distance issue in other ways (e.g.
> record the reclaim level instead of memcgid direct). I think 24 bit is
> good for now? As I've never see anyone ran into issues
> with even 16 bits in production.

I think you misunderstood me. I'm not proposing raising the private id
- I'm proposing we raise the size of the refcnt field. That's the
thing that gets saturated.

Technically I think you can reproduce it with baseline - if my napkin
math is correct, if a cgroup has more than 8TB worth of swap entries
(4k size), then the refcnt will be saturated. You can already create
an 8TB swapfile today, IIUC (even without vswap).

Anyway that can be fixed as the independently from swap
virtualization. One thing at a time.