Re: [PATCH v3 00/11] Virtual Swap Space (Swap Table Edition)

[Nhat Pham]

On Fri, Aug 7, 2026 at 2:07 AM Chris Li <chrisl@xxxxxxxxxx> wrote:
>
> Hi Nhat,
>
> First of all, thank you very much for addressing the feedback
> regarding the swap metadata size concern and for stopping the
> punishment of zram usage. I'm unsure how to proceed with your earlier
> VS series (before swap table version V2), given the previous concerns.
>
> I was a bit nervous when you reverted the swap table and replaced it
> with something that performed worse in earlier series. I'm not
> attached to the swap table. The performance regression for existing
> use cases simply doesn't make sense to me. Thanks again that is no
> longer the case.

Eh, I'm open minded. I try lots of stuff - I tried the swap table
direction twice actually.

>
> On Thu, Aug 6, 2026 at 11:43 AM Nhat Pham <nphamcs@xxxxxxxxx> wrote:
> >
> > Changelog:
> > * v2 [v2] -> v3:
> >     * Rebased onto current mm-unstable.
> >     * Add a runtime vm.vswap_enabled sysctl and CONFIG_VSWAP_DEFAULT_ON
> >       to gate vswap allocation.
> >     * More cleanups and small bug fixes.
> >     * Split THP swapin enablement into its own patch (patch 5).
> >     * Add production workload benchmark results, and drop RFC tag.
> > * v1 [v1] -> v2:
> >     * Rebased to a newer mm-unstable tip.
> >     * Fix a bunch of assorted issues (incorrect zswap store failure
> >       rollback, vswap_init() failure handling, rmap-encoding collision,
> >       etc.) and clean up the code (rename a bunch of functions to
> >       more closely follow existing patterns, etc.).
> >     * Some more code clean up and simplification: some renamings to more
> >       closely follow existing patterns, move vswap backing check to
> >       __swap_cache_add_check, store zero state in the swap_table for
> >       vswap entries, etc.. Many of these are proposed by Kairui Song
> >       in [1].
> >     * Defer memcg_table allocation on physical clusters until the first
> >       vswap-backing slot installs. Saves ~512 bytes per physical cluster
> >       that only serves vswap-backing slots (this is the new patch 8).
> >     * Widen swap_info_struct->max and ->pages (and the swapoff unuse-path
> >       index) so vswap supports ~8 PB of swap space (this is the new
> >       patch 9).
> >     * Split the physical-swap-backend patch into three for reviewability:
> >       the core backend (patch 3), zswap writeback to physical swap
> >       (patch 4), and reclaim of cache-only physical slots (patch 5). No
> >       functional change.
> >     * Add kerneldoc for the vswap API.
> >     * Add some benchmark numbers for zswap case.
> >
> >
> > I. Context and Motivation
> > =========================
> >
> > Currently, when an anon page is swapped out, a slot in a backing swap
> > device is allocated and stored in the page table entries that refer to
> > the original page. This slot is also used as the "key" to find the
> > swapped out content, as well as the index to swap data structures, such
> > as the swap cache, or the swap cgroup mapping. Tying a swap entry to its
> > backing slot in this way is performant and efficient when swap is purely
> > just disk space, and swapoff is rare.
> >
> > However, the advent of many swap optimizations has exposed major
> > drawbacks of this design. The first problem is that we occupy a physical
> > slot in the swap space, even for pages that are NEVER expected to hit
> > the disk: pages compressed and stored in the zswap pool, zero-filled
> > pages, or pages rejected by both of these optimizations when zswap
> > writeback is disabled. This is arguably the central shortcoming of
> > zswap:
> > * Resource-wise, it is hugely wasteful in terms of disk usage. At Meta,
> >   we size swapfile in the order of 25-50% of host RAM, depending on flash
> >   availaiblity. This is a lot of flash for a fleet of our size, and
> >   with universal zswap enablement, most of this is wasted for zswap
> >   entries.
> >
> > * In deployments when no disk space can be afforded for swap (such as
> >   mobile and embedded devices), users cannot adopt zswap, and are forced
> >   to use zram. This is confusing for users, and creates extra burdens
> >   for developers, having to develop and maintain similar features for
> >   two separate swap backends (writeback, cgroup charging, THP support,
> >   etc.). For instance, see the discussion in [2].
> >
> > * Tying zswap (and more generally, other in-memory swap backends) to
> >   the current physical swapfile infrastructure makes zswap implicitly
> >   statically sized. This does not make sense, as unlike disk swap, in
> >   which we consume a limited resource (disk space or swapfile space) to
> >   save another resource (memory), zswap consumes the same resource it is
> >   saving (memory). The more we zswap, the more memory we have available,
> >   not less. We are not rationing a limited resource when we limit
> >   the size of the zswap pool, but rather we are capping the resource
> >   (memory) saving potential of zswap. Under memory pressure, using
> >   more zswap is almost always better than the alternative (disk IOs, or
> >   even worse, OOMs), and dynamically sizing the zswap pool on demand
> >   allows the system to flexibly respond to these precarious scenarios.
> >
> > * Operationally, static provisioning the swapfile for zswap poses
> >   significant challenges, because the sysadmin has to prescribe how
> >   much swap is needed a priori, for each combination of
> >   (memory size x disk space x workload usage). It is even more
> >   complicated when we take into account the variance of memory
> >   compression, which changes the reclaim dynamics (and as a result,
> >   swap space size requirement). The problem is further exacerbated for
> >   users who rely on swap utilization (and exhaustion) as an OOM signal.
> >
> >   All of these factors make it very difficult to configure the swapfile
> >   for zswap: too small of a swapfile and we risk preventable OOMs and
> >   limit the memory saving potentials of zswap; too big of a swapfile
> >   and we waste disk space and memory due to swap metadata overhead.
> >   This dilemma becomes more drastic in high memory systems, which can
> >   have up to TBs worth of memory.
> >
> > Swap virtualization is the answer to these issues, with three properties:
> >
> > 1. Decoupled backends. For zswap in particular, this means we eliminate
> >    the unused storage space, and allows zswap to be used in systems that
> >    do not have enough storage capacity for physical swap (without having
> >    to resort to silly hacks). Zero-filled swap pages and swap-cache-only
> >    folios also benefit here.
> >
> > 2. Dynamic swap space. Since virtual swap is not tied to any physical
> >    resource, we can make it infinite and dynamically grow it on demand.
> >    This massively simplifies operational provisioning, and increases the
> >    utilization of compressed swap backends (zswap). Dynamicity also
> >    reduces overhead on unused swap capacity.
> >
> > 3. Efficient backend transfer. The virtualization scheme should not
> >    introduce PTE/rmap walking overhead for backend transfer. This
> >    is crucial for systems that want to support multiple swap backends
> >    in a tiering fashion (for e.g zswap -> disk swap).
> >
> > For more historical contexts and references, please take a look at
> > the cover letter of the older vswap submissions ([3] and [v2]).
> >
> > II. Design
> > ==========
> >
> > When we compile kernel with CONFIG_VSWAP, a special vswap device is
>
> Does the CONFIG_VSWAP only make sense for zswap right now? No other
> swap usage can benifit from CONFIG_VSWAP.

Good point, yeah.

Maybe have CONFIG_VSWAP depends on CONFIG_ZSWAP, until another use
case comes (I was thinking of discontig physical swapping, but this
damn patch series is long.)

>
> > allocated at boot time, and all swapped out pages try to allocate from
> > this device first, falling back to a physical swap device on failure.
>
> Does it create a new user visible behavior change where users don't
> need to swapon and can start using VSWAP for zswap?
> That is a user-visible behavior change and we need to be more cautious about it.

Userspace behavior change *is* the whole point here though. I want the
users to not having to specify a static capacity :)

Furthermore, the implication of not breaking existing users'
deployment is that if users want this new feature, they have to change
their existing config. It's just about where it makes the most sense
(a compile time config? boot time config? runtime config? some
combinations of these?)

However, I absolutely agree with you about being cautious with the
exact API. I'm still evaluating the best way to bring about this
(perhaps a boot time parameters is better than this runtime knob, but
let me prototype and test things).

>
> I think a system should not use zswap or any type of swap if no device
> is swapped on.

Why not? If we don't specify the size, and just want on/off knob (say
at boot time), seems like an overkill to use this interface. But maybe
I'm missing something.

>
> Have vm.vswap_enabled is no the answer to address the new API change
> because existing distro that use fstab to control swap will need to
> jump through hooks.
> Previously, using fstab to control was at least consistent for all swap types.
>
> > Routing can also be turned off at runtime with the vm.vswap_enabled
> > sysctl, which defaults to 0 unless CONFIG_VSWAP_DEFAULT_ON=y. It is
> > allocation-only: new swapouts go straight to physical swap, while
> > entries already backed by vswap keep being served and drain as they
> > are faulted back in or freed.
> >
> > These swap entries can subsequently acquire backend on-demand, such as
>
> What do "These" refer to? Are they entries already backed by vswap?

Yep. Vswap entries

>
> > a zswap entry, or a slot on a physical swap device.
> >
> > We repurpose much of the existing swap_table infrastructure and
> > swapfile allocator for this new vswap device, with two notable
> > differences:
> > * Clusters are dynamically allocated on demand and managed through
> >   an xarray. This in turn allows us to avoid static provisioning and
> >   let swap space grow dynamically.
> >
> > * Each cluster of this new vswap device has a virtual_table that stores
> >   the backend information of the entries in the cluster (see below).
> >
> > Diagrams:
> >
> >   Case 1: vswap entry (virtualized)
> >
> >   PTE                  swap_cluster_info_dynamic
> >   vswap_entry          +---------------------------------+
> >   (swp_entry_t) ------>| swap_cluster_info (ci)          |
> >                        | +----------------------------+  |
> >                        | | swap_table                 |  |
> >                        | |   PFN / Shadow             |  |
> >                        | | memcg_table                |  |
> >                        | | count,flags,order          |  |
> >                        | | lock, list                 |  |
> >                        | +----------------------------+  |
> >                        |                                 |
> >                        | virtual_table                   |
> >                        | +----------------------------+  |
> >                        | | NONE                       |  |
> >                        | | SWAPFILE(swp_entry_t)      |  |
> >                        | | ZSWAP(struct zswap_entry*) |  |
> >                        | +----------------------------+  |
> >                        +---------------------------------+
> >                               |
> >                               | SWAPFILE resolves to
> >                               v
> >                        PHYSICAL CLUSTER (swap_cluster_info)
> >                        +--------------------------+
> >                        | swap_table per-slot:     |
> >                        |   NULL   - free          |
> >                        |   PFN    - cached folio  |
> >                        |   Shadow - swapped out   |
> >                        |   Pointer- vswap rmap    |
> >                        |   Bad    - unusable      |
> >                        |                          |
> >                        | Vswap-backing slot:      |
> >                        |   Pointer(C|swp_entry_t) |
> >                        |     rmap back to vswap   |
> >                        +--------------------------+
> >
> >   Case 2: direct-mapped physical entry (no vswap)
> >
> >   PTE                  PHYSICAL CLUSTER (swap_cluster_info)
> >   phys_entry           +--------------------------+
> >   (swp_entry_t) ------>| swap_table per-slot:     |
> >                        |   NULL   - free          |
> >                        |   PFN    - cached folio  |
> >                        |   Shadow - swapped out   |
> >                        |   Bad    - unusable      |
> >                        +--------------------------+
> >
> > struct swap_cluster_info_dynamic {
> >     struct swap_cluster_info ci;       /* swap_table, lock, etc. */
> >     unsigned int index;                /* position in xarray */
> >     struct rcu_head rcu;               /* kfree_rcu deferred free */
> >     atomic_long_t *virtual_table;      /* backend info, 8 B/slot */
> > };
>
> No a big fan of this two personality data structure thing depending on
> whether it is VS or not.
> If ci is the common part, I prefer to keep it separate and leave it alone.

I don't get what you meant. It is separate, and alone, no?

Originally Baoquan and I were thinking of shoving the virtual table
into "struct swap_cluster_info". I tried prototyping that one out, but
it looks uglier than I anticipated, so I dropped that approach.

>
> Also the extension is too vswap specific, it does not apply to other
> swap device types that might need their own private extension.
> You can take the VFS layer as an example. There is a VFS layer generic
> inode, which is common and shared by all file systems. And then you
> have filesystem-specific inodes as extensions, e.g. ext4_inode. The
> ext4_inode does not contain VFS inode. You don't see VFS having a code
> path like: if it is ext4, get the inode this way, else if f2fs, get
> the inode that way.


Hmmm, I'm not an FS expert. But struct ext4_inode_info does contain a
generic vfs inode field ("struct inode vfs_inode"), no? That is
actually more analogous than I would think: we have a vswap-specific
cluster struct (struct swap_cluster_info_dynamic) that wraps the
generic, shared metadata struct (struct swap_cluster_info).

I also never claim to re-implement vfs for swap. We take inspirations
where appropriate, but we're just writing our own thing at this point
:)

>
> In the first swap abstraction LPC talk, where I co-hosted with Yosry,
> I talked about the alternative approach: "VFS-like swap layers". That
> is exactly what I have in mind. We are getting very close to
> fulfilling that promise via swap ops and xswap extension interfaces.

I remember that talk. I don't remember how backend transfer would be
implemented in that proposal.

>
> I think implementing the generic interface first is simpler than
> implementing the non-generic vswap interface, ripping it out to
> replace it with a generic interface, and then putting back the generic
> modified version of vswap.
> If the two personality vswap xarray lookup gets in first, it will
> ultimately take more work to achieve the desired VFS-like extendable
> swap operations.
>
> I am happy to spend some time working with you to discuss the generic
> adopted version of vswap, if you are open to it. Or if you don't want
> to waste time on it. I can have someone else or myself come up with
> the generic adopted version of vswap for you to review, which I prefer
> less.

I would love to hear more about it, but so far I haven't quite
understood how it's going to achieve all of the requirements vswap is
trying to solve.

>
> Another piece of feedback is to please come up with a plan to submit
> your vswap changes piecemeal rather than as one long series. There is
> a lot of change like swap charging, that deserves a separate
> discussion before it gets merged. Look, the swap table changes took
> four phases. Each phase achieved a smaller milestone, with four of
> them ultimately reaching the finish line. I wish vswap had a similar
> piecemeal plan.

Lemme think about this a bit more. But in the meantime, could you at
least take a look at the general directions? For your use case (zswap
as the terminal swap level), I think the first two patches are the
most relevant, but the rest is crucial too for a generic use case.