Re: [RFC PATCH 00/10] mm/swap: ghost swapfile with backend switching via Redirect entries

[Chris Li]

On Tue, Jul 21, 2026 at 1:06 AM Baoquan He <baoquan.he@xxxxxxxxx> wrote:
>
> On 07/17/26 at 09:02am, Nhat Pham wrote:
> > On Fri, Jul 17, 2026 at 7:27 AM Baoquan He <baoquan.he@xxxxxxxxx> wrote:
> > >
> > > On 07/16/26 at 09:49am, Nhat Pham wrote:
> > >
> > > Thanks for the thoughtful reply and the very constructive proposal.
> > > Your suggestion about moving forward in small, incremental steps in
> > > last mail is really appealing — I think that's the right way to make
> > > progress on this. I need a little time to think it through more
> > > carefully before giving you a proper response. Will get back to you
> > > soon.
> >
> > Yeah let me know what you think. I'm planning to send out v3 with some
> > more cleanups and fixes, but I'm happy to hear what you and Chris
> > wants to make it as useful as possible for everyone.
>
> Hi Nhat,
>
> I've thought about it more carefully. Here's what I think the path
> forward could look like. I'll try to lay it out from the bottom up.
>
> 1. Data structure
>
>    Instead of introducing a separate swap_cluster_info_dynamic wrapper,
>    I'd prefer to add a per-slot backend pointer array directly into
>    struct swap_cluster_info:

I agree with that. The current VS patch is a great improvement over
the previous version of VS. However it places the cluster info and
swap table in different locations when accessing swap table-related
information. The code path requires different locks as well, the
xarray needs to take the xa lock.  This kind of dual-personality issue
will make reasoning harder. Keeping the swap table in the same place
is simpler in my mind.

The other issue I want to add is that, using xarray as a swap
allocator is unverified. It has no size limit which is one of its
claimed advantages, but also one of its drawbacks. Because it does not
have swap_full() concept. It will never do the normal if swap_full()
fail to allocate, let's do swap cache related. That code path cannot
trigger because the xarray has no size limit.
There are other considerations as well. It works fine if VS only wants
to allocate 4K swap entries. If VS later wants to go into the mthp
swap entry allocation. I don't think xarray has similar considerations
as the current swap allocator regarding fragment lists, etc. The
xarray isn't designed to be used that way, at least not in the common
usage situation.  You are likely running into your xarray allocation
specific issue.

Therefore, I think we should keep the current swap entry allocator and
the swap table data structure storing locations. Just extend the
per-swap-entry to allow custom implementation for VS to store the
zswap or even the write back info. Grow the swap device size if needed
after the normal swap cache reclaim path. That is likely a better way
to proceed.

>
>    struct swap_cluster_info {
>        ...
>    #ifdef CONFIG_ZSWAP
>        atomic_long_t *vs_pointer;   /* or xs_pointer — per-slot backend pointers */
>    #endif
>        struct list_head list;
>    };
>
>    This is allocated together with or alongside the swap table when the
>    cluster is first used.  No wrapper struct, no new ->cluster_info
>    pointer type change — just one extra field that is NULL for clusters
>    that don't need it.

I think there are other ways to allocate and grow that array. Maybe we
could unify this with other per swap-ops devices by having their own
extnesion on swap ops private pointer. On the other hand, we can do
the simplest thing to unblock and move forward. This data structure is
internal, we can always change it later.
>
>    The name is bike-sheddable: vs_pointer (virtual-space pointer) or
>    xs_pointer (cross-space pointer), xs_table (xswap extension table),
>    aux_table (auxiliary table), priv_table (private data table),
>    slot_priv (per-slot private data) or s_ext (slot extension).  The
>    point is that it stores a pointer to the backend owner of each slot —
>    initially just struct zswap_entry *, eventually extensible to other
>    backend types. I prefer to name it ext_table, but we have had
>    extend_table. My second favorite option is aux_table. What's your
>    choice?
>
> 2. Eliminate the zswap xarray
>
>    Once vs_pointer is in place, zswap can store and load entries
>    through vs_pointer[ci_off] instead of xa_store/xa_load on the
>    per-device zswap xarray.  The cluster lock protects the access.
>    This is exactly the same idea as your virtual_table, just placed
>    directly in swap_cluster_info rather than in a wrapper.

Yes. I would like to get rid of the xarray from zswap or VS.

>
>    The benefit: zswap_swapon and zswap_swapoff become no-ops (no
>    global xarray to allocate or tear down).  The per-cluster allocation
>    is lazy — only clusters that actually see a zswap store allocate the
>    array.
>
> 3. First step for ghost swap / virtual swap: dynamic growth and shrink,
You mean xswap :-)

>    no writeback yet
>
>    I think we agree that the first landing should be narrow in scope:
>    - A virtual swap device that decouples PTE swap entries from
>      physical backing.
>    - Backed only by zswap (and zero pages) initially.
>    - No writeback to physical swap.
>    - Dynamic growth and shrink of the cluster space.
>
>    For the dynamic cluster management, I'd like to try a lazy vmalloc
>    approach: allocate the cluster_info array as a vmalloc region, and
>    use vm_area_map_pages() to map newly populated clusters on demand.
>    Growth is automatic via page faults; shrink means reclaiming the
>    physical pages backing freed clusters.
>
>    The vmalloc approach has O(1) lookup (simple array index), which
>    matters on the swapout hot path.  The trade-off is that shrink is
>    coarser than xarray-based management (we can't easily unmap an
>    individual cluster from the middle of the vmalloc region).  But for
>    the initial landing where swap usage tends to be ratchet-like, I
>    think this is a reasonable trade-off.  We can always switch to
>    xarray later if shrink granularity becomes a real issue.

The xarray allocator also has a fragmentation issue where not all
memory released from the xarray returns to the system as free pages.
Using kvmalloc grow it will wait some unused cluster info. But let's
put it into perspective: cluster info is shared by 512 swap entries.
Each swap entry is 4K. So the memory used by swap cluster is pretty
small. Much smaller than 1 byte per swap entry. We can also modify the
cluster info array, making it an array of pointers. Then the waste
will be even smaller (if unused). At the cost of one more pointer per
swap_cluster_info. We can consider all this later, not at this current
stage.

> 4. Writeback and rmap as follow-up

I agree that adding cluster info growth as the first step. I recall
Kairui has an RFC patch series to implement virtual entry lookup
(still using the xarray). I think the combination of growth and
indirection lookup should provide most of what VS depends on from the
core swap point of view.


>    Once the basic virtual swap device is upstream, we add writeback
>    support:
>    - When zswap store fails, fall back to allocating a physical swap
>      slot and writing out.
>    - The vs_pointer entry is updated to point to the physical slot
>      (tagged to distinguish from a zswap_entry pointer).
>    - rmap for physical -> virtual reverse lookup, needed for swapoff
>      and physical slot reclaim.
>
>    This is a natural extension: the vs_pointer is already per-slot, so
>    adding a new backend type is a tag-bit change, not a rewrite.
>
> 5. Longer-term: swap tiering
>
>    Looking further ahead, the per-slot pointer can also serve as the
>    basis for a swap tiering model.  Imagine a memcg with multiple
>    tiers: ghost/virtual swap at the top, then pmem, then SSD, then
>    HDD.  Swapout goes to the top tier first; when that fills, data
>    cascades down to the next tier.  Each swap device only needs to
>    know about the device directly below it — a simple "push down"
>    interface.

I like that a lot.

>
>    This is a longer-term vision, but having a uniform per-slot backend
>    pointer from the start keeps the door open without painting us into
>    a corner.
>
> 6. Naming
>
>    We have three names floating around: "ghost swap", "virtual swap",
>    and Chris suggested "xswap".  We're going to be talking about this
>    thing a lot, and having a single agreed-upon name would help avoid
>    confusion in discussions, commit messages, and documentation.  I
>    don't have strong feelings about which one — but I'd like us to
>    settle on something everyone can live with.

I recall Rik suggested taking VS and ghost swap and creating something
new that is better than each. I agree with that collaboration
suggestion. I totally welcome a name that follows that spirit.

>
> 7. How to swapon a ghost / virtual swap device
>
>    There are several approaches on the table right now:
>
>    a) A sparse file created via truncate (Chris's approach): a swap
>       file that looks large (e.g. 1GiB) but only has one real backing
>       page.  Works with the existing swapon syscall, but the sparse
>       file has to live somewhere and the filesystem interaction is a
>       bit of a hack.

I give up on that truncated hack, but what if we make the swap file
really small, e.g. 4K and then dynamically grow it at runtime. Then it
is not a hack anymore. No more truncated file. We can find many ways
to improve it to get rid of the hack. That was the original reason to
send out the ghost swapfile RFC: a starting point for discussions.

>
>    b) A boot-time, always-present device (your VSS approach): the
>       vswap device is created unconditionally at late_initcall when
>       CONFIG_VSWAP=y.  No userspace action needed, but no runtime
>       control either — you can't create or destroy it, and you can't
>       have per-memcg or per-workload virtual swap spaces.

I don't think we should have swap without swap on. I prefer keeping
the swap on/off interface even for memory based swap.

>
>    c) A swapon flag: something like SWAP_FLAG_VIRTUAL or
>       SWAP_FLAG_GHOST passed to swapon(2), which tells the kernel to
>       treat the swap device as a ghost device with no backing storage.

You mean SWAP_FLAG_XSWAP :-)

>
>    d) A dedicated sysfs / debugfs knob: create and size the ghost
>       swap device at runtime via a file under /sys/kernel/mm/swap/.

For the most part we should allow the swap device to grow
automatically for the virtual entry internally without userspace
knobs. We might still want a knob for the max limit of the virtual
entries. I don't think it should be unlimited. I am open to
suggesting.

Chris

>
>    e) Something I'm missing?
>
>    Each has trade-offs between flexibility, simplicity, and the
>    existing swap management model.  I'd like to hear what you and
>    Chris think — especially whether the boot-time-only nature of the
>    current VSS design is a deliberate simplification or something
>    you'd be open to revisiting.
>
> Curious to hear what you think — especially if you see any structural
> issues with putting vs_pointer directly in swap_cluster_info rather
> than in a wrapper.  If the wrapper is important for your v3 design,
> I'd like to understand the rationale better.
>
> Thanks,
> Baoquan
>