Re: [RFC PATCH 00/10] mm/swap: ghost swapfile with backend switching via Redirect entries

[Nhat Pham]

On Thu, Jul 9, 2026 at 4:39 AM Baoquan He <baoquan.he@xxxxxxxxx> wrote:
>
>
> Thanks for the thorough review, Nhat.
> >
> > We don't want to put additional operational burden on users. Besides,
> > expecting userspace to react to swap space being filled, especially
> > under memory pressure (at which point userspace processes might be
> > stalling already), is tricky.
>
> The manual sysfs extension is the minimal viable interface for the RFC,
> not the final choice. But even in its current form, it is a dramatic
> improvement over the status quo.
>
> Today, to use zswap you MUST provision a physical swap device of the
> same size as the swap capacity wanted — even though most of that disk
> space will never be touched.  With ghost swap, you swapon a ghost
> device at whatever size you expect to need (e.g. 64 GB), and if your
> workload grows, you grow it via sysfs.  That is already a step
> change in operational simplicity: no disk pre-provisioning, no wasted
> storage, and a one-line resize when needed.

If these are the only three requirements you have, in fact, you can
already achieve something equivalent to a ghost swapfile: just put
zswap on top of any of the in-memory swapfile (and perhaps disable
writeback, although with careful limit setting it's almost impossible
to trigger writeback with zswap in newer kernels anyway).  I know this
works, because I even run tests with zswap-on-top of zram :) You can
swapon new swapfiles to support your "userspace-triggered" growth
strategy.

>
> Automatic growth can be considered later if it's really demanded.

It is really, really necessary. Please read my older virtual swap
space patch series for why it's necessary (see [5]).

>
> >
> > This is especially bad when redirection entry still eats up space in
> > the upper level (ghost zswap) swapfile. You will run out of zswap
> > space really fast under concurrent zswap store and zswap writeback.
>
> I think there is a misunderstanding here.  A Redirect entry does not
> consume an *additional* ghost slot.  Before writeback, the ghost slot
> held a Pointer entry (zswap).  After writeback, that same slot holds a
> Redirect entry (pointing to the physical device).  It's a state
> transition — one slot, one entry — not an extra allocation.  The ghost
> device's total capacity is unchanged by writeback activity.

No, what I meant is, after writeback, you still hold on to the old
slot on the ghost device, AND a new slot on the physical device,
correct?

>
> >
> > >   sysfs, up to a current maximum of 1TB.  This is implemented through
> > >   lazy vmalloc: a sparse 64MB virtual address space is reserved at
> > >   swapon time to cover the full 1TB range, but physical pages for
> >
> > This range seems a bit small, especially from something essentially fixed.
>
> The 1TB limit comes from a conservative 64 MB sparse vmalloc reservation
> in RFC. It certainly can be redefined as a bigger one which is more
> reasonable in reality. For most real-world deployments today, 1 TB per
> ghost device — with support for multiple ghost devices — is more than
> sufficient.

I beg to differ. We already have multi-terrabyte scale machines in
production in multiple places, and more and more overcommitting. At
least make this configurable.

>
> >
> > >   cluster_info[] structures are allocated only on demand as the device
> > >   grows.  The cluster_info pointer never changes, so all existing
> > >   accessors (__swap_offset_to_cluster, cluster_index) work unchanged.
> > >
> > > - sysfs control.  Each ghost device exposes attributes under
> > >   /sys/kernel/mm/ghost_swap/ghost_<N>/:
> > >     path (RO) — backing device name
> > >     max  (RW) — current max pages; writing a larger value triggers
> > >                 swap_ghost_extend_max()
> > >
> > > Design overview
> > > ---------------
> > > A swap table slot can now hold a Redirect entry:
> > >
> > >   Redirect: |----- physical swp_entry_t -----|101|
> > >
> > > This stores a plain swp_entry_t pointing to a slot on a real (physical)
> > > swap device.  The low 3-bit mark (0b101) distinguishes it from Pointer
> > > (0b100), PFN (0b10), Shadow (0b1), and NULL (0) entries.
> > >
> > > The lifecycle is:
> > >
> > > 1. swapon a ghost device — reserves sparse vmalloc area (64MB virtual)
> > >    and physically backs only the initial range.  The device starts with
> > >    no physical backing store: pages swapped out go to zswap, and zswap
> > >    pool evictions go to any concurrently attached physical device.
> >
> > If you think this data structure is superior to the xarray for dynamic
> > swap address, it's a fairly simple change on top of what I had in the
> > virtual swap RFC, no?
>
> I appreciate the suggestion. However, the difference between the two
> approaches goes beyond the cluster allocation strategy — it's
> the overall architecture:
>
> - Virtual swap introduces a mandatory, always-on vswap device (CONFIG_VSWAP).
>   with per-cluster virtual_table arrays, xarray-backed dynamic clusters.
>   per-CPU allocation caching.
>
> - Ghost swap extends the existing swap device model with one new entry
>   type (Redirect, 0b101) and one new flag (SWP_GHOST).  It reuses
>   swapon/swapoff, the existing allocator, and the swap cache
>   essentially unchanged.  No Kconfig option is needed.
>
> These are two explorations of the same problem, while virtual swap goes
> far beyond the problem itself, the decoupling of physical swap and
> virtal zswap. We can discuss and choose one approach.

If my problems are real, why do we have to choose? If we go with my
approach, all of our problems are solved. If we go with your approach,
then your problem is solved, with a promise that my problems will be
tackled in the future.

>
> >
> > It can be grafted on top of what I had in a subsequent patch, or
> > squashed into my first patch. If you have concerns with me
> > implementing your idea here, I'd take your code if it proves to be
> > superior.
> >
> > >
> > > 2. zswap writeback from ghost: when the zswap pool needs to evict,
> > >    zswap_writeback_ghost() allocates a physical slot from any real
> > >    swap device, writes the data there, and plants a Redirect entry
> > >    in the ghost's swap table pointing to the physical slot.
> >
> > What if user swapoff the physical swapfile, whose slot is backing a
> > redirection entry of a ghost swapfile, rather than a PTE. Has this
> > been handled yet?
>
>
> This is a gap in the current RFC.  Both swapoff and physical-slot
> reclaim would need a reverse mappingfrom the physical slot back to the
> ghost entry to be handled robustly. This can be added in a future version

To be clear, this is not a feature gap. In the current form of your
RFC, it is an actual bug: it breaks writeback *completely*. Swapoff
(and reclaim of swap-cache-only physical slots) leaks ghost zswap
swapfile slots that are still backed by these reclaimed slots.

I understand that this is just an RFC, or rather a proof-of-concept of
what you want to propose. But you can't claim it both way. It is only
simpler if you're dropping writeback altogether.

> if there is consensus that writeback-to-physical is a necessary part of
> the design.

I'll not spend my time argue for the necessity of  writeback. I've
already written essays about it, but the fact that we're implementing
new mechanism to trigger writeback (see [4]), optimization on the
writeback path (see [1]), and the fact that even ZRAM has some
writeback mechanism, mean multiple parties are interested. We need to
make it better, not digging ourselves into a hole that prevent it from
moving forwards.

>
> I think the primary motivation for ghost swap is to enable zswap
> usage *without* a backing physical swap device at all — the "No backing
> store required" mode described in the cover letter.   In that mode,
> writeback never happens (there is no physical device to write to), so
> the swapoff and reclaim questions do not arise. Ghost writeback to a
> physical device is an optional extension for users who want the safety
> net of disk-backed swap. Surely we can discuss and decide if we need
> add it or not.
> >
> > Similarly, say the physical swapfile is full, and you attempt to
> > reclaim a physical swap slot that is swap-cache-only (swap count = 0).
> > Say this physical swap slot was allocated from a previous zswap
> > writeback attempt, i.e it has a redirection entry in the ghost zswap
> > swapfile associated with it. Did we handle it somehow?
> >
> > My apologies if I missed it somehow.
> >
> > >
> > > 3. Swap-in from ghost: swap_read_folio() detects the ghost device
> > >    (SWP_GHOST), resolves the Redirect to the physical swp_entry_t,
> > >    and forwards the read I/O to the physical device.
> > >
> > > 4. When the swap cache folio is removed, the Redirect entry is restored
> > >    from an xarray (redirect_xa) that persists the mapping, so
> > >    subsequent reads still find the physical backing.
> > >
> > > 5. When the physical slot is freed, the ghost slot is also freed,
> > >    cascading through swap_range_free().
> >
> > I don't get how *any* of this is simpler from what I had. If anything,
> > it's more complicated. With a vswap device, you just allocate a
> > physical swap slot (which you also did), change the backend, submit
> > the IO (which you also did), and that's it. No need to juggle this
> > redirect_xa entry. What does this redirect_xa buy us?
>
> The redirect_xa persists the ghost→physical mapping across swap-cache
> folio teardown.
>
> Ghost: redirect_xa[offset] = phys  (only allocated when Redirect
>        is active) + swap_table[slot] = Redirect(phys) once the
>        swap cache folio is removed

So it's basically the same thing as vswap's vtable, which houses the
redirection, except:

1. It's not cluster-partitioned, so more lock contention.

2. It only holds the redirection temporarily. Which complicates the
logic, because now you have to add more logic to handle the transition
of the redirection entry to the ghost swapfile's swap table?

So it's potentially less efficient, and logic-wise, just as if not
more complicated?

>
> >
> > Not to mention now it entagles generic swap logic into
> > backend-specific code, unnecessarily. And we have more state in the
> > state machine to consider: With a generic vswap device, vswap -> disk
> > swap directly is the same state as vswap -> written back zswap entry.
> > With this zswap ghost swapfile design, you have to treat the latter
> > separately, as there is a redirection entry involved, which means
> > concurrent swap operations interact with it.
>
> I'd argue the opposite.  Ghost swap touches the generic swap paths in
> very few places:
>
>   - swap_read_folio:    one SWP_GHOST branch → resolve Redirect
>   - __swap_cluster_free_entries: one Redirect check → cascade free
>   - __swap_cache_do_del_folio:   one redirect_xa lookup → restore
>   - zswap_writeback_entry:       one SWP_GHOST branch → ghost writeback

That's because you just don't handle all the cases required to make it
correct (rmap) and efficient (per-device per-CPU allocation caching)
:)

>
> Ghost swap only affects behavior when a ghost device has been
> explicitly swapon'd, and even then only on the paths listed above.
>
> This is different than virtual swap which is more ambitious. As I said,
> ghost swapfile focuses on the problem itself, and based on the swap
> table.

I'm considering adding a sysctl knob to enable/disable vswap at
runtime in the next version of my patch series. This is trivial, as
even with CONFIG_VSWAP, both the new and the old code paths already
co-exist (as I still keep the old code path as a potential fallback
for the cases when vswap slot allocation fails for whatever reason).

That means that you can get functionally the same thing here if you
want - the old behavior, even with CONFIG_VSWAP.

>
> >
> > 2. No reverse mapping, or at least some mechanisms to handle the case
> > where you have to go from physical swap slots to the upper layer.
>
> Acknowledged — a reverse mapping can be added in a future version if
> writeback-to-physical becomes a priority.

More like, if you want writeback-to-physical *correct* at all.

> >
> > 3. Charging behavior difference, and deduplication of metadata when we
> > have multiple swap levels. (In fact, you seem to be double charging a
> > writtenback entry here towards the swap counter, not just duplicating
> > the metadata space)
>
> >
> > 4. Infrastructure to make sure allocation from this swap device and
> > allocation from the backing swap device does not invalidate each
> > other's per-cpu allocation cache. You seem to just go for the slow
> > path with alloc_phys_swap_slot() for zswap writeback?
>
> Yes, alloc_phys_swap_slot() uses the slow-path allocator
> (cluster_alloc_swap_entry), which is correct for this low-frequency
> operation.  zswap writeback is not a hot allocation path — it runs
> from the zswap shrinker under memory pressure, typically hundreds to
> thousands of operations per second at most.  The fast-path per-CPU
> caches remain dedicated to folio_alloc_swap(), which handles the
> high-frequency swapout path.  There is no cache invalidation problem
> because the two paths use different allocation mechanisms on different
> devices.

I'm nooot entirely sure about this. Zswap writeback can be quite a hot
path - there is work on optimizing it (see [1] for a recent attempt to
limit the expensive rstat flushing here - and this flushing is *per
shrinker iteration*, whereas the physical device allocation will be
*per writeback*, which is much, much more frequent).

> >
> > It's fine to disagree with any or all of these design choices of mine
> > - I'm receptive to discussion. But it's misleading to say you have a
> > simpler design here, IMHO. My design choices resulted from multiple
> > thoughtful iterations (a lot of them could be found in my
> > pre-swap-table version of vswap) and real production pains.
>
> You're right that "simpler" needs qualification, and I'll update the
> cover letter accordingly.  What I meant — and what I should have stated
> explicitly — is that ghost swap is less invasive to the existing swap
> infrastructure:
>
>   - It does not require a Kconfig option.
>   - It does not create a boot-time device that lives for the kernel's
>     lifetime.

I don't think these 2 are significant.

>   - It does not introduce a new per-cluster data structure
>     (swap_cluster_info_dynamic / virtual_table).

Instead, you add a redirect_xa xarray, which, as I noted above, plays
a *very* similar role (store the redirection), except a bit worse
performance-wise and more complicated.

Again, unless you're willing to drop writeback :)

>   - It does not add a separate per-CPU allocation cache.

This is *by design*, because I want to avoid writeback invalidating
the main allocation path.

If the argument is that it is not a problem, then I'm happy to drop
it. But I don't think that's the case - in fact, as Kairui have
pointed out, what we're moving towards is *every* swap device having
its own per-CPU allocation cache (see [2]). It's really necessary, not
just for vswap, but also for swap tiering (see [3]). So i don't think
the argument (that we don't need separate caching) here is correct,
and it's also immaterial soon enough (once we have generic per-device
per-CPU allocation cache I'll drop the vswap-specific one).

>   - It does not split the swap entry namespace into "vswap" vs
>     "physical" — a ghost device is just another swap device with one
>     extra flag and one extra swap-table entry type.

Actually, if we're to take in your full design, it's *two* entry
types. It's the redirection entry (if you want to support writeback) +
the Pointer entry (for your zswap "optimization").

>
> I think VSS goes beyond solving the "zswap without backing store"
> problem — it is fundamentally an architectural restructuring of the
> swap subsystem, with virtual swap as a first-class abstraction layered
> above physical devices.  That is both its strength (comprehensive,
> future-proof) and its trade-off (more invasive to existing paths).
> I respect that.
>
> My goal with ghost swap is different: address the immediate pain point
> with the smallest possible change to the existing infrastructure.  Both
> approaches have value, and I hope the community discussion will help
> identify the right balance.

That's because I believe "zswap without backing store" just by itself
is a bit narrow of a problem. It's certainly nice to fix, but it's
more of a nuisance - you already have userspace hacks for it (as I
mentioned above).

The problem I'm trying to solve is to support:

1. Writeback.

2. Dynamicity.

3. Decoupled backends.

all of which are motivated by real production issues, not some
theoretical problems. I'm concerned that if we only focus on the third
goal, we'll dig ourself into a hole that prevents us from solving 1
and 2 efficiently down the line.

>
> Thank you again for the detailed review — it has already clarified
> which parts of the design need more thought and documentation, and will
> make the next version stronger.

I really don't want to be antagonistic, but I hope you'd take the real
production pains that we've had (and have tried to communicate in
multiple mailing threads, across a timeline of almost 2 years at this
point) seriously.

Collaborations should go both ways. I've gone out of my way to try to
address the concerns of various parties, from spending *multiple
weeks* testing and investigating performance regression on zram
backend (which my company does not use), to a rewrite/re-design of
virtual swap to accommodate parties who wished to opt out of virtual
swap for now.

I hope you can extend the same good will to our needs :) I've included
you (and other swap folks such as Chris and Kairui) in my cc-lists. If
you have concerns, you could have commented. Instead, you decided to
send a patch series, which is basically just the ghost swapfile, with
a bit of afterthought to handle writeback and dynamicity, rather
inefficiently (no dedicated per-CPU caching), and not even correctly
(the lack of rmap means swapoff / swap-cache-only physical swap slot
reclaim is broken).

I respect you and Chris very much to assume bad faith, but please work
with me rather than against me.

[1]: https://lore.kernel.org/all/20260702180908.150136-1-yunzhao@xxxxxxxxxxxxxx/
[2]: https://lore.kernel.org/all/CAMgjq7BhOn48xEyC=2j837R7qddfjeBVHMiRqdx8no4ZEBpBLg@xxxxxxxxxxxxxx/
[3]: https://lore.kernel.org/linux-mm/ah0BfRFv285gWph8@yjaykim-PowerEdge-T330/
[4]: https://lore.kernel.org/all/20260629112032.20423-1-jiahao.kernel@xxxxxxxxx/
[5]: https://lore.kernel.org/all/20260505153854.1612033-1-nphamcs@xxxxxxxxx/