Re: [PATCH v5 4/6] mm/zswap: Implement proactive writeback

[Yosry Ahmed]

On Fri, Jul 10, 2026 at 3:04 AM Hao Jia <jiahao.kernel@xxxxxxxxx> wrote:
>
>
>
> On 2026/7/10 04:44, Yosry Ahmed wrote:
> > On Wed, Jul 8, 2026 at 7:15 AM Hao Jia <jiahao.kernel@xxxxxxxxx> wrote:
> >>
> >>
> >>
> >> On 2026/7/7 03:33, Yosry Ahmed wrote:
> >>> On Thu, Jul 2, 2026 at 5:32 AM Hao Jia <jiahao.kernel@xxxxxxxxx> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 2026/7/1 19:45, Hao Jia wrote:
> >>>>>
> >>>>>
> >>>>> On 2026/7/1 00:10, Yosry Ahmed wrote:
> >>>>>>>> Before going through more versions we need to figure out if this will
> >>>>>>>> pivot to be a proactive demotion interfcae for swap tiering.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes. Should I drop patches 4-6 in the next version and wait for swap
> >>>>>>> tiering to be finalized?
> >>>>>>> We can try to get the non-memcg parts (patches 1-3) merged upstream
> >>>>>>> first. This would also give them plenty of time to bake and catch any
> >>>>>>> potential regressions. Thoughts?
> >>>>>>
> >>>>>> Patches 1-2 can be sent and merged separately, yes. For patch 2,
> >>>>>> please include some numbers for the writeback performance before and
> >>>>>> after batching.
> >>>>>
> >>>>> I'd love to collect some performance data. Do you have any recommended
> >>>>> benchmarks for this?
> >>>>>
> >>>>
> >>>> Perhaps the following test case could work?
> >>>>
> >>>> Test Setup:
> >>>> - Total memory: 32 GB
> >>>> - zswap settings: max_pool_percent=1, accept_threshold_percent=50,
> >>>> shrinker_enabled=N
> >>>> - cgroup constraint: memory.max=1G
> >>>> - Workload: Run the following stress-ng command inside the cgroup for
> >>>> 120s to
> >>>>      continuously force zswap store failures and trigger shrink_worker():
> >>>>
> >>>>      bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
> >>>>      exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t
> >>>> 120s -q'
> >>>>
> >>>> The following comparison results were collected over multiple runs via
> >>>> bpftrace
> >>>> and the 'written_back_pages' sysfs interface:
> >>>>
> >>>>                             Baseline         Patched
> >>>> ---------------------------------------------------
> >>>> shrink_worker wakeups       5,587             878
> >>>> shrink_memcg calls      7,823,853       2,347,320
> >>>> written_back                  257         781,214
> >>>>
> >>>> Conclusion:
> >>>> Under the same workload and duration, the patched kernel shows a
> >>>> significant reduction
> >>>> in both shrink_worker wakeups and shrink_memcg calls, while successfully
> >>>> executing a
> >>>> much higher volume of page writebacks.
> >>>
> >>> Hmm this is actually a bit concerning. Yes, we are invoking the
> >>> shrinker less, but we're writing back *a lot* more memory, orders of
> >>> magnitude more. We are using a batch size of 64, and making ~1/3 of
> >>> the calls to shrink_memcg(), so the number of written back pages
> >>> should be ~20x more, not 3000x more? I think I am missing something.
> >>>
> >>> Also, ideally, the batching wouldn't result in significantly more
> >>> writeback, but a similar amount of writeback over less shrinker
> >>> invocations. If we are writing back significantly more pages then the
> >>> batching logic is probably too aggressive?
> >>
> >> Apologies, I think the test I constructed has a bit of a problem. This
> >> test has very, very heavy memory pressure and is already a very abnormal
> >> case.
> >>
> >> The zswap entry returns the first time because of "second chance" after
> >> setting referenced to false. For the baseline, it scans 1 page per node
> >> each time for 16 loops. During the test, shrink_worker() basically exits
> >> at about 16 pages each time.
> >>
> >> Since stress-ng periodically and randomly writes to this 4G memory, it
> >> keeps triggering zswapin and then waiting to zswapout new zswap entries
> >> after falling below the pool threshold. When the speed of zswapin/out is
> >> far greater than the scanning speed of shrink_worker(), a large number
> >> of zswap entries cannot wait until the second scan for writeback. New
> >> entries are stored on the zswap LRU list again, and the referenced of
> >> the new zswap entries is set to true again. During the test, it was
> >> found that 99.21% of the return values of shrink_memcg_cb() in the
> >> baseline kernel were LRU_ROTATE.
> >
> > Hmm if I understand correctly, you are saying that the current
> > upstream code is actually failing to writeback when it should in the
> > previous test case with very high memory pressure, but it is with
> > batching? If that's the case, I think it's actually really good data
> > to include. However, we should make sure that's what's actually
> > happening. If the current shrinker is not keeping up and failing to
> > writeback, we should observe:
> > 1. shrink_worker() hitting MAX_RECLAIM_RETRIES continuously and bailing.
> > 2. zswap usage consistently remains at/near the limit, and not going
> > down to the acceptance threshold.
> > 3. zswap_store() failing to accept pages and the pages going directly
> > to disk, causing an LRU inversion (hotter pages on disk, colder pages
> > in zswap).
> >
> > Can you confirm that this is what's observed with the high pressure test case?
> >
>
> Apologies, my previous explanation might not have been very clear.
>
> For an entry to be written back, the shrinker must scan the *same* entry
> twice: the first scan sets referenced to false and returns ROTATE, and
> only during the second scan can it be written back.
>
> If a swap entry is zswapin'd between the first and second scan (meaning
> the entry is no longer on the zswap LRU), then this swap entry will not
> be written back by the shrinker. Therefore, the second scan must occur
> before this entry is zswapin'd for it to be possible to be written back.
> So, if the baseline scanning speed is far slower than the lifecycle
> speed of the swap entries, it results in only scanning once. In the
> baseline kernel, 99.21% of the return values of shrink_memcg_cb() are
> LRU_ROTATE, while the patched kernel's shrink_worker() scans at least 64
> * 16 entries in a single pass, resulting in only 58.7% of the return
> values of shrink_memcg_cb() being LRU_ROTATE.

Right, my question is, is the high rate of LRU_ROTATE leading to
failure to writeback in a way that causes zswap store failures (and
pages skipping zswap and going directly to swap)?

>
>               Baseline         Patched
> ---------------------------------------------------
> zswapin       929,096          281,196   <----
> zswapout      982,731        1,058,746
>
> After modifying the kernel to remove the "second chance" algorithm, the
> test data of the baseline kernel roughly conforms to our expectations.
>
> Therefore, the constructed test has a bit of a problem. The second test
> case in my previous email can ensure that the entries on the zswap LRU
> are relatively stable.
>
> It is worth noting that the capacity of the zswap pool will not remain
> constant all the time. Since stress-ng periodically writes to and reads
> from this 4G memory, accessing the pages in the zswap pool will trigger
> zswapin, causing the zswap pool size to decrease. Other pages attempting
> to zswapout into zswap via zswap_store() will fail (until the zswap pool
> size becomes 160M (accept_threshold_percent=50)), but the zswap_store()
> path will wake up shrink_worker() to try to perform scanning and
> writeback. Then stress-ng will continue to read and write memory,
> continuously triggering zswapin to further reduce the zswap pool size
> until the threshold, at which point shrink_memcg() will also stop. Then,
> new zswap entries (with referenced as true) are added to the zswap LRU
> again via zswap_store(), until the zswap pool size reaches 320MB (32 GB
> (Total memory) * 1%). This process repeats continuously during the test.
>
> Thanks,
> Hao
>