[RFC PATCH v4 0/2] mm/zsmalloc: defer object free to per-cpu batches
From: Hongfeng Chen
Date: Fri Oct 16 2026 - 08:11:12 EST
Freeing swap entries is expensive when unmapping a VMA that holds many of
them. `zap_pte_range()` eventually calls `zram_slot_free_notify()`, which
frees one compressed object per page. As a result, the unmapping task
has to pay for every `zs_free()` itself. On Android, this slows down the
exit of a task killed to reclaim memory, precisely when that memory is
needed.
Two earlier attempts by Lei and Zhiguo introduced a new thread in the MM
core to asynchronously collect and free swap entries [1][2], which
seems overcomplicated. Wenchao's RFC v3 [3] moved the deferral into
zsmalloc, but introduced push/drain callbacks that zram and zswap had
to implement, potentially violating software layer boundaries. This v4
RFC defers freeing inside `zs_free()` itself, leaving callers unchanged.
Patch 1 makes `zs_free()` append the handle to a per-CPU batch of 64
handles instead of freeing it immediately. Once the batch is full, it
is handed to an unbound drain worker through a lockless list, and the
worker frees the objects. Releasing empty zspages is also deferred to
the worker. If the per-CPU state cannot be allocated, `zs_free()` falls
back to its original path.
With the zsmalloc work moved off the unmapping path by patch 1,
contention on atomic operations in zram statistics becomes significant.
Patch 2 moves the contended counters to per-CPU storage.
Using the following microbenchmark:
// SPDX-License-Identifier: GPL-2.0
/* Times munmap() on a region pushed out to zram: MB. */
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>
#ifndef MADV_PAGEOUT
#define MADV_PAGEOUT 21
#endif
static void die(const char *what)
{
perror(what);
exit(1);
}
static double now_ms(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1e3 + ts.tv_nsec / 1e6;
}
/* About 2.8:1 under lzo-rle, objects in 16 size classes. */
static void fill_data(char *p, size_t bytes)
{
size_t i, k, n;
memset(p, 0, bytes);
for (i = 0; i < bytes; i += 4096) {
long *q = (long *)(p + i);
n = 106 + i / 4096 % 16 * 8;
for (k = 0; k < n; k++)
q[k] = k * 8 * 2654435761UL;
}
}
static char *prepare(size_t bytes)
{
char *p = mmap(NULL, bytes, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (p == MAP_FAILED)
die("mmap");
fill_data(p, bytes);
if (madvise(p, bytes, MADV_PAGEOUT))
die("madvise");
return p;
}
int main(int argc, char **argv)
{
size_t bytes = (argc > 1 ? strtoul(argv[1], NULL, 10) : 256) << 20;
char *p = prepare(bytes);
double t0, t1;
raise(SIGSTOP); /* ready */
t0 = now_ms();
munmap(p, bytes);
t1 = now_ms();
printf("%.6f %.6f\n", t0, t1);
return 0;
}
Performance results:
ROCK 5B+ (RK3588: 4x Cortex-A55 at 1.8GHz, 4x Cortex-A76 at 2.4GHz,
performance governor), single process munmap on A55:
times in ms, speedups over base
Base Patch1 Patch1+2
size time time speedup time speedup
64MB 13.81 3.62 3.81x 3.28 4.21x
128MB 28.50 7.18 3.97x 6.49 4.39x
192MB 42.19 10.71 3.94x 9.69 4.36x
256MB 56.06 14.26 3.93x 12.89 4.35x
512MB 109.50 28.43 3.85x 25.66 4.27x
1024MB 217.10 56.70 3.83x 51.15 4.24x
Same machine, single process munmap on A76:
times in ms, speedups over base
Base Patch1 Patch1+2
size time time speedup time speedup
64MB 5.59 1.58 3.54x 1.49 3.76x
128MB 11.86 3.16 3.75x 2.95 4.02x
192MB 17.18 4.72 3.64x 4.45 3.86x
256MB 22.89 6.29 3.64x 5.87 3.90x
512MB 45.15 12.52 3.61x 11.64 3.88x
1024MB 89.16 24.75 3.60x 23.12 3.86x
x86 (Intel Core i7-14700HX, P-cores locked at 2.1GHz with turbo off),
single process munmap:
times in ms, speedups over base
Base Patch1 Patch1+2
size time time speedup time speedup
64MB 2.87 1.50 1.92x 1.14 2.53x
128MB 6.00 2.93 2.05x 2.21 2.71x
192MB 9.33 4.35 2.15x 3.33 2.80x
256MB 12.51 5.78 2.17x 4.38 2.85x
512MB 24.96 11.39 2.19x 8.62 2.90x
1024MB 50.19 22.78 2.20x 17.05 2.94x
1-4 process munmap of 256MB each:
times in ms, speedups over base
Base Patch1 Patch1+2
mode time time speedup time speedup
1p 13.18 5.78 2.28x 4.35 3.03x
2p 25.83 14.41 1.79x 4.57 5.65x
3p 35.70 22.14 1.61x 4.68 7.63x
4p 44.25 28.71 1.54x 4.75 9.31x
[1] https://lore.kernel.org/all/20240805153639.1057-1-justinjiang@xxxxxxxx/
[2] https://lore.kernel.org/all/20250909065349.574894-1-liulei.rjpt@xxxxxxxx/
[3] https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xxxxxxxxxx/
Changes since v3:
- Defer inside zs_free() only and drop the push/drain callbacks, so zram
and zswap do not change. Yosry and Nhat objected to the layering in v3;
Nhat also pointed out that v3 leaves torn-down entries on the zswap LRU.
- Buffer handles in per-cpu batches of 64 allocated on demand, instead of
per-cpu pages rotated through a preallocated pool.
- New patch 2: keep the hot zram statistics per cpu.
- Link to v3: https://lore.kernel.org/linux-mm/20260508060724.3810904-1-haowenchao@xxxxxxxxxx/
Changes since v2:
- Use per-cpu single-page buffers instead of a global list; the hot path
only writes into the local CPU's buffer with preemption disabled
- Add a page pool for buffer rotation: when the current buffer is full,
swap it with a free page from the pool and queue the full page for drain
- Introduce push/drain callback ops so that zram and zswap can each define
their own element size and drain logic (zram stores u32 slot indices,
zswap stores unsigned long handles)
- Drop the lock optimization patches it will be submitted separately as
part of a dedicated zsmalloc lock contention series
- Link to v2: https://lore.kernel.org/all/20260421121616.3298845-1-haowenchao@xxxxxxxxxx/
Hongfeng Chen (2):
mm/zsmalloc: defer object free to per-cpu batches and a drain worker
zram: keep the hot statistics per cpu
drivers/block/zram/zram_drv.c | 171 +++++++++++++++++++++++++++-----
drivers/block/zram/zram_drv.h | 16 ++-
mm/zsmalloc.c | 182 ++++++++++++++++++++++++++++------
3 files changed, 309 insertions(+), 60 deletions(-)
base-commit: f4e9810097537d862c128935c36216bcf0c31cb2
--
2.54.0