Re: [RFC PATCH] mm: preserve physical order in swap fallback splitting

[Chengfeng Lin]

Hi Kairui,

Thanks. I agree that the original microbenchmark alone does not justify
an additional reclaim-specific helper.

The ordering here is among the children of one THP split. They enter the
current reclaim list, so preserving their physical order does not require
a physically ordered LRU.

I added a synthetic hot/cold test on the same MM base (2967f3ccdb5f), patch
and file-swap/NVMe/IOMMU setup. The main THP-reclaim condition touches
256 MiB of anonymous memory in shuffled 2 MiB chunks, then reads a 64 MiB
hot region for two seconds. At the start of another two-second read
window, an external controller requests 64 MiB via memory.reclaim with
swappiness=200. It uses MADV_HUGEPAGE, without MADV_PAGEOUT, forced collapse
or direct LRU manipulation. Reader/reclaimer CPUs are 2/4.

Two separate controls use base (4 KiB) pages with reclaim, and THPs without
reclaim. The primary metric is the calling thread's CPU time inside
memory.reclaim, including the patch's list-ordering work. It is not wall
time or total-system CPU time; setup, verification and cleanup are outside
the measurement.

For THP reclaim, separate path checks observed 33 splits per run on both
kernels. Post-run checks in clean timing found 66 MiB of cold pages in
swap and no hot pages in that state. Each round used fresh
baseline/prototype/baseline boots, with nine samples per condition per
boot. Each table baseline averages that round's two baseline points;
rounds are not pooled:

  THP reclaim-call CPU    baseline    prototype    saving per call
  First round            38.35 ms     34.56 ms     3.80 ms (9.90%)
  Second round           38.17 ms     34.94 ms     3.23 ms (8.45%)
  Third round            38.41 ms     34.70 ms     3.71 ms (9.66%)

THP reclaim CPU baseline return drift was at most 0.47%, and the maximum
within-group CV was 2.16%. Base-page reclaim CPU changed by +0.58%, +0.16%
and -0.37% across the three rounds.

The secondary metric is hot-read rate: repeatedly reading one 64-bit word
per 4 KiB hot page, in million reads/s. These changes cover the entire second
two-second window, including time after reclaim returns. The no-reclaim
control keeps the same window without issuing a request. Percentages are
relative to each round's baseline; positive means faster reads:

  Condition          Patched split path    Round 1    Round 2    Round 3
  THP reclaim                Yes            +2.80%     -1.07%     +1.97%
  Base-page reclaim          No             +7.93%     +0.28%     +6.72%
  No reclaim                 No             +8.96%     -1.42%     +8.03%

Read-rate CV reached 3.72% for third-round base pages; every other table
entry had a maximum within-group CV of at most 2.59%. Differences also
preceded reclaim in rounds one and three. Their cause remains unknown:
the controls do not enter the patched path, so I do not attribute their
read-rate gains to the patch. These unexplained differences remain a
limitation of the comparison, not evidence of application speedup.

The hot/cold reproducer and run instructions are attached. I'll also look
into a more general THP-splitting cleanup.

Thanks,
Chengfeng
Hot/cold anonymous-memory reclaim reproducer
============================================

cache_reclaim.c is the unchanged source used for the three hot/cold rounds.
It is separate from the original RFC's MADV_PAGEOUT reproducer. Build with:

  gcc -O2 -g -std=gnu11 -Wall -Wextra -Werror cache_reclaim.c -o cache_reclaim
  ./cache_reclaim --selftest

Prerequisites
-------------
Use a dedicated Linux test machine, not a production host. The measured pair
was mm-stable 2967f3ccdb5f and that same tree with the original RFC patch.
This source requires 4 KiB pages, online/available CPUs 2 and 4, cgroup v2 with
the memory controller enabled at its root, memory.reclaim with swappiness=,
and the memory.stat swapcached field. Run as root for cgroup and pagemap access.
Keep at least 4 GiB MemAvailable and 512 MiB unused swap before each invocation.

Use an existing file-backed swap area on NVMe with translated IOMMU mappings
for the reported backend comparison. zswap must be disabled. THP must permit
MADV_HUGEPAGE allocations; the program rejects fewer than 192 MiB of initial
anonymous THPs in the THP conditions. It does not force collapse or retry until
this state appears. An unsupported interface or failed state check is not a
fast timing sample. Do not create a swap device or change machine policies just
by running this source. Record the actual configuration before comparisons.

The reported runs used full preemption, performance governor on CPUs 2/4 and
turbo disabled, with identical VM settings on the paired kernels. This minimal
recipe does not change those policies. Set up and restore any chosen policies
separately, and do not compare differently configured machines or boots.

One bounded invocation
----------------------
Run the following in a root Bash shell from the directory containing the
binary. Change mode to base or idle for the separate controls. The recipe
refuses to enable a missing root controller, never reuses a cgroup, and only
cleans up its own cgroup. It does not reboot or change global VM settings.

  (
    set -euo pipefail
    mode=thp
    test "$(id -u)" = 0
    test "$(getconf PAGESIZE)" = 4096
    test "$(cat /sys/module/zswap/parameters/enabled)" = N
    grep -qw memory /sys/fs/cgroup/cgroup.subtree_control
    awk '/^MemAvailable:/ {ok=($2>=4194304)} END {exit !ok}' /proc/meminfo
    awk '/^SwapFree:/ {ok=($2>=524288)} END {exit !ok}' /proc/meminfo
    taskset -c 2,4 true
    group=/sys/fs/cgroup/ks-f02-aged-reproducer-$$
    mkdir "$group"
    cleanup() {
      if test -n "$(cat "$group/cgroup.procs")"; then
        echo 1 > "$group/cgroup.kill"
      fi
      for i in {1..20}; do
        test -z "$(cat "$group/cgroup.procs")" && break
        sleep 0.1
      done
      test -z "$(cat "$group/cgroup.procs")" && rmdir "$group"
    }
    trap cleanup EXIT
    echo 536870912 > "$group/memory.max"
    echo 268435456 > "$group/memory.swap.max"
    timeout --signal=TERM --kill-after=2s 35s env KS_ALLOW_AGED_RECLAIM=1 \
      ./cache_reclaim "$mode" "$group"
    cat "$group/memory.events"
  )

Record stdout, stderr and exit status. Each invocation uses a fresh cgroup.
memory.events must remain zero; investigate nonzero events rather than treating
such a run as comparable. The parent/worker also have 22/18-second alarms and
parent-death protection. A hard kernel hang cannot be cancelled by these guards.

Workload and output
-------------------
All modes populate 256 MiB in shuffled 2 MiB chunks. One quarter is a 64 MiB hot
region, interleaved with cold chunks. The child reads one word per hot 4 KiB
page for two seconds, then repeats for another two seconds. At the start of
the second window the external parent requests 64 MiB of memory.reclaim with
swappiness=200, except in idle mode. Reader and caller are pinned to CPUs 2/4.

  thp   MADV_HUGEPAGE allocation plus reclaim: main condition.
  base  MADV_NOHUGEPAGE allocation plus reclaim: separate 4 KiB control.
  idle  MADV_HUGEPAGE allocation without reclaim: separate read-rate control.

The program does not use MADV_PAGEOUT, MADV_COLLAPSE, or direct LRU manipulation.
It checks initial residency, THP size, post-window pagemap state, all data words,
and child completion. The root parent remains outside the tested cgroup.

  reclaim_cpu_ns: caller thread CPU around the memory.reclaim write (primary).
  reclaim_wall_ns: elapsed time around that write, including waiting.
  service_lookups / service_wall_ns * 1000: million reads/s in the ENTIRE
    second two-second window, including time after reclaim returns (secondary).
  aged_lookups / aged_wall_ns * 1000: read rate before the reclaim request.
  cold_swapped * 4096 / 1048576: MiB of cold pages in swap at the post-window
    check, not a continuous trace or all submitted disk bytes.
  hot_swapped: hot pages found in swap at that same check.
  whole_cpu_ns: complete parent plus child CPU; includes preparation and checks.

EAGAIN from memory.reclaim is recorded, not silently retried. Check achieved
reclaim and page state even when the syscall returns success. The three THP
rounds reported 66 MiB of cold pages in swap and zero hot pages in that state.
Do not require a different machine to reproduce exactly that over-reclaim.

This program alone does not prove that a particular split or DMA path was hit.
In our separate, non-timing path checks both kernels split 33 THPs per THP run;
the base/idle controls did not enter that split path. Do not leave probes on
during clean timing, or promote state-only checks to path evidence.

Reported clean comparisons used fresh baseline/prototype/baseline boots,
one warmup and nine samples per condition per boot, in thp/base/idle order.
The two baseline points were averaged within each round; rounds were not pooled.
The reproducible finding was reduced reclaim-caller CPU, not proven application
speedup: read-rate differences also appeared in controls outside the patch path.
#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <inttypes.h>
#include <sched.h>
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/prctl.h>
#include <sys/resource.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <time.h>
#include <unistd.h>

/* Synthetic hot-cache service, not a claim about a named production app. */
#define MIB (1024UL * 1024)
#define BYTES (256 * MIB)
#define CHUNK (2 * MIB)
#define PAGE 4096UL
#define CHUNKS (BYTES / CHUNK)
#define REQUEST (64 * MIB)
#define AGE_NS 2000000000ULL
#define SERVICE_NS 2000000000ULL

static void die(const char *s) { perror(s); exit(1); }
static void check(int ok, const char *s) { if (!ok) { errno = EINVAL; die(s); } }
static uint64_t now(clockid_t c) {
    struct timespec t;
    if (clock_gettime(c, &t)) die("clock_gettime");
    return (uint64_t)t.tv_sec * 1000000000ULL + (uint64_t)t.tv_nsec;
}
static void pin(int cpu) {
    cpu_set_t set; CPU_ZERO(&set); CPU_SET(cpu, &set);
    if (sched_setaffinity(0, sizeof(set), &set)) die("affinity");
}
static void transfer(int fd, void *buf, size_t size, int sending) {
    size_t done = 0;
    while (done < size) {
        ssize_t n = sending ? write(fd, (char *)buf + done, size - done)
                            : read(fd, (char *)buf + done, size - done);
        if (n < 0 && errno == EINTR) continue;
        check(n > 0, "pipe transfer"); done += (size_t)n;
    }
}
static void put(int fd, void *buf, size_t n) { transfer(fd, buf, n, 1); }
static void get(int fd, void *buf, size_t n) { transfer(fd, buf, n, 0); }
static uint64_t pattern(size_t i) { return (uint64_t)i * 0x9e3779b97f4a7c15ULL ^ 0x12d8b064e5f7963aULL; }
static uint64_t item(const char *path, const char *wanted) {
    FILE *f = fopen(path, "r"); char line[512], key[128]; uint64_t value;
    if (!f) die(path);
    while (fgets(line, sizeof(line), f)) {
        if (sscanf(line, "%127s %" SCNu64, key, &value) == 2 && !strcmp(key, wanted)) {
            fclose(f); return value;
        }
    }
    fclose(f); die("missing stat field"); return 0;
}
static uint64_t scalar(const char *group, const char *name) {
    char path[512]; uint64_t n; snprintf(path, sizeof(path), "%s/%s", group, name);
    FILE *f = fopen(path, "r"); if (!f) die(path);
    check(fscanf(f, "%" SCNu64, &n) == 1, "scalar"); fclose(f); return n;
}
static uint64_t huge_kb(pid_t pid, uintptr_t start) {
    char path[64], line[512]; unsigned long a, b; int selected = 0; uint64_t n;
    snprintf(path, sizeof(path), "/proc/%d/smaps", pid);
    FILE *f = fopen(path, "r"); if (!f) die(path);
    while (fgets(line, sizeof(line), f)) {
        if (sscanf(line, "%lx-%lx", &a, &b) == 2) selected = (a == start && b == start + BYTES);
        if (selected && sscanf(line, "AnonHugePages: %" SCNu64 " kB", &n) == 1) {
            fclose(f); return n;
        }
    }
    fclose(f); die("mapping not found"); return 0;
}
struct pages { uint64_t present, swapped, hot_swapped, cold_swapped, absent; };
static struct pages pages(pid_t pid, uintptr_t start) {
    char path[64]; struct pages p = {0}; uint64_t ent[512];
    snprintf(path, sizeof(path), "/proc/%d/pagemap", pid);
    int fd = open(path, O_RDONLY); if (fd < 0) die(path);
    for (size_t i = 0; i < BYTES / PAGE; i += 512) {
        check(pread(fd, ent, sizeof(ent), (off_t)((start / PAGE + i) * 8)) == sizeof(ent), "pagemap read");
        for (size_t j = 0; j < 512; j++) {
            int present = !!(ent[j] & (1ULL << 63)), swapped = !!(ent[j] & (1ULL << 62));
            check(!(present && swapped), "pagemap flags"); p.present += present; p.swapped += swapped;
            if (swapped) { if (((i + j) * PAGE / CHUNK) % 4 == 0) p.hot_swapped++; else p.cold_swapped++; }
            p.absent += !present && !swapped;
        }
    }
    close(fd); return p;
}
struct service { uint64_t lookups, sum, cpu_ns, wall_ns; };
static struct service serve(volatile uint64_t *mem, uint64_t duration) {
    struct service s = {0}; uint64_t wall = now(CLOCK_MONOTONIC), cpu = now(CLOCK_THREAD_CPUTIME_ID);
    /* Cyclic reads of one word per hot page; cold 3/4 is left untouched. */
    do {
        for (size_t c = 0; c < CHUNKS; c += 4)
            for (size_t p = 0; p < CHUNK / PAGE; p++) {
                s.sum += mem[(c * CHUNK + p * PAGE) / 8]; s.lookups++;
            }
    } while (now(CLOCK_MONOTONIC) - wall < duration);
    s.cpu_ns = now(CLOCK_THREAD_CPUTIME_ID) - cpu; s.wall_ns = now(CLOCK_MONOTONIC) - wall;
    return s;
}
__attribute__((__noinline__)) void ks_reclaim_phase(int active) {
    __asm__ __volatile__("" : : "r"(active) : "memory");
}
static void worker(int in, int out, const char *group, int thp) {
    if (prctl(PR_SET_PDEATHSIG, SIGKILL)) die("pdeathsig");
    check(getppid() != 1, "parent gone"); alarm(18); pin(2);
    char path[512], pid[32]; snprintf(path, sizeof(path), "%s/cgroup.procs", group);
    int fd = open(path, O_WRONLY); if (fd < 0) die(path);
    int n = snprintf(pid, sizeof(pid), "%d", getpid()); check(write(fd, pid, (size_t)n) == n, "join cgroup"); close(fd);
    void *raw = mmap(NULL, BYTES + CHUNK, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    check(raw != MAP_FAILED, "mmap");
    uintptr_t start = ((uintptr_t)raw + CHUNK - 1) & ~(CHUNK - 1);
    size_t lead = start - (uintptr_t)raw, tail = CHUNK - lead;
    if (lead && munmap(raw, lead)) die("trim prefix");
    if (tail && munmap((void *)(start + BYTES), tail)) die("trim suffix");
    if (madvise((void *)start, BYTES, thp ? MADV_HUGEPAGE : MADV_NOHUGEPAGE)) die("THP allocation hint");
    uint64_t *mem = (uint64_t *)start; unsigned order[CHUNKS]; uint32_t seed = 0x7f28c301U;
    for (unsigned i = 0; i < CHUNKS; i++) order[i] = i;
    for (unsigned i = CHUNKS - 1; i > 0; i--) {
        seed = seed * 1664525U + 1013904223U; unsigned j = seed % (i + 1), tmp = order[i]; order[i] = order[j]; order[j] = tmp;
    }
    for (unsigned c = 0; c < CHUNKS; c++) {
        size_t first = order[c] * CHUNK / 8;
        for (size_t i = first; i < first + CHUNK / 8; i++) mem[i] = pattern(i);
    }
    struct service aged = serve(mem, AGE_NS); put(out, &start, sizeof(start)); put(out, &aged, sizeof(aged));
    char token; get(in, &token, 1); check(token == 'S', "start token"); token = 'A'; put(out, &token, 1);
    struct service active = serve(mem, SERVICE_NS); put(out, &active, sizeof(active));
    get(in, &token, 1); check(token == 'V', "verify token");
    uint64_t checked = 0;
    for (size_t i = 0; i < BYTES / 8; i++) { check(mem[i] == pattern(i), "content mismatch"); checked++; }
    put(out, &checked, sizeof(checked)); if (munmap(mem, BYTES)) die("munmap"); _exit(0);
}
int main(int argc, char **argv) {
    if (argc == 2 && !strcmp(argv[1], "--selftest")) {
        check(BYTES == 256 * MIB && CHUNKS == 128 && REQUEST == 64 * MIB && CHUNKS % 4 == 0, "shape");
        puts("selftest=pass no_runtime_changes=1"); return 0;
    }
    check(argc == 3 && getenv("KS_ALLOW_AGED_RECLAIM"), "explicit owned experiment required");
    const char *mode = argv[1], *group = argv[2];
    check(!strcmp(mode, "thp") || !strcmp(mode, "base") || !strcmp(mode, "idle"), "case");
    const char prefix[] = "/sys/fs/cgroup/ks-f02-aged-";
    check(!strncmp(group, prefix, sizeof(prefix) - 1), "owned cgroup prefix");
    int thp = strcmp(mode, "base") != 0, reclaim = strcmp(mode, "idle") != 0;
    alarm(22); pin(4); uint64_t whole_wall = now(CLOCK_MONOTONIC), whole_cpu = now(CLOCK_PROCESS_CPUTIME_ID);
    int to[2], from[2]; if (pipe(to) || pipe(from)) die("pipe");
    pid_t child = fork(); if (child < 0) die("fork");
    if (!child) { close(to[1]); close(from[0]); worker(to[0], from[1], group, thp); }
    close(to[0]); close(from[1]); uintptr_t start; struct service aged, active;
    get(from[0], &start, sizeof(start)); get(from[0], &aged, sizeof(aged));
    struct pages before = pages(child, start); uint64_t ahp_before = huge_kb(child, start);
    check(before.present == BYTES / PAGE && !before.swapped && !before.absent, "initial full residency");
    check(thp ? ahp_before >= 192 * 1024 : ahp_before == 0, "initial THP state");
    uint64_t swap_before = scalar(group, "memory.swap.current"), mem_before = scalar(group, "memory.current");
    char path[512]; snprintf(path, sizeof(path), "%s/memory.reclaim", group);
    int fd = open(path, O_WRONLY); if (fd < 0) die(path);
    char token = 'S'; put(to[1], &token, 1); get(from[0], &token, 1); check(token == 'A', "active token");
    ks_reclaim_phase(1); uint64_t wall = now(CLOCK_MONOTONIC), cpu = now(CLOCK_THREAD_CPUTIME_ID);
    const char req[] = "67108864 swappiness=200\n"; errno = 0;
    ssize_t n = reclaim ? write(fd, req, sizeof(req) - 1) : (ssize_t)sizeof(req) - 1;
    int error = n < 0 ? errno : 0;
    uint64_t cpu_ns = now(CLOCK_THREAD_CPUTIME_ID) - cpu, wall_ns = now(CLOCK_MONOTONIC) - wall;
    ks_reclaim_phase(0); close(fd);
    check(error == 0 || error == EAGAIN, "unexpected reclaim error");
    check(error || n == sizeof(req) - 1, "reclaim short write");
    get(from[0], &active, sizeof(active));
    struct pages after = pages(child, start); uint64_t ahp_after = huge_kb(child, start);
    uint64_t swap_after = scalar(group, "memory.swap.current"), mem_after = scalar(group, "memory.current");
    snprintf(path, sizeof(path), "%s/memory.stat", group);
    uint64_t swapcache = item(path, "swapcached"), pgscan = item(path, "pgscan"), pgsteal = item(path, "pgsteal");
    token = 'V'; put(to[1], &token, 1); uint64_t checked; get(from[0], &checked, sizeof(checked));
    int status; struct rusage usage; check(wait4(child, &status, 0, &usage) == child && WIFEXITED(status) && WEXITSTATUS(status) == 0, "worker exit");
    check(checked == BYTES / 8, "full data verification");
    uint64_t child_cpu = (uint64_t)(usage.ru_utime.tv_sec + usage.ru_stime.tv_sec) * 1000000000ULL
                       + (uint64_t)(usage.ru_utime.tv_usec + usage.ru_stime.tv_usec) * 1000ULL;
    printf("{\"case\":\"%s\",\"bytes\":%lu,\"hot_bytes\":%lu,\"requested\":%lu,\"child\":%d,"
           "\"aged_wall_ns\":%" PRIu64 ",\"aged_lookups\":%" PRIu64 ",\"huge_before_kb\":%" PRIu64 ",\"huge_after_kb\":%" PRIu64 ","
           "\"initial_present\":%" PRIu64 ",\"after_present\":%" PRIu64 ",\"after_swapped\":%" PRIu64 ",\"hot_swapped\":%" PRIu64 ",\"cold_swapped\":%" PRIu64 ",\"absent\":%" PRIu64 ","
           "\"reclaim_errno\":%d,\"reclaim_cpu_ns\":%" PRIu64 ",\"reclaim_wall_ns\":%" PRIu64 ",\"service_cpu_ns\":%" PRIu64 ",\"service_wall_ns\":%" PRIu64 ",\"service_lookups\":%" PRIu64 ","
           "\"memory_before\":%" PRIu64 ",\"memory_after\":%" PRIu64 ",\"swap_before\":%" PRIu64 ",\"swap_after\":%" PRIu64 ",\"swapcached\":%" PRIu64 ",\"pgscan\":%" PRIu64 ",\"pgsteal\":%" PRIu64 ","
           "\"checked_words\":%" PRIu64 ",\"whole_cpu_ns\":%" PRIu64 ",\"whole_wall_ns\":%" PRIu64 "}\n",
           mode, BYTES, BYTES / 4, reclaim ? REQUEST : 0, child,
           aged.wall_ns, aged.lookups, ahp_before, ahp_after, before.present, after.present, after.swapped, after.hot_swapped, after.cold_swapped, after.absent,
           error, cpu_ns, wall_ns, active.cpu_ns, active.wall_ns, active.lookups, mem_before, mem_after, swap_before, swap_after, swapcache, pgscan, pgsteal,
           checked, now(CLOCK_PROCESS_CPUTIME_ID) - whole_cpu + child_cpu, now(CLOCK_MONOTONIC) - whole_wall);
    close(to[1]); close(from[0]); return 0;
}