Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - redis-memtier

[Bharata B Rao]

Redis-memtier

In-memory Redis server is loaded with a ~64 GB dataset
(62.2M keys x 1 KB) whose pages are then explicitly migrated to the CXL
node (lower tier), while the Redis server and the memtier client run on a
top-tier DRAM node.  The measurement phase drives GET traffic with
memtier (16 threads x 100 conns, PASSES=12) over 50% of the keyspace so
that repeatedly-accessed lower-tier pages become promotion candidates.
We compare throughput/latency and the kernel promotion activity across
the base kernel and the pghot kernel, and across hotness sources (NUMA-
balancing hint faults vs. the IBS hardware memory profiler).

System: AMD Zen6, Nodes 0,1 = DRAM (top tier); node 2 = CXL (lower tier).

Cases:
  C1 base/NUMAB0     base kernel, promotion OFF (baseline)
  C2 base/NUMAB2     base kernel, NUMAB tiering promotion ON
  C3 pghot/NUMAB2    pghot kernel, promotion via pghot, source =
                     NUMA hint faults (kmigrated batched migration)
  C4 pghot/IBS-10000 pghot, source = IBS profiler (period 10000), NUMAB=0
  C5 pghot/IBS-5000  same as C4 but IBS period=5000 (2x sampling)

=========================================================================
Table 1: Benchmark metrics (memtier)
=========================================================================
Case               NUMAB   Ops/sec  vs base    Avg    p50    p99   p99.9
                                                          latency (ms) ->
------------------------------------------------------------------------
C1 base/NUMAB0         0   281,424      ref 181.62 180.22 348.16  362.50
C2 base/NUMAB2         2   295,551    +5.0% 173.18 168.96 307.20  354.30
C3 pghot/NUMAB2        2   296,269    +5.3% 172.42 169.98 327.68  364.54
C4 pghot/IBS-10000     0   281,645    +0.1% 181.67 179.20 350.21  362.50
C5 pghot/IBS-5000      0   281,665    +0.1% 181.72 180.22 348.16  362.50
------------------------------------------------------------------------

=========================================================================
Table 2: Page-migration metrics (vmstat delta over measurement)
=========================================================================
Legend: C1=base/NUMAB0  C2=base/NUMAB2  C3=pghot/NUMAB2
        C4=pghot/IBS-10000  C5=pghot/IBS-5000  ('-'=counter absent)

metric                       C1         C2         C3        C4        C5
-------------------------------------------------------------------------
pgpromote_success             0 10,433,576 10,433,484   594,762 1,128,469
pgpromote_candidate_nrl       0 10,433,576 10,433,484   594,762 1,128,469
numa_pte_updates              0 20,333,714 20,344,211         0         0
numa_hint_faults              0 10,433,576 10,433,533         0         0
numa_pages_migrated           0 10,433,576 10,433,484   594,762 1,128,463
pgmigrate_success             0 10,433,576 10,433,484   594,762 1,128,463
pghot_recorded_accesses       -          - 10,433,533   595,426 1,130,753
pghot_recorded_hintfaults     -          - 10,433,533         0         0
pghot_recorded_hwhints        -          -          0 1,000,187 2,026,558
pgdemote_kswapd               0          0          0         0         0
-------------------------------------------------------------------------

=========================================================================
Key observations
=========================================================================
1. Promotion helps this workload: enabling NUMAB tiering promotion lifts
   throughput from 281,424 to 295,551 ops/sec (+5.0%) and cuts average
   latency 181.6 -> 173.2 ms.
2. pghot-hintfault has no regression vs base as it matches the base tiering
   case.
3. pghot-hwhint sources promotes far fewer pages than hint faults and it
   barely shows any improvement over base notier case.
4. Halving the period (10000->5000) ~doubles hw hints (1.00M->2.03M) and
   promotions (0.59M->1.13M) as expected, but is still far short of the
   hot set, so throughput does not yet move.