Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - microbenchmark numbers

[Bharata B Rao]

Microbenchmark
==============
Multi-threaded application with 64 threads that access memory(8G) at
4K granularity repetitively and randomly. The number of accesses per
thread and the randomness pattern for each thread are fixed beforehand.
The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on
CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated
detecting and migrating them. The benchmark score is the time taken to
finish the accesses in microseconds. The sooner it finishes the better it is.
All the numbers shown below are average of 3 runs.

Table 1: Benchmark completion time (lower is better), avg of 3 runs
64 threads, random 4K access over 8G, mem on CXL node 2, cpu node 0

Config    Time (us)   Time (s)  Speedup
---------------------------------------
C1       97,165,273       97.2    1.00x
C2       40,378,842       40.4    2.41x
C3       95,431,823       95.4    1.02x
C4       40,455,945       40.5    2.40x
C5       41,289,646       41.3    2.35x
C6       64,260,194       64.3    1.51x

Speedup = C1 (base-NUMAB0) time / config time (>1.00x is faster)

Legend:
  C1 = base kernel, NUMAB0 (hot page promotion disabled)
  C2 = base kernel, NUMAB2 (hot page promotion enabled)
  C3 = pghot kernel, NUMAB0 (hint faults source disabled)
  C4 = pghot kernel, NUMAB2 (hint faults source enabled)
  C5 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=1 (NUMAB0)
  C6 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=2 (NUMAB0)


Table 2: Page migration / hotness vmstat counters (avg of 3 runs)
'-' = counter not present for that kernel/config

ctr           C1         C2         C3         C4         C5         C6
-----------------------------------------------------------------------
M1             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M2             0  2,283,787          0  2,097,152  1,966,188  1,444,783
M3             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M4             0  2,097,152          0  2,097,152          0          0
M5             0  2,283,787          0  2,097,152          0          0
M6             -          -          0  2,097,152  1,968,079  3,508,998
M7             -          -          0  2,097,152          0          0
M8             -          -          0          0  5,654,097  5,422,763
M9             -          -          0          0  5,654,097  5,422,763
M10            -          -          0          0  3,439,001  1,701,889
M11            -          -          0          0  1,879,453  3,105,182
M12            -          -          0          0  5,654,097  5,422,763
M13            -          -          0          0          0          0

Counter legend:
  M1 = pgpromote_success
  M2 = pgpromote_candidate_nrl
  M3 = numa_pages_migrated
  M4 = numa_pte_updates
  M5 = numa_hint_faults
  M6 = pghot_recorded_accesses
  M7 = pghot_recorded_hintfaults
  M8 = pghot_recorded_hwhints
  M9 = hwhint_total_events
  M10 = hwhint_dram_accesses
  M11 = hwhint_extmem_accesses
  M12 = hwhint_useful_events
  M13 = hwhint_dropped_events

Key observations
================
- The base NUMAB0 vs NUMAB2 numbers clearly show the benefit of promotion.
- pghot-hintfault matches the base tiering case both in speedup and number
  of promottions.
- pghot-hwhint also matches the base tiering case both in speedup and
  number of promotions.