[RFC PATCH v2 0/9] mm/damon: hardware-sampled access reports
From: Ravi Jonnalagadda
Date: Thu Sep 10 2026 - 13:17:38 EST
This series lets DAMON take its access information from a hardware sampler
instead of from a page-table scan, and lets a scheme's score be weighted by what
that sampler reported.
The change from v1 [1] is that it is now built on the data attribute probe
infrastructure that has since landed in mm-new: a PMU is expressed as one more
probe on a context, with its own weight and probe hits, rather than as a
mechanism of its own beside the ops set. The rest of this posting is the same
substrate reworked onto that, plus two consumers.
This is not a merge request. The series is based on damon/next at the
base-commit below, which moves, so the tree it was built and tested from is also
on
https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26
It is posted for design feedback, on the substrate and on where it belongs in
the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's
second milestone, now open, is a first data attribute monitored through
`damon_report_access()`, and that is what a sampling PMU is here. So this
series keeps that function and its callers and replaces its body: the reporting
path a hardware sampler needs cannot take a mutex, and the drain has to reach a
virtual-address context as well as a physical one. The shape of the ring, the
drain and the sysfs surface are what is most useful to review.
## Why a unified perf-event substrate
DAMON derives its access information from the PTE Accessed bit. A sampling PMU
carries what that bit cannot: which addresses the hardware went to, and how
often it went there. Many machines already have such a unit, and more than one
kind of it, so what this series is after is letting DAMON's regions be tuned
from whichever perf-based hardware source a machine offers rather than from the
Accessed bit alone. A sampler does not arrive on a kdamond's terms, though: it
delivers an address when the hardware decides to, in NMI context, with no
relation to the sampling interval and no mm to walk.
The alternative is a backend per PMU vendor, each owning its own configuration,
sysfs knobs and lifecycle. The perf-event direction [3] avoids that: let DAMON
register kernel-counter perf events and consume samples from any sampling PMU the
perf core already knows about. This series follows it, adding one substrate below
the ops sets rather than an ops set per PMU -- a report ring that any in-kernel
access source can push into, and a drain that folds those reports into region
probe hits on the aggregation boundary the kdamond already has. What running it
across vendors needed on top of that direction is:
- per-CPU lockless rings between the NMI sample handler and the kdamond drain,
- a ring partitioned by probe class, so the page-fault primitive and a PMU can
populate one context without either seeing the other's reports,
- per-CPU events that follow CPU hotplug, armed when the kdamond starts and
disarmed and drained when it stops,
- a per-PMU owner, so two contexts cannot claim the same PMU type,
- whichever address a PMU does report carried on the report and matched
against the context's own address space, so one source serves a paddr or a
vaddr context without a backend per address space.
This is tested with PEBS on Intel and IBS on AMD, both configured as `perf_event`
attributes on a probe and using the perf core's event plumbing rather than
per-vendor MSR code. A third source has already been written against the same
ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer
drained in process context instead of an overflow callback, and which the
roadmap [2] places in its third milestone.
The partitioning is what lets promotion and demotion run in one context. A
sampler says which regions are hot; it says nothing about which are cold, because
a sampler that reports nothing about a page cannot distinguish untouched from
unsampled. Region age is what a demotion scheme matches on, and age comes from
the page-fault primitive. With the ring partitioned by class both are live at
once: the probe supplies hotness, the primitive supplies age, and two schemes
over the same regions can move memory in both directions under one kdamond.
The two classes are not symmetric, and the asymmetry is in where a report comes
from rather than in what it carries. A page fault has no monitoring context at
report time, so a per-context ring is not expressible for it: those reports go to
one global set of per-CPU rings, drained by the single context whose page-fault
primitive is enabled. A PMU overflow handler does carry the context that armed
the event, so those reports go to that context's own per-CPU rings and each
context drains only what it owns. Both classes share the ring layout and the
producer side, so the barrier pairing cannot drift between them.
```
page-fault report PMU sample, NMI context
no ctx at report time handler carries the owning ctx
| |
v v
+----------------------------+ +----------------------------+
| one global per-CPU ring | | per-CPU rings per context |
| set, shared by every ctx | | that armed a perf event |
+----------------------------+ +----------------------------+
| drained by the one ctx | | each ctx drains only the |
| with the pf primitive on | | rings it owns |
+----------------------------+ +----------------------------+
| |
+------------------+------------------+
|
drain on the aggregation boundary,
each report matched to a region
|
+---------------+---------------+
| |
region age region probe hits
demotion scheme promotion scheme
```
That is the property this series exists to enable, and three things follow from
it:
- One measurement drives both directions, at whichever scope the mode gives.
Bandwidth comes from resctrl MBM, which is where the discussion on a
platform-independent bandwidth signal for tiering has pointed [6]: a
virtual-address context reads a monitoring group created for the processes
it monitors, and a physical-address context reads the root monitoring
group, so the same controller is scoped to a set of processes or to the
whole machine without changing what it does. Reading it there also means a
bandwidth allocation reserved for an application bounds what the controller
optimises within. It spreads the hot set across nodes in the ratio that
maximises achieved bandwidth when bandwidth is what the workload is short of,
and it demotes cold pages off the near node in the same context at the same
time -- either proactively under a fixed quota or only once DRAM is under
memory pressure. Neither direction is configured as a target ratio; the
proportion or weight the controller writes bounds how much memory moves while
the score decides which pages move, and a hardware sample makes that score
proportional to the traffic a region generates rather than one bit per page
per scan. Earlier work on bandwidth-driven interleaving [4] measured the same
effect from userspace.
- Bandwidth and capacity are expanded at once rather than one being chosen.
Static weighted interleaving expands bandwidth, but it is a placement rule
rather than a decision about any particular page, so cold pages keep their
share of DRAM. NUMA balancing expands usable DRAM by promoting whatever is
hot, but it does so without reference to what the far node can deliver, so it
keeps promoting after the near node has stopped being the better place to
read from. Here the hot set's distribution comes from a bandwidth
measurement and the cold set is demoted on region age, in one context, so the
far node's bandwidth is used and DRAM is not held by memory nothing is
touching. The decision itself is a quota goal the kernel already carries:
`node_eligible_mem_bp` [8] bounds how much of a node a scheme may fill, so
what the controller supplies is the bandwidth measurement, not the
placement.
- When bandwidth is not the constraint it reverts to latency-first tiering on
its own. The search starts with everything on the near node and moves memory
outward only when the measurement improves, so on a workload that does not
saturate that node no step outward improves anything: the hot set stays in
DRAM for the lower access latency and cold pages keep being demoted. The
same configuration covers both cases, with nothing to switch on and no ratio
to revise.
## What the series adds
1. `mm/damon/vaddr: support page fault access check primitive` -- implements
the page-fault primitive for a virtual address space context; it was
available for physical addresses only. The marker whose fault reports the
access is installed through the target mm and the vma of the sampling
address directly, and prepare_access_checks() dispatches on the enabled
primitive as the physical address space one does.
2. `mm/damon/core: read the CPU number with preemption disabled` -- the report
path runs in fault context, which is preemptible. Read the CPU number with
preemption disabled and keep it disabled across the report, so the recorded
CPU stays consistent with the per-CPU ring the report is queued to.
3. `mm/damon/paddr: lock the folio for the page fault primitive rmap walk` --
take the folio lock unconditionally around the walk, as the other reverse
mapping walks in DAMON do, so every mapping type reaches it locked, and
release the reference the lookup took on each exit path.
4. `mm/damon: add damos_node_eligible_mem_bp tracepoint` -- a per-tick
tracepoint over the node-eligible-memory quota goal evaluation, exposing
the goal's target and current values, so the loop a bandwidth-driven
controller steers is visible to a tracer.
5. `mm/damon/core: add per-probe-class report rings and unified drain` -- the
substrate. Per-CPU SPSC rings an NMI-context source can publish into,
partitioned by probe index into a global page-fault ring and a per-context
perf ring, plus the kdamond-side drain that matches each report to a region
by binary search over a per-target snapshot and credits it to that region's
probe hits on the aggregation boundary. The address space of the target
selects which address of a report is matched, and pid targets are filtered
by thread group id.
6. `mm/damon: add perf-event overflow handler feeding the report ring` -- an
ops-agnostic perf-event source whose overflow handler turns a PMU sample
into a report, setting each address field the PMU reported as valid. A PMU
that reports a virtual address only can therefore drive a virtual-address
context, one that reports a physical address a physical-address context, and
the same source serves either without a backend per address space. Per-CPU
events are armed and released through cpuhp callbacks, and a per-PMU owner
keeps two contexts from claiming the same PMU type.
7. `mm/damon/ops-common: use probe-weighted score when probe weights are set`
-- lets a scheme's frequency subscore come from the probe hits, weighted per
probe class, so what the sampler reported reaches the tiering decision.
With no weights set the subscore comes from the access rate as before.
8. `mm/damon: add perf_event prep for PMU-driven hotness probes` -- the sysfs
surface and the event lifecycle: a perf_event prep action carrying the PMU
type, the event config and the sample attributes per probe, with per-CPU or
single-instance arming depending on how many counters the PMU needs.
Arming is deferred on a context built for a commit, so a weight-only commit
leaves the running event untouched.
9. `mm/damon/tests/drain-kunit: kunit for report rings and unified drain` --
unit tests for the rings and the drain: inject and drain, overflow on wrap,
producer routing by probe index, ring partitioning, pf-ring ownership,
thread-group filtering, the address space match, and both primitives live
on one context.
Patches 1-4 apply standalone on damon/next and are useful without the rest;
5-8 are the substrate and its first two consumers, and 9 covers them. If the
first four read right, they need not wait on the rest of this series -- happy
to repost them on their own for damon/next if that is easier to take. Two of
them, 2 and 3, correct code that is already in mm-new rather than adding
anything new, so they could instead go as a small fixes series carrying
Fixes: tags; guidance on which of the two routes is preferred is welcome.
## Userspace setup model
The runs were driven by an `auto_tier` subcommand added to damo on the branch
below, which reads achieved bandwidth from resctrl MBM and searches for the hot
set's distribution that maximises it, while the demotion scheme runs alongside
it in the same context. That tooling is not part of this posting; it is on
https://github.com/ravis-opensrc/damo/tree/damo/auto-tier-bw-2026-09-08
- `--bw_source resctrl` is the option this adds: the controller reads achieved
bandwidth from an MBM monitoring group and searches, rather than being given
a distribution to install.
- Everything else -- intervals, schemes, filters, the probe's `perf_event`
attributes -- is what the generator writes from the command line shown: 5 ms
sampling, 100 ms aggregation, a 1 s ops update.
- Both configurations select proactive demotion, so they demote continuously
rather than waiting for pressure, and both were left closed loop throughout.
- The search is the algorithm described in [4].
### Configuration A: AMD IBS Op, paddr ops, system-wide
```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness ibs \
--near_node 0 --far_node 4 \
--cold_demote --cold_demote_mode proactive \
-o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```
- Scope is the machine, not a process set. The distribution is steered
through `node_eligible_mem_bp` quota goals over each node's own physical
ranges, which the generator reads from `/proc/iomem`.
- AMD Turin, DRAM on node 0 and a CXL node. IBS Op at a `sample_period` of
262144 with `sample_phys_addr` set, which is what a physical-address context
needs from the sample.
- Workload is a hot-and-cold mix so both schemes have something to act on:
`multiload` drives the bandwidth-hungry hot set, and a second process
allocates on the near node, touches it once and goes idle, leaving pages
that age out for the demotion scheme to find.
### Configuration B: Intel PEBS L3-miss, vaddr ops, per-PID
```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness pebs \
--pid $HOT_PID --pid $COLD_PID \
--near_node 0 --far_node 1 \
--cold_demote --cold_demote_mode proactive \
--min_nr_regions 1000 --max_nr_regions 20000 \
-o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```
- Scope is the processes the monitoring group names and nothing else. The
distribution is steered through the hot scheme's `DamosDest` weights [9].
- Intel Granite Rapids, DRAM on node 0 and CXL on node 1. PEBS L3-miss at
`sample_freq` 5003 with `precise_ip` 2.
- Same workload shape.
## What the runs show
Per-node reference, each figure measured by binding the same workload to one
node:
```
Granite Rapids, 64 threads x 8 GiB
node 0 1.0 TB DRAM DRAM only 269,885 MB/s
node 1 2.0 TB CXL CXL only 249,903 MB/s
Turin, 32 threads x 4 GiB
node 0 386 GB DRAM DRAM only 113,596 MB/s
node 4 1.0 TB CXL CXL only 33,253 MB/s
```
- The two machines differ in the way that matters: on the Turin the far node
is 3.4x slower than the near one, on the Granite Rapids the two are within
8%.
- Each configuration gets one graph, the climb: the run itself, one mark per
decision taken from the bandwidth just measured, with the share it moved to
under each mark. The settled line below it is the median over the samples
the run went on to hold at that share.
### Configuration B, virtual-address mode
```
MB/s climb
460k | *
440k | *
420k | *
400k |
380k | *
360k | *
340k | * *
320k | *
300k | * *
280k | *
260k | * *
--------+------------+--------------+---------------+------------+--
1 4 7 10 13
92 80 68 56 44
decision index, near-node share (%)
settled share 44, 430,547 MB/s held
```
- Thirteen decisions to settle at 44%, no target given: about 60% more than
the better of the two nodes on its own, and the share does not change again
over the 81 samples that follow.
- Cold demotion runs over the same interval, in the same context, actuated by
the same kdamond: 34 regions totalling 15.4 GiB applied, and the idle
process ends with 4,099,980 pages -- 15.6 GiB -- on the CXL node having
started on DRAM.
- The promotion scheme's applied byte count is flat across the demotion ramp,
291.1 GiB before against 291.3 GiB after, so the recovered capacity is the
demotion scheme's.
### Configuration A, physical-address mode
```
MB/s climb
136k |
134k | * *
132k | * *
130k | *
128k |
126k | *
124k |
122k | *
120k |
118k | *
+----+-----+-----+-----+-----+-----+-----+-----+--
1 2 3 4 5 6 7 8
92 88 84 80 76 78 80 78
decision index, near-node share (%)
settled share 78, 134,107 MB/s held
```
- Eight decisions to settle at 78, from a start with almost everything on the
near node: the same algorithm reaches the optimum share on its own from the
bandwidth reading alone, and cold pages are demoted to the far node while it
does so.
- A distribution beats either end because both nodes supply at once: at the
settled share the near node is supplying 104,618 MB/s against the 113,596 it
reaches with the whole working set bound to it, so the further 37,669 MB/s
arriving over CXL is bandwidth it could not have supplied at any share.
- The idle process ends with 2,863,156 pages -- 10.9 GiB -- on the CXL node,
out of the 32 GiB it allocated on DRAM and stopped touching. Residency is
the attributable figure in this mode rather than a per-scheme applied count,
because the schemes share physical ranges.
### What the runs are meant to establish
Not the settled share and not the absolute bandwidth; those are properties of
these machines and these workloads. What reproduces is:
- A controller reading achieved bandwidth converges unattended, from a
configuration naming no target, to a distribution that beats either node on
its own, and holds it once found.
- A demotion scheme recovers near-node capacity in the same context while that
happens.
- One code path does both, steering `node_eligible_mem_bp` over physical
ranges system-wide on one machine and `DamosDest` weights over a named
process group on the other.
## Beyond a CPU PMU
Nothing above is specific to PEBS or IBS. A source qualifies if it can report
an accessed address to the ring, which is the only thing the substrate asks of
it. A CXL device's Hotness Monitoring Unit, whose r3.2 perf driver has been
posted [7] exposing its hotlist through an AUX buffer, or a custom monitoring
unit on an accelerator or a memory expander, reports exactly that, and a
backend delivering those reports through a perf event reaches the same drain,
the same probe hits and the same schemes already in the tree. Such a unit sees
the traffic that reaches its own device rather than a system-wide view, so what
it contributes is the hot set resident on that device: the tiering logic can
promote those pages to DRAM without the host having to detect that hotness
itself, and without depending on what produced the reports. That posting names
driving tiering from such a unit as the intent and the in-kernel step as future
work, and consuming it as a DAMON access-check primitive was raised there as
one way to get it; a backend on this substrate is that path.
[1] v1 of this series
https://lore.kernel.org/damon/20260529165640.820-1-ravis.opensrc@xxxxxxxxx/
[2] Roadmap for extending DAMON beyond pte-accessed bit
https://lore.kernel.org/damon/20260525225208.1179-1-sj@xxxxxxxxxx/
[3] mm/damon: introduce perf event based access check
https://lore.kernel.org/damon/20260423004211.7037-1-akinobu.mita@xxxxxxxxx/
[4] B. Tabatabai, R. Jonnalagadda et al., "Bandwidth Speaks, We Listen:
Dynamic Interleaving for Tiered Memory", ISMM 2026.
https://dl.acm.org/doi/10.1145/3814942.3816137
[5] mm/damon/perf: add ARM SPE AUX backend
https://lore.kernel.org/damon/20260816142222.689624-1-kunwu.chan@xxxxxxxxx/
[6] A platform-independent subsystem for bandwidth information, and resctrl
as that source
https://lore.kernel.org/linux-mm/d952a84f-332e-8f7a-4816-2c1cbd8f5b00@xxxxxxxxxx/
[7] CXL Hotness Monitoring Unit perf driver
https://lore.kernel.org/linux-mm/20241121101845.1815660-1-Jonathan.Cameron@xxxxxxxxxx/
[8] mm/damon: add node_eligible_mem_bp goal metric, merged for v7.2
https://lore.kernel.org/linux-mm/20260428030520.701-1-ravis.opensrc@xxxxxxxxx/
[9] mm/damon/vaddr: allow interleaving in migrate_{hot,cold} actions, merged
for v6.17
https://lore.kernel.org/linux-mm/20250709005952.17776-1-bijan311@xxxxxxxxx/
Ravi Jonnalagadda (9):
mm/damon/vaddr: support page fault access check primitive
mm/damon/core: read the CPU number with preemption disabled
mm/damon/paddr: lock the folio for the page fault primitive rmap walk
mm/damon: add damos_node_eligible_mem_bp tracepoint
mm/damon/core: add per-probe-class report rings and unified drain
mm/damon: add perf-event overflow handler feeding the report ring
mm/damon/ops-common: use probe-weighted score when probe weights are
set
mm/damon: add perf_event prep for PMU-driven hotness probes
mm/damon/tests/drain-kunit: kunit for report rings and unified drain
include/linux/damon.h | 136 ++++-
include/trace/events/damon.h | 32 +
mm/damon/Kconfig | 18 +
mm/damon/Makefile | 1 +
mm/damon/core.c | 995 +++++++++++++++++++++++++++----
mm/damon/ops-common.c | 21 +-
mm/damon/paddr.c | 37 +-
mm/damon/perf_source.c | 489 +++++++++++++++
mm/damon/perf_source.h | 54 ++
mm/damon/sysfs.c | 266 ++++++++-
mm/damon/tests/.kunitconfig | 4 +
mm/damon/tests/core-kunit.h | 2 +-
mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
mm/damon/tests/perf-kunit.h | 133 +++++
mm/damon/vaddr.c | 83 ++-
15 files changed, 3220 insertions(+), 142 deletions(-)
create mode 100644 mm/damon/perf_source.c
create mode 100644 mm/damon/perf_source.h
create mode 100644 mm/damon/tests/drain-kunit.h
create mode 100644 mm/damon/tests/perf-kunit.h
base-commit: e1f34dce183a96fc93bf9a42dcdc0ec0bf82a3d2
--
2.43.0