[PATCH RFC v3 00/15] arm,fs/resctrl: ARM MPAM memory-level MSC and CPU-less NUMA nodes

[Fenghua Yu]

This series extends memory-bandwidth monitoring and control to ARM MPAM
memory-level MSCs, including CPU-less NUMA nodes, and documents the
resulting resctrl interfaces.

Background

On x86, MBA bandwidth control is typically associated with the L3 cache:
schemata lines use L3 cache ids and monitor domains appear as mon_L3_XX.
On some ARM MPAM systems the MBA control and MBWU counters are backed by
a memory-level MSC rather than the L3 cache MSC.  A memory-level MSC is
represented as a NUMA node, and that node commonly has no CPUs of its own
(a memory-only node that still participates in bandwidth control).

For that resource the schemata line is MB_NODE and its identifiers are
NUMA node ids.  Monitor domains are named mon_NODE_XX.  MBM counter
assignment is exposed as mbm_NODE_assignments.  An L3-cache MSC keeps the
MB schemata line, mon_L3_XX domains, and mbm_L3_assignments.

info/MB/control_mode stays legacy.  emulated_by is left empty, so legacy
and native mode expose the same schemata line.  MB is not emulated by
MB_NODE, and the assignment file name does not follow control_mode.

Because the memory-level MSC lives on a CPU-less NUMA node, the driver
must handle nodes with no local CPUs.  mpam_ris_get_affinity() derives
affinity from the component's NUMA node id, so a memory-only node would
otherwise end up with an empty CPU mask and never be registered.  When
the derived mask is empty, the driver falls back to cpu_possible_mask
(masked by MSC accessibility) so the node still gets a resctrl domain.
The borrowed affinity is kept out of class->affinity: those CPUs already
contribute through CPU-ful nodes in the class, and subtracting the
borrowed mask on teardown would remove CPUs that other live nodes still
depend on.  Domain setup iterates all MSC components per CPU (keyed by
component), so each component — including these CPU-less memory nodes —
gets its own control and monitor domain.  A memory hotplug lock and a
NUMA node notifier bring those domains online and offline.

Examples

** Memory-level MSC (for example a platform with CPU-less nodes).
   control_mode is legacy by default.  Both legacy and native mode show
   the same schemata line.  Domain ids are NUMA node ids.

info/MB/control_mode          # [legacy] native
info/MB/schemata/MB_NODE

schemata:
MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
     L3:1=ffff;2=ffff

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_NODE_01/mbm_total_bytes

CTRL_MON and MON groups expose mbm_NODE_assignments when MBM is backed
by the NUMA-node MSC.

** L3-cache MSC only (no memory-level MSC).  Both modes show MB:.
   Domain ids are L3 cache ids.

info/MB/schemata/MB

schemata:
MB:1=100;2=100
L3:1=ffff;2=ffff

mon_data/mon_L3_01/llc_occupancy
mon_data/mon_L3_01/mbm_total_bytes

CTRL_MON and MON groups expose mbm_L3_assignments.

The patches can be compiled but not tested on x86 yet.

The 15 patches are grouped as follows:

1. resctrl monitor infrastructure (patches 1-5)

   Generalise the monitor infrastructure beyond hard-wired L3 paths:
   de-hardcode L3 monitor plumbing, expose MBA MBM counter assignment as
   mbm_NODE_assignments, name node-scoped monitor domains mon_NODE_<id>,
   add a node-scope MBM total event, and make MBM handling resource-aware.

2. ARM MPAM memory-level MSC (patches 6-8)

   Map monitor classes to L3 or MBA, initialise ABMC on the resource that
   owns the MBWU monitor, refine class selection so a memory class can
   back MBWU, and iterate all MSC components per CPU when creating
   domains.  A memory-class MSC names its control NODE, so the schemata
   line is MB_NODE.  A cache MSC keeps MB.

3. CPU-less NUMA nodes (patches 9-11)

   Take the memory hotplug lock with the CPU hotplug lock, fall back to
   cpu_possible_mask for a memory MSC with no local CPUs, and add a NUMA
   node notifier so CPU-less domains come online and offline.

4. Documentation (patches 12-14)

   Document mbm_NODE_assignments, memory-level MB control and NUMA nodes
   in the arm64 MPAM guide, and MB_NODE schemata lines and mon_NODE_*
   monitoring in the resctrl documentation.

5. Test (patch 15)

   Add a KUnit test for CPU-less NUMA node affinity handling.

Apply the patches on top of [1].

Testing

  # KUnit (CPU-less affinity)
  sudo cat /sys/kernel/debug/kunit/mpam_devices_test_suite/results

This series is in https://github.com/fyu1/linux/  cpu_less.rfc.v3/
which is on top of [1].

Change Log:
RFC v3:
- Drop MB control emulation.  MB is no longer emulated by MB_NODE.
  control_mode stays legacy and emulated_by is empty, so legacy and
  native mode show the same schemata line.  A memory-level MSC shows
  "MB_NODE:" and uses NUMA node ids.  An L3-cache MSC shows "MB:" and
  uses L3 cache ids (Ben).
- Drop the MB emulation kselftest and the emulation documentation (Ben).
- Drop mbm_assign_scope_mode.  "mbm_NODE_assignments" is exposed when
  MBM is backed by a NUMA-node MSC.  "mbm_L3_assignments" is exposed
  when MBM is backed by an L3-cache MSC.  The file name does not follow
  control_mode.
- Series is now 15 patches.

RFC v2:
- Remove fix patches 1-2 which are implemented in [1]
- Remove the emulation patches 3-8 which are implemented in [1]
- Update emulated_by to *emulated_by in patch 11. Ben seems
  doesn't like to emulate MB by MB_NODE. If that's case, I can
  remove this patch. But for now, I keep this patch for further
  discussion.
- Add patch 13 to implement legacy and native modes to emulate
  "mbm_L3_assignments" by "mbm_NODE_assignments" in patch 13.
- Add patch 9 for memory hotplug locking (Ben)
- Add patch 12 for memory hotplug notifier (Ben)
RFC v2:
https://lore.kernel.org/lkml/20260831172245.42253-1-fenghuay@xxxxxxxxxx/

RFC v1:
https://lore.kernel.org/lkml/cover.1784217438.git.fenghuay@xxxxxxxxxx/

[1] Reinette's resctrl control branch RFC v2: git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v2.4

Fenghua Yu (13):
  resctrl: De-hardcode L3 monitor infrastructure
  resctrl: Expose MBA MBM counter assignment sysfs
  resctrl: name node-scoped monitor domains mon_NODE_<id>
  resctrl: Add node-scope MBM total event
  resctrl: Make MBM paths resource-aware
  arm_mpam: Support memory-level MSCs and ABMC per class
  arm_mpam: Refine L3 topology and class selection
  arm_mpam: Include all MSC components during domain setup
  arm_mpam: Handle CPU-less numa nodes
  Documentation: resctrl: document mbm_NODE_assignments
  Documentation: arm64: mpam: document memory-level MB control and NUMA
    nodes
  Documentation: resctrl: document NUMA-node MBA domains and mon_NODE
    monitoring
  arm_mpam: Add KUnit test for CPU-less NUMA node affinity

James Morse (2):
  fs/resctrl: Take memory hotplug lock whenever taking CPU hotplug lock
  arm_mpam: resctrl: Add NUMA node notifier for domain online/offline

 Documentation/arch/arm64/mpam.rst     |  95 ++--
 Documentation/filesystems/resctrl.rst |  54 ++-
 drivers/resctrl/mpam_devices.c        |  59 ++-
 drivers/resctrl/mpam_internal.h       |   8 +
 drivers/resctrl/mpam_resctrl.c        | 664 +++++++++++++++++++++-----
 drivers/resctrl/test_mpam_devices.c   |  67 +++
 fs/resctrl/ctrlmondata.c              |   6 +-
 fs/resctrl/internal.h                 |   9 +-
 fs/resctrl/monitor.c                  | 226 ++++++---
 fs/resctrl/pseudo_lock.c              |   3 +
 fs/resctrl/rdtgroup.c                 | 148 ++++--
 include/linux/memory.h                |   1 +
 include/linux/resctrl.h               |  31 +-
 include/linux/resctrl_types.h         |  15 +-
 14 files changed, 1079 insertions(+), 307 deletions(-)

-- 
2.53.0