[RFC PATCH v2 0/9] fs/resctrl: Support mba_MBps on MPAM

[Jinjie Ruan]

Resctrl offers two ways to describe an MBA policy:

- The percentage interface writes a fraction of the maximum bandwidth
  into the hardware control register

- The MBA software controller (mounted with -o mba_MBps) takes
  an absolute target in MiB/s and adjusts that percentage based on
  the MBM counters sampled once per second.

The problem is that a percentage alone isn't enough. How much bandwidth
a percentage actually corresponds to depends on the memory frequency,
the channel count, how many threads are generating traffic, the access
pattern, and the traffic of other groups. And neither MPAM nor x86
resctrl reports the maximum bandwidth in MiB/s. So an administrator who
wants to enforce an absolute budget (a tenant SLA, a noisy-neighbour
limit) has to measure and re-tune the percentage on every machine. x86
resctrl has had this software controller since 2018; MPAM has never been
able to use it.

This series makes mba_MBps work on MPAM.

What this series does
=====================

MPAM refuses mba_MBps in two places:

- Its MBM counters are assignable, and supports_mba_mbps() refuses
  the option when they are.

- Its maximum bandwidth control is a soft limit by default, which
  can't hold a single requester down to the target.
  MPAMF_MBW_IDR.MAX_LIM reports whether the hard limit exists.

MPAM's monitor groups also each take a bandwidth counter from a shared
pool, while nothing relates the number of groups to the number of
counters: a group created once the pool is empty is invisible to the
controller.

MPAM also makes two generic defects reachable:

- Writing the control register sleeps (smp_call_on_cpu()), while
  the worker runs under the RCU read lock

- A 16-bit fraction lets the minimum control value reach zero, which
  divides by zero in the estimator

The patches:

- Patches 1 to 3 fix the two generic defects

- Patches 4 and 5 make the loop runnable on MPAM: the worker takes the
  CPU hotplug lock around the sleeping control update, and assignable
  counters are accepted with a fixed assignment mode

- Patches 6 and 7 handle the maximum: mba_MBps is refused where only
  the soft limit exists, and the driver sets HARDLIM.

- Patches 8 and 9 cap the CLOSID/RMID space resctrl may use to the
  number of counters while the controller is enabled, so that every
  group it measures has a counter.

x86 has no assignable counters in the mode where the controller is
allowed, and does not set the soft-limit flag, so its behaviour is
unchanged.

Changes in RFC v2
=============

- Extract the step decision into a helper, with no behaviour change
- Use bw_gran instead of min_bw in the estimate
- Drop the warning when a monitor group gets no counter, as
  the new cap makes it unreachable (Ben)
- While the controller is enabled, cap the CLOSID/RMID space to the
  number of bandwidth counters (Ben)
- Link to RFC v1:
 - https://lore.kernel.org/all/20261109132621.3862497-1-ruanjinjie01@xxxxxxxxx/
 - https://lore.kernel.org/all/dc1398f4-b81c-4ad8-b706-df89a1622e6c@xxxxxxx/

Testing
=======

With the MPAM MSC model in QEMU:

- From a 9595 MiB/s baseline, a 7000 MiB/s target converges to
  6978 MiB/s and a 9000 MiB/s target to 8732 MiB/s.

- The memory-side MSC register shows MBW_MAX.HARDLIM set.

- A modelled MSC that implements only the soft limit refuses the
  mba_MBps mount, while a plain mount succeeds.

- With 8 counters and 32 PARTIDs, an mba_MBps mount reports 8 CLOSIDs
  and 8 RMIDs, and the 8th mkdir fails with ENOSPC instead of
  creating a group the controller cannot see

- With 4 PMGs and 64 counters, a plain mount reports 32 CLOSIDs and
  128 RMIDs while an mba_MBps mount reports 32 and 64: only the RMIDs
  per CLOSID shrink

- With mbm_assign_on_mkdir forced on while the controller is in use,
  a write of 0 is refused, and the user's value is restored after
  unmount

- The KUnit tests for the MPAM driver pass.

Jinjie Ruan (9):
  fs/resctrl: Extract the software controller's step decision
  fs/resctrl: Avoid division by zero in the MBA software controller
  fs/resctrl: Do not update bandwidth from a failed counter read
  fs/resctrl: Take the CPU hotplug lock for software controller updates
  fs/resctrl: Allow mba_MBps with assignable MBM counters
  fs/resctrl: Reject mba_MBps when the maximum is only a soft limit
  arm_mpam: Program the hard limit for the software controller
  x86,fs/resctrl: Let the architecture cap the RMIDs for the software
    controller
  arm_mpam: Cap the CLOSID/RMID space by the bandwidth counters for
    mba_MBps

 Documentation/filesystems/resctrl.rst |   7 +-
 arch/x86/include/asm/resctrl.h        |   6 ++
 drivers/resctrl/mpam_devices.c        |  55 +++++++++---
 drivers/resctrl/mpam_internal.h       |   4 +
 drivers/resctrl/mpam_resctrl.c        |  56 +++++++++++-
 fs/resctrl/monitor.c                  | 123 +++++++++++++++++++-------
 fs/resctrl/rdtgroup.c                 |  66 +++++++++++---
 include/linux/arm_mpam.h              |   2 +
 include/linux/resctrl.h               |   2 +
 9 files changed, 259 insertions(+), 62 deletions(-)

-- 
2.34.1