[RFC PATCH 31/31] x86,fs/resctrl: Update Documentation for region aware RDT

[Chen Yu]

Support region based MBM display and MBA schemata, update
the document accordingly.

Signed-off-by: Chen Yu <yu.c.chen@xxxxxxxxx>
---
 Documentation/filesystems/resctrl.rst | 110 ++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)

diff --git a/Documentation/filesystems/resctrl.rst b/Documentation/filesystems/resctrl.rst
index e4b66af55ffb..cac008996eba 100644
--- a/Documentation/filesystems/resctrl.rst
+++ b/Documentation/filesystems/resctrl.rst
@@ -28,6 +28,7 @@ SMBA (Slow Memory Bandwidth Allocation)				""
 BMEC (Bandwidth Monitoring Event Configuration)			""
 ABMC (Assignable Bandwidth Monitoring Counters)			""
 SDCIAE (Smart Data Cache Injection Allocation Enforcement)	""
+RMBA (Region Aware Memory Bandwidth Allocation)			""
 =============================================================== ================================
 
 Historically, new features were made visible by default in /proc/cpuinfo. This
@@ -651,6 +652,11 @@ When monitoring is enabled all MON groups will also contain:
 	"core" of the CPU (arithmetic units, TLB, L1 and L2 caches, etc.). They
 	do not include L3 cache, memory, I/O devices etc.
 
+	If region aware RDT is enabled, several region related files are created.
+	For example, if there are 4 regions, the corresponding files are
+	mbm_region0_bytes, mbm_region1_bytes, mbm_region2_bytes and
+	mbm_region3_bytes.
+
 	All other events report decimal integer values.
 
 	In a MON group these files provide a read out of the current value of
@@ -993,6 +999,56 @@ is formatted as:
 
 	SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
 
+Region Aware Memory Bandwidth Allocation and Monitor
+----------------------------------------------------
+Intel hardware supports Region-Aware Memory Bandwidth Allocation (MBA)
+and Region-Aware Memory Bandwidth Monitoring (MBM). With Region-Aware
+MBA, independent bandwidth control (throttling) of L3 domain bandwidth
+to multiple regions is supported, enabling users to dynamically rebalance
+bandwidth control limits across different memory regions, each of which
+may have distinct bandwidth, latency, and capacity characteristics.
+Region-Aware MBM includes the capability to independently track multiple
+domains that are simultaneously accessing several memory regions. These
+memory regions correspond to different levels of memory tiers, such as
+directly attached memory (Tier 1), CXL-attached memory (Tier 2), and
+CXL accelerator devices with attached memory. Note, the region ID is
+per socket scope. Intel platform supports up to 4 regions for now.
+
+Suppose there are 2 regions in each socket:
+
+  +------------------------+------------------------+
+  | +--------+ +--------+  | +--------+ +--------+  |
+  | | L3_00  | | L3_01  |  | | L3_02  | | L3_03  |  |
+  | +--------+ +--------+  | +--------+ +--------+  |
+  | +-------------------+  | +-------------------+  |
+  | |      IMC1         |  | |      IMC2         |  |
+  | +-------------------+  | +-------------------+  |
+  | +-------+   +-------+  | +-------+   +-------+  |
+  | | DDR1  |   |  CXL1 |  | | DDR2  |   |  CXL2 |  |
+  | +-------+   +-------+  | +-------+   +-------+  |
+  |  socket0               |  socket1               |
+  +------------------------+------------------------+
+
+In above graph, from the perspective of a CPU associated with
+the L3_00 domain, Region 0 typically represents the local memory
+region (DDR1), while Region 2 typically represents the remote
+memory region (DDR2). Similarly, for CPUs attached to L3_00,
+Region 1 represents the local memory region (CXL1), and Region 3
+represents the remote memory region (CXL2). Here the terms "local"
+and "remote" here are defined at the socket level.
+
+Take the region aware MBM for example. For the L3_00 domain, the
+memory bandwidth of Region 0 refers to the data transferred when
+the L3 miss occurs in L3_00 and the data is refilled from DDR1 -
+note, data refilled from L3_01 to L3_00 is not counted in. The
+bandwidth of Region 2, by contrast, refers to the data refilled
+from DDR2. Similarly, the same calculation logic applies to
+Region 1 (CXL1) and Region 3 (CXL2).
+
+For Region-Aware MBA, when setting values for Region 0 on CPUs
+attached to L3_00, this configuration controls the traffic generated
+when data is transferred between cores and L3_00 targeting DDR1.
+
 Reading/writing the schemata file
 ---------------------------------
 Reading the schemata file will show the state of all resources
@@ -1048,6 +1104,60 @@ For example, to allocate 8GB/s limit on the first cache id:
       MB:0=2048;1=2048;2=2048;3=2048
       L3:0=ffff;1=ffff;2=ffff;3=ffff
 
+Reading/writing the schemata file (on Intel systems) with region MBA feature
+----------------------------------------------------------------------------
+The schemata file provides fine grained control over each memory region.
+Every region has its own set of controls, and each of them appears as a
+separate line named "MB_REGION<n>_<type>", where <n> is the region number
+and <type> is one of OPT, MIN or MAX. Only the control types the hardware
+reports in the MARC ACPI sub-table are present.
+
+The legacy "MB" control is emulated by the MAX control of every region, so
+writing "MB" throttles all regions at once, while writing an individual
+"MB_REGION<n>_MAX" line throttles just that region.
+
+Reading and writing the schemata file is illustrated below for a platform
+with 2 memory regions. For example, to adjust the tier1 local memory
+bandwidth (usually the DDR):
+
+::
+
+  # cat schemata
+              MB:0=100;1=100;2=100;3=100
+  MB_REGION0_OPT:0=511;1=511;2=511;3=511
+  MB_REGION0_MIN:0=511;1=511;2=511;3=511
+  MB_REGION0_MAX:0=511;1=511;2=511;3=511
+  MB_REGION1_OPT:0=511;1=511;2=511;3=511
+  MB_REGION1_MIN:0=511;1=511;2=511;3=511
+  MB_REGION1_MAX:0=511;1=511;2=511;3=511
+              L3:0=3ff;1=3ff;2=3ff;3=3ff
+
+  # echo "MB_REGION0_MAX:1=200" > schemata
+  # cat schemata
+              MB:0=100;1=100;2=100;3=100
+  MB_REGION0_OPT:0=511;1=511;2=511;3=511
+  MB_REGION0_MIN:0=511;1=511;2=511;3=511
+  MB_REGION0_MAX:0=511;1=200;2=511;3=511
+  MB_REGION1_OPT:0=511;1=511;2=511;3=511
+  MB_REGION1_MIN:0=511;1=511;2=511;3=511
+  MB_REGION1_MAX:0=511;1=511;2=511;3=511
+              L3:0=3ff;1=3ff;2=3ff;3=3ff
+
+The value range of a region control is reported by the hardware and is
+available under the resource_schemata directory of that control, together
+with the rest of its properties.
+
+Users can obtain tiering information via sysfs, specifically
+through the path /sys/bus/memory_tier/devices/memory_tier.<id>/nodelist.
+The user can query the /sys/firmware/acpi/memory_ranges to find
+the corresponding node id for a specific region number.
+
+Once users have the corresponding node numbers, they can use the command
+numactl -H to retrieve additional node-related information if they need
+to further understand what each node represents. This information includes
+details such as the CPUs associated with the node (if any), the memory size
+of each node, and the distance between nodes.
+
 Cache Pseudo-Locking
 ====================
 CAT enables a user to specify the amount of cache space that an
-- 
2.43.0