Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
From: Fenghua Yu
Date: Wed Jul 22 2026 - 20:17:35 EST
Hi, Reinette,
On 7/14/26 15:06, Reinette Chatre wrote:
Hi Fenghua,
On 7/10/26 1:59 PM, Fenghua Yu wrote:
On 6/25/26 08:43, Reinette Chatre wrote:
On 6/24/26 6:26 PM, Fenghua Yu wrote:
On 6/24/26 15:22, Reinette Chatre wrote:
On 6/24/26 12:08 PM, Fenghua Yu wrote:
On 5/29/26 11:06, Reinette Chatre wrote:
As Shaopen and Ben mentioned earlier, we are working on two MPAM
features that may need to change schemata interface. The CPU-less
feature was discussed on LPC (although the interfaces will be
slightly different from the LPC).
I know. Here is where I tried to engage with you on needed interfaces after LPC:
https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@xxxxxxxxx/
MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
1. L3 and memory together on each processor MSC
2. L3 in processor MSC and memory control/monitoring in different memory MSCs.
Ben said there is type 3 platform:
3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.
On type 1 platform, schemata is legacy:
MB:1=100;2=100 <-- cache id 1 and 2 as domain id
On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
26 as domain id
On type 2 platform, there won't be "MB:" line. Numa 0 and 1
are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
memory nodes allocation.
On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:
MB:1=100;2=100 <-- cache id 1 and 2 as domain id
MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10,
18, 26 as domain id
(to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it
explicit that it is memory bandwidth allocation at node scope)
I am trying to consider how this can be accomplished while also considering all the other
new hardware features that resctrl need to support. Consider, for example, AMD's "Global
MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@xxxxxxx/) that throttles
memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time
the plan is to support this with a second control associated with the MB resource that can
allocate memory bandwidth at node scope. See
https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@xxxxxxx/
If resctrl creates a new resource for node scoped memory bandwidth allocations to support these
"type 2" systems then that will result in an inconsistent interface between architectures that
we should avoid.
Have you been listening in on the discussions surrounding emulated controls? Considering that,
would it be possible to support the "MB" control on a type "2" system but have it be backed by
(emulated by) the underlying "MB_NODE" control?
resctrl could expose both controls on these "type 2" systems but make it clear that "MB"
is emulated by "MB_NODE". For example:
info/
└── MB/
└── resource_schemata/
└── MB/
└── MB_NODE/
User will see both controls in schemata file but when changes are made to "MB" control it
will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control
that will establish familiarity with the interface at which point resctrl can drop the
"MB" control from the schemata file on these "type 2" systems.
Having the MB resource available with an MB control will keep resctrl backward compatible
if there are any tools that expect that. If backward compatibility is not of concern then
resctrl could initialize with the emulated control disabled by default. See discussion at
https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@xxxxxxxxx/
that describes a new resctrl capability in support of RISC-V and RDT.
With this resctrl could initialize with:
info/
└── MB/
└── resource_schemata/
├── MB/
│ ├── MB_NODE/
│ │ └── status:enabled
│ └── status:disabled
└── mode:legacy [native]
With above a "type 2" system will boot with its schemata file just containing the "MB_NODE"
control while info/MB describes the memory bandwidth resource.
On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.
Is this directory OK?
info/
└── MB/
└── resource_schemata/
├── MB/
│ ├── MB_NODE/
│ │ └── status:disabled
│ └── status:enabled
├── MB_NODE/
└── mode:node
1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/
I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE
hardware control exists.
2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node
I assume you mean "scope" instead of "mode"? (more below)
3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)
Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.
From what I understand these "type 3" machines could be simplified to:
info/
└── MB/
└── resource_schemata/
├── MB/
│ └── scope:L3
└── MB_NODE/
└── scope:NODE
Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want
to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface
may end up looking like:
info/
└── MB/
└── resource_schemata/
├── MB/
│ ├── MB_MAX/
│ │ └── scope:L3
│ └── scope:L3
└── MB_NODE/
└── scope:NODE
A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE
Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control
so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE
on these systems.
Would that work for these systems?
...
There is another MPAM feature called MBW Max hardlimit which sets
"MB:" allocation as hardlimit (i.e. MBW throttling percentage must
be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
percentage on each domain. By default hardlimit is off (0) and can
be turned on to set MBW Max hardlimit on a domain.
ack. This sounds like a new control associated with the MB resource.
This is a boolean control as Dave highlighted in previous discussion so
resctrl would need to know its properties.
See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@xxxxxxxxxxxxxxx/
Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)
For exmple:
MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
MB:0=100;1=100;2=80;10=100;18=100;26=100
On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
ack.
L3:0=fff;1=fff
# echo 'MB_MIN:0=50' > schemata
# cat schemata
MB_MAX:0=100;1=100
MB_MIN:0=50;1=100
MB:0=100;1=100
L3:0=fff;1=fff
Writing to the dummy control will call a dummy callback that just prints to the
kernel log:
"resctrl: Updata temporary MIN control on domain 0 with user value 50"
Example output of info/MB/:
/sys/fs/resctrl/info/MB/thread_throttle_mode:max
/sys/fs/resctrl/info/MB/num_closids:15
/sys/fs/resctrl/info/MB/delay_linear:1
/sys/fs/resctrl/info/MB/min_bandwidth:10
Add two new MB info RO files:
1. /sys/fs/resctrl/info/MB/domain_id
It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
This proposal introduces a *global* property to the MB *resource*? It does not seem as though
this takes into account *anything* about how resctrl can support new hardware that has been
discussed before, during, or after LPC. You have not participated in these discussions and
now make an orthogonal proposal that does not take into account *any* of the requirements
that we have been struggling with for months.
Why should this proposal be taken seriously? In your absence folks have been trying to
accommodate how these upcoming products and be supported and the "scope" file associated with
a control is intended to communicate to user space how the domain ID should be interpreted.
Why are you proposing something entirely different here without even acknowledging current
approach and explaining why it does not work for you?
So can I change this part to adding the following files in info dirctory?
1. For numa memory bw allocation (MBN):
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
/sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
This is not just about adding files to the info directory. The files, directories, their relationships,
and content have meaning. All I see from these proposals is an attempt to slap some new files into
resctrl without any consideration to present consistent interface to users and without consideration of
other architectures that need to be supported by resctrl.
resctrl needs to provide a generic and consistent interface to user space irrespective of the
underlying architecture. Architectures cannot just slap some new files for their convenience.
2. /sys/fs/resctrl/info/MB/max_lim
It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
Again this adds another *global* property to the MB resource but then above you
describe the new "MB_HLIM" schemata file entry that implies that it is a new control
for the MB resource. Having it be a new control for the MB resource matches earlier
discussions. To support this I thus expect it to be exposed as a new control with
potentially a new type if any of the existing planned types do not suffice.
How about adding these MB_HLIM dir and files in info?
/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
/sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
This presents "MB_HLIM" as a *resource* to user space. It is not a resource
but a *control* of a resource, no? I thus expect it to instead look something like
below that makes it clear that MB_HARDMAX is a control of the MB resource.
info
└── MB
└── resource_schemata
├── MB
└── MB_HARDMAX
Yes, this makes sense. I have changed to this hierarchy.
Thank you very much for considering this approach.
[ MB_MAXHLIM: I use this name for MBW_MAX hard limit feature as Dave Martin suggested before. He also suggested MB_HARDMAX. Either name is good for me. I use MB_MAXHLIM to explain MBW_MAX hard limit for now.]
Some implementation thoughts:
MBW_MAX hard limit itself is not a MB control. Rather, it configures MB control, i.e. turn on MB control's hard limit or turn off its hard limit. So MBW_MAX hard limit doesn't have properties like bandwidth_gran, delay_linear, etc. MBW_MAX hard limit's property is only a boolean type.
So I would think it maybe a configuration inside a control.
Similar configurations could be hard limit for cache capacity in MPAM.
Maybe can add "configs" inside resctrl_ctrl. Schemata and info/MB/resource_schemata/MB will show/write the configurations per control?
For this configuration or future configurations, add "configs" list in:
struct resctrl_ctrl {
struct list_head entry;
enum resctrl_scope scope;
struct list_head domains;
enum resctrl_ctrl_type type;
enum resctrl_ctrl_name name;
struct resctrl_ctrl *emulated_by;
struct list_head configs; <--- Add configs for this control
union {
struct resctrl_cache cache;
struct resctrl_membw membw;
};
};
A resctrl control can have one or multiple configurations. Currently MBW_MAX hard limit is the only one. But the infrastrucutre supports multiple configurations per control.
schemata:
MB:1=100 <-- MBW_MAX on L3 id 1
MB_MAXHLIM:1=0 <-- turn on/off MBW_MAX hardlimit on L3 id 1
L3:1=fff
info/
├── MB
│ ├── bandwidth_gran
│ ├── delay_linear
│ ├── min_bandwidth
│ ├── num_closids
│ └── resource_schemata
│ ├── MB
│ │ ├── configs
│ │ │ └── MB_MAXHLIM
│ │ │ └── type <--- bool
│ │ ├── max
│ │ ├── min
│ │ ├── resolution
│ │ ├── scale
│ │ ├── scope
│ │ ├── status
│ │ ├── tolerance
│ │ ├── type
│ │ └── unit
│ └── mode
Is this a valid way to handle MB_MAX hard limit (and future more configurations per control)?
Thanks.
-Fenghua