Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept
From: Fenghua Yu
Date: Thu Jul 23 2026 - 18:12:11 EST
Hi, Ben,
On 7/21/26 06:23, Ben Horgan wrote:
Hi Reinette,
On 7/20/26 23:54, Reinette Chatre wrote:
Hi Ben,
On 7/20/26 6:30 AM, Ben Horgan wrote:
Hi Reinette,
On 7/17/26 17:00, Reinette Chatre wrote:
Hi Ben,
On 7/17/26 5:20 AM, Ben Horgan wrote:
On 7/16/26 18:07, Reinette Chatre wrote:
On 7/16/26 9:44 AM, Ben Horgan wrote:
On 7/16/26 17:04, Reinette Chatre wrote:
Hi Ben, Chenyu, and Tony,
On 7/16/26 7:59 AM, Ben Horgan wrote:
On 7/15/26 16:41, Reinette Chatre wrote:
On 7/15/26 1:34 AM, Ben Horgan wrote:
On 7/14/26 23:06, Reinette Chatre wrote:
...
What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will beAlternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two neverInteresting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
"mode" and can just toggle back if they want to use "MB" directly again.
legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
into the controls it is being emulated with. The only insight to this would continue to be via the
hierarchy in the info directory, which would show relationship but not the actual control values.
Do you think there could be a need for users (excluding validation?) that may want to see the underlying
control values used to emulate a legacy control?
They may want to see the underlying values to be able to migrate there legacy configuration to the
native configuration but they can just write their legacy configuration and then toggle the mode to
see the values in the native mode.
I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
here would be that users familiar with the new controls would forever need to switch the mode before being
able to use them.
This does also make the new controls slightly harder to discover.
"legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
easier to discover and let user space see the underlying values, but not break a user space that
may read schemata file, change a few values, and write entire file back.
My initial impression is that this would work but is probably unnecessary.
ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of
pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find
unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.
Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to
change then doesn't the user get the same information with just the burden of toggling the mode and
then reading the schemata a second time. This does rely on the user being aware of the new behaviour
though.
If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive?
There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect
changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface
is in use is to address your earlier point that not doing so will make new controls slightly harder to discover.
(more below)
How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
example below.
Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
MPAM and region aware MBA?
(*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?
I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up
with a control state that isn't reachable just by configuring the legacy schema.
Could you please highlight the scenario you refer to?
My expectation here was that for any value of MB, X, the value for each of the regions would be the
same, Y. (Not sure if this is actually the case.)
MB: X
# MB_REGION0_MAX:Y
# MB_REGION1_MAX:Y
# MB_REGION2_MAX:Y
# MB_REGION3_MAX:Y
So, if one of the regions was to be changed individually then it would be in a state not reachable
by just changing the legacy control, MB. Potentially this complicated switching mode as well as
uncommenting and writing a schemata.
Indeed. This scenario was highlighted in slide 8 of
https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
and discussed between Tony and Dave Martin in the thread starting at
https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/
Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions.
On a high level I understood that the underlying controls will always, as you state below, "show the
full story" and if user space interacts with them (instead of using the legacy control) then the
control values associated with the legacy control cannot be relied on to represent accurate state to
the point that it may even be better to not be visible.
You are right, this is a good motivation to not allow user to modify underlying controls when in
"legacy" mode.
The remaining open is whether resctrl should display the (read-only) underlying control
values in schemata file when in "legacy" mode. By default this does not work for region-aware since
it will at least initially use different hardware interfaces for the different controls, essentially
this means that there is no actual emulation and the legacy and region aware control are both
legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".
The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a
control to start with underlying controls disabled so that they are not displayed in schemata file
when legacy mode is enabled.
If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer
be needed for region-aware. RISC-V also considered using this "status" that I think may also be
solved with this approach, I am not sure though.
I was thinking that in legacy mode that the region values would always be kept the same but perhaps
the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the
full story. 'best estimate' could be difficult to choose but allows switching mode to be
non-destructive.
For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting
that MPAM would only use emulation for exposing finer grade controls to the user and not allowing
different underlying hardware to be used for the emulation. To me, it seems reasonable that a new
control would involve a new schema but I do appreciate the benefit of being able to continue to use
the old interface without stopping a new interface being used. The danger is that there are
unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different
from the user expectations.
I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level
problem to be that user space may have expectation that for every "resource" represented by a directory in
info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user
space tools is not obvious to me though: there is already an exception since there is the
"L3_MON" "resource" that does not have a schemata file entry.
To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via
an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl
is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB
then resctrl should not be forced to always provide such emulation.
I would appreciate your thoughts here.
Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for
resource emulation.
As emulation adds complication and has the potential to break other user expectations I hope we can
manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a
resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains
backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the
NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.
Can we get around this by by choosing a different organization of the info/ directory? We already
have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION)
and move the scope to be a property at info/<resource>/scope rather than
info/<resource>/resource_schemata/ctrl/scope.
Good point that L2 and L3, which is the same resource at different scope, are treated as different
resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources.
The point here is that it is at a different scope so more specifically what a "domain ID" in the
schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.
Ok. This makes sense to me. I had confused myself about what MB_REGION is.
You will notice I get a bit lost in the discussion below so getting back here I would like to
clarify if you propose that info/ contains a directory for each allocation scope of each resource with that
directory containing descriptions of all the controls for that resource at the indicated scope or do
you propose that info/ contains a directory for each control?
The former, info/ contains a directory for each allocation scope of each resource.
info
├── L2
│ ├── resource_schemata
│ │ ├── L2
│ │ ├── L2_CMAX
│ │ └── L2_CMIN
│ └── scope : L2
├── L3
│ ├── resource_schemata
│ │ ├── L3
│ │ ├── L3_CMAX
│ │ └── L3_CMIN
│ └── scope : L3
├── MB
│ ├── resource_schemata
│ │ ├── MB
│ │ │ └── MB_MAX
│ │ ├── MB_MIN
│ │ ├── MB_PBM
│ │ └── MB_PROP
│ └── scope : L3
└── MB_NODE
├── resource_schemata
│ ├── MB_NODE_MAX
│ ├── MB_NODE_MIN
│ ├── MB_NODE_PBM
│ └── MB_NODE_PROP
└── scope : NUMA NODE
Can MB_REGION be considered an orthogonal new control or does using it require that the traditional
intel MB (delay) not be configured?
The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both
concurrently. Not sure if this answers your question.
Thanks, this answers my question.
Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA"
feature I am currently aware of the following possible controls:
MB_REGION0_MIN
MB_REGION0_MAX
MB_REGION0_OPT
MB_REGION1_MIN
MB_REGION1_MAX
MB_REGION1_OPT
MB_REGION2_MIN
MB_REGION2_MAX
MB_REGION2_OPT
MB_REGION3_MIN
MB_REGION3_MAX
MB_REGION3_OPT
On an MPAM system:
info
├── MB
│ ├── resource_schemata
│ │ ├── MB
│ │ └── MB_MAX
│ └── scope
└── MB_NODE
├── resource_schemata
│ └── MB_NODE
└── scope
Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is
used by driver as the underlying hardware control for the percentage based MB exposed to
user space. Thus, when user space changes the "MB" control via schemata file it is expected
to also impact the underlying "MB_MAX" control. By representing them as above this relationship
is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control
to show this relationship. Looks like you plan to change this but from what I understand this
emulation is still relevant?
Please just consider this a mistake. MB_MAX should be a child of MB.
info/
├── MB
│ ├── resource_schemata
│ │ └── MB
│ │ └── MB_MAX
│ └── scope
└── MB_NODE
├── resource_schemata
│ └── MB_NODE_MAX (Changed from MB_NODE)
└── scope
When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
from above that it would either start out by itself having the properties of the underlying
"MAX" control or is the plan to have it be a percentage based control backed by the
underlying "MAX" hardware control?
The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
but at a different location in the SoC, at the memory controller rather than in the L3.
To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
be MB_NODE_MAX.
My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
pattern for control names and the pattern for resource names being <resource_type>_<scope>.
(allowing for <scope> or <control_type> being missing to match existing naming.)
I don't think we should introduce more percent based controls and for new controls we can introduce
a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/
as discussed previously. Although, I have been pondering on whether we can do a bit better.
We could use hexadecimal point based format for controls which are a proportion of a resource and
have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
independent of the granularity of the control (number of parts).
0 is represented as 0x0
1 as 0x1
1/2 as 0x0.8
7/256 as 0x0.07
1/2**28 0x0.00000001
etc
This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
hope this would be useful for other architectures too. I am preparing some RFC patches on top of
your PoC for consideration of this idea and to explore some of the proposals discussed relating to
generic schemata and how they land in practice from the MPAM side.
I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
info/MB_NODE/resource_schemata/<control>?
Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
complications that it is a property of the MBW_MAX control and that it may be configurable for each
PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.
For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
except the meaning of the bit is reversed. We can just use a consistent name in s/w though.
Is it possible to view MBW_MAX hard limit, CMAX soft limit, or future similar things as a "configuration" for a control, instead of a "control" itself?
A "configurations" is different from a "control" in that:
1. The configuration configures the control, e.g. toggle hard limit on MB control.
2. The configuration doesn't have the control's properties (e.g. gran).
3. The configuration is similar to "event_configs" in MB_MON, that can also configure MB_MON's event_filter.
MBW_MAX hard limit configures MB/MB_MAX control's hard limit. CMAX soft limit configures L3 control's soft limit.
Is it OK to implement configurations in a control?
Then in struct resctrl_ctrl, add "list_head configs" to track all configurations in this control.
$ cat schemata
MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM_NODE is a configuration of MB_NODE control
MB_MAXHLIM_NODE:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
# MB_MAXHLIM is a configuration of MB control
MB_MAXHLIM:0=1;1=0;2=0;10=0;18=0;26=0;34=0;35=0
L3:1=ffff;2=ffff
info/
│ └── resource_schemata
│ ├── MB (simulated by MB_NODE)
│ │ ├── configs # all configs in MB control
│ │ │ └── MB_MAXHLIM # MB_MAXHLIM config in MB control
│ │ │ └── type # boolean
│ │ ├── max
│ │ ├── MB_NODE # simulated MB
│ │ │ ├── configs # all configs in MB_NODE control
│ │ │ │ └── MB_MAXHLIM_NODE # MB_MAXHLIM config in MB_NODE
│ │ │ │ └── type
│ │ │ ├── max
│ │ │ ├── min
│ │ │ ├── resolution
│ │ │ ├── scale
│ │ │ ├── scope
│ │ │ ├── status
│ │ │ ├── tolerance
│ │ │ ├── type
│ │ │ └── unit
│ │ ├── min
│ │ ├── resolution
│ │ ├── scale
│ │ ├── scope
│ │ ├── status
│ │ ├── tolerance
│ │ ├── type
│ │ └── unit
│ └── mode
Thanks.
On an x86 system:
info
├── MB
│ ├── resource_schemata
│ │ ├── MB
│ │ └── MB_MAX (Finer grained MB, more below)
│ └── scope
└── MB_REGION
├── resource_schemata
│ └── MB_REGION
└── scope
Do you think this helps?
"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
Combine that with up to (currently) four regions each with three controls I find an interface like above
potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
a new separate "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
that is not presented in above hierarchy.
OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
previously seem s better fit than what I suggested.
I think I am missing quite a bit here as I try to navigate an interface so different from what we have
discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have
discussed so far.
Certainly, I don't think we have got to the bottom of this yet.
The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/
Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?
Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular
domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.
This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
but it exposed a different interface to the user. If I understand correctly intel have an option to
have finer grained control of MB (delay) as well and so it would make sense to use a common name
rather than just going for the MPAM centric name of MB_MAX.
Apologies but I was not able to parse above.
Ok, let me try to explain again (although it's probably not what we want to do). My intent here was
to try and explore whether we can reuse naming and controls across architectures in the same way we
already have for the L2/L3 cache portion bitmap and the existing MB control.
To quote from a previous mail of yours:
https://lore.kernel.org/lkml/a84af037-6439-4362-be07-d45143e06309@xxxxxxxxx/
"""
For example, on an MPAM system (if I understand correctly) the user may see:
info/
└── MB/
└── resource_schemata/
├── MB/
│ └── MB_MAX/
└── MB_MIN/
Compared with a possible implementation on Intel that looks like:
info/
└── MB/
└── resource_schemata/
├── MB/
│ └── MB_OPT/
├── MB_MAX/
└── MB_MIN/
"""
In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the
legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't
make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.
Thanks,
Ben
Reinette
-Fenghua