Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre]

Hi Ben,

On 7/20/26 6:30 AM, Ben Horgan wrote:
> Hi Reinette,
> 
> On 7/17/26 17:00, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 7/17/26 5:20 AM, Ben Horgan wrote:
>>> On 7/16/26 18:07, Reinette Chatre wrote:
>>>> On 7/16/26 9:44 AM, Ben Horgan wrote:
>>>>> On 7/16/26 17:04, Reinette Chatre wrote:
>>>>>> Hi Ben, Chenyu, and Tony,
>>>>>>
>>>>>> On 7/16/26 7:59 AM, Ben Horgan wrote:
>>>>>>> On 7/15/26 16:41, Reinette Chatre wrote:
>>>>>>>> On 7/15/26 1:34 AM, Ben Horgan wrote:
>>>>>>>>> On 7/14/26 23:06, Reinette Chatre wrote:
>>
>> ...
>>
>>>>>>>>> Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
>>>>>>>>> need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
>>>>>>>>> "mode" and can just toggle back if they want to use "MB" directly again.
>>>>>>>> Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
>>>>>>>> legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
>>>>>>>> resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
>>>>>>>> into the controls it is being emulated with. The only insight to this would continue to be via the
>>>>>>>> hierarchy in the info directory, which would show relationship but not the actual control values.
>>>>>>>> Do you think there could be a need for users (excluding validation?) that may want to see the underlying
>>>>>>>> control values used to emulate a legacy control?
>>>>>>>
>>>>>>> They may want to see the underlying values to be able to migrate there legacy configuration to the
>>>>>>> native configuration but they can just write their legacy configuration and then toggle the mode to
>>>>>>> see the values in the native mode.
>>>>>>>
>>>>>>>>
>>>>>>>> I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
>>>>>>>> here would be that users familiar with the new controls would forever need to switch the mode before being
>>>>>>>> able to use them.
>>>>>>>
>>>>>>> This does also make the new controls slightly harder to discover.
>>>>>>>
>>>>>> What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be
>>>>>> "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
>>>>>> but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
>>>>>> easier to discover and let user space see the underlying values, but not break a user space that
>>>>>> may read schemata file, change a few values, and write entire file back.
>>>>>
>>>>> My initial impression is that this would work but is probably unnecessary.
>>>>
>>>> ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of
>>>> pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find
>>>> unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.
>>>
>>> Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to
>>> change then doesn't the user get the same information with just the burden of toggling the mode and
>>> then reading the schemata a second time. This does rely on the user being aware of the new behaviour
>>> though.
>>>
>>> If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive?
>>> More on this below.
>>
>> There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect
>> changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface
>> is in use is to address your earlier point that not doing so will make new controls slightly harder to discover.
>> (more below)
>>
>>>>>> How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
>>>>>> MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
>>>>>> between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
>>>>>> example below.
>>>>>>
>>>>>> Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
>>>>>> MPAM and region aware MBA?
>>>>>>
>>>>>> (*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?
>>>>>
>>>>> I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up
>>>>> with a control state that isn't reachable just by configuring the legacy schema.
>>>>
>>>> Could you please highlight the scenario you refer to?
>>>
>>> My expectation here was that for any value of MB, X, the value for each of the regions would be the
>>> same, Y. (Not sure if this is actually the case.)
>>>
>>> MB: X
>>> # MB_REGION0_MAX:Y
>>> # MB_REGION1_MAX:Y
>>> # MB_REGION2_MAX:Y
>>> # MB_REGION3_MAX:Y
>>>
>>> So, if one of the regions was to be changed individually then it would be in a state not reachable
>>> by just changing the legacy control, MB. Potentially this complicated switching mode as well as
>>> uncommenting and writing a schemata.
>>
>> Indeed. This scenario was highlighted in slide 8 of
>> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>> and discussed between Tony and Dave Martin in the thread starting at
>> https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/
>>
>> Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions.
>> On a high level I understood that the underlying controls will always, as you state below, "show the
>> full story" and if user space interacts with them (instead of using the legacy control) then the
>> control values associated with the legacy control cannot be relied on to represent accurate state to
>> the point that it may even be better to not be visible. 
>> You are right, this is a good motivation to not allow user to modify underlying controls when in
>> "legacy" mode. 
>>
>> The remaining open is whether resctrl should display the (read-only) underlying control
>> values in schemata file when in "legacy" mode. By default this does not work for region-aware since
>> it will at least initially use different hardware interfaces for the different controls, essentially
>> this means that there is no actual emulation and the legacy and region aware control are both
>> legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".
>>
>> The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a
>> control to start with underlying controls disabled so that they are not displayed in schemata file
>> when legacy mode is enabled.
>>
>> If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer
>> be needed for region-aware. RISC-V also considered using this "status" that I think may also be
>> solved with this approach, I am not sure though.
>>
>>> I was thinking that in legacy mode that the region values would always be kept the same but perhaps
>>> the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the
>>> full story. 'best estimate' could be difficult to choose but allows switching mode to be
>>> non-destructive.
>>>
>>> For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting
>>> that MPAM would only use emulation for exposing finer grade controls to the user and not allowing
>>> different underlying hardware to be used for the emulation. To me, it seems reasonable that a new
>>> control would involve a new schema but I do appreciate the benefit of being able to continue to use
>>> the old interface without stopping a new interface being used. The danger is that there are
>>> unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different
>>> from the user expectations.
>>
>> I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level
>> problem to be that user space may have expectation that for every "resource" represented by a directory in
>> info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user
>> space tools is not obvious to me though: there is already an exception since there is the
>> "L3_MON" "resource" that does not have a schemata file entry.
>>
>> To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via
>> an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl
>> is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB
>> then resctrl should not be forced to always provide such emulation.
>>
>> I would appreciate your thoughts here.
> 
> Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for
> resource emulation.
> 
> As emulation adds complication and has the potential to break other user expectations I hope we can
> manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a
> resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains
> backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the
> NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.
> 
> Can we get around this by by choosing a different organization of the info/ directory? We already
> have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION)
> and move the scope to be a property at info/<resource>/scope rather than
> info/<resource>/resource_schemata/ctrl/scope.

Good point that L2 and L3, which is the same resource at different scope, are treated as different
resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources.
The point here is that it is at a different scope so more specifically what a "domain ID" in the
schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.

You will notice I get a bit lost in the discussion below so getting back here I would like to
clarify if you propose that info/ contains a directory for each allocation scope of each resource with that
directory containing descriptions of all the controls for that resource at the indicated scope or do
you propose that info/ contains a directory for each control?

> 
> Can MB_REGION be considered an orthogonal new control or does using it require that the traditional
> intel MB (delay) not be configured?

The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both
concurrently. Not sure if this answers your question.

Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA"
feature I am currently aware of the following possible controls:
MB_REGION0_MIN
MB_REGION0_MAX
MB_REGION0_OPT
MB_REGION1_MIN
MB_REGION1_MAX
MB_REGION1_OPT
MB_REGION2_MIN
MB_REGION2_MAX
MB_REGION2_OPT
MB_REGION3_MIN
MB_REGION3_MAX
MB_REGION3_OPT


> On an MPAM system:
> 
> info
> ├── MB
> │   ├── resource_schemata
> │   │   ├── MB
> │   │   └── MB_MAX
> │   └── scope
> └── MB_NODE
>     ├── resource_schemata
>     │   └── MB_NODE
>     └── scope

Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is
used by driver as the underlying hardware control for the percentage based MB exposed to
user space. Thus, when user space changes the "MB" control via schemata file it is expected
to also impact the underlying "MB_MAX" control. By representing them as above this relationship
is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control
to show this relationship. Looks like you plan to change this but from what I understand this
emulation is still relevant?

When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
from above that it would either start out by itself having the properties of the underlying
"MAX" control or is the plan to have it be a percentage based control backed by the
underlying "MAX" hardware control?

I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
info/MB_NODE/resource_schemata/<control>?

> 
> On an x86 system:
> 
> info
> ├── MB
> │   ├── resource_schemata
> │   │   ├── MB
> │   │   └── MB_MAX (Finer grained MB, more below)
> │   └── scope
> └── MB_REGION
>     ├── resource_schemata
>     │   └── MB_REGION
>     └── scope
> 
> 
> Do you think this helps?

"REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
Combine that with up to (currently) four regions each with three controls I find an interface like above
potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
scope?) and then *it* contains the twelve new controls within its resource_schemata directory?

Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
that is not presented in above hierarchy.

I think I am missing quite a bit here as I try to navigate an interface so different from what we have
discussed so far.
I would like to explore with more detail how this interface can handle the different scenarios we have
discussed so far.

The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
except for one configuration where it is "SYSTEM"(?) scope.
Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/

> 
> This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
> but it exposed a different interface to the user. If I understand correctly intel have an option to
> have finer grained control of MB (delay) as well and so it would make sense to use a common name
> rather than just going for the MPAM centric name of MB_MAX.

Apologies but I was not able to parse above.

Reinette