Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Ben Horgan]

Hi Reinette,

On 7/20/26 23:54, Reinette Chatre wrote:
> Hi Ben,
> 
> On 7/20/26 6:30 AM, Ben Horgan wrote:
>> Hi Reinette,
>>
>> On 7/17/26 17:00, Reinette Chatre wrote:
>>> Hi Ben,
>>>
>>> On 7/17/26 5:20 AM, Ben Horgan wrote:
>>>> On 7/16/26 18:07, Reinette Chatre wrote:
>>>>> On 7/16/26 9:44 AM, Ben Horgan wrote:
>>>>>> On 7/16/26 17:04, Reinette Chatre wrote:
>>>>>>> Hi Ben, Chenyu, and Tony,
>>>>>>>
>>>>>>> On 7/16/26 7:59 AM, Ben Horgan wrote:
>>>>>>>> On 7/15/26 16:41, Reinette Chatre wrote:
>>>>>>>>> On 7/15/26 1:34 AM, Ben Horgan wrote:
>>>>>>>>>> On 7/14/26 23:06, Reinette Chatre wrote:
>>>
>>> ...
>>>
>>>>>>>>>> Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
>>>>>>>>>> need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
>>>>>>>>>> "mode" and can just toggle back if they want to use "MB" directly again.
>>>>>>>>> Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
>>>>>>>>> legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
>>>>>>>>> resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
>>>>>>>>> into the controls it is being emulated with. The only insight to this would continue to be via the
>>>>>>>>> hierarchy in the info directory, which would show relationship but not the actual control values.
>>>>>>>>> Do you think there could be a need for users (excluding validation?) that may want to see the underlying
>>>>>>>>> control values used to emulate a legacy control?
>>>>>>>>
>>>>>>>> They may want to see the underlying values to be able to migrate there legacy configuration to the
>>>>>>>> native configuration but they can just write their legacy configuration and then toggle the mode to
>>>>>>>> see the values in the native mode.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
>>>>>>>>> here would be that users familiar with the new controls would forever need to switch the mode before being
>>>>>>>>> able to use them.
>>>>>>>>
>>>>>>>> This does also make the new controls slightly harder to discover.
>>>>>>>>
>>>>>>> What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be
>>>>>>> "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
>>>>>>> but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
>>>>>>> easier to discover and let user space see the underlying values, but not break a user space that
>>>>>>> may read schemata file, change a few values, and write entire file back.
>>>>>>
>>>>>> My initial impression is that this would work but is probably unnecessary.
>>>>>
>>>>> ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of
>>>>> pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find
>>>>> unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.
>>>>
>>>> Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to
>>>> change then doesn't the user get the same information with just the burden of toggling the mode and
>>>> then reading the schemata a second time. This does rely on the user being aware of the new behaviour
>>>> though.
>>>>
>>>> If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive?
>>>> More on this below.
>>>
>>> There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect
>>> changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface
>>> is in use is to address your earlier point that not doing so will make new controls slightly harder to discover.
>>> (more below)
>>>
>>>>>>> How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
>>>>>>> MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
>>>>>>> between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
>>>>>>> example below.
>>>>>>>
>>>>>>> Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
>>>>>>> MPAM and region aware MBA?
>>>>>>>
>>>>>>> (*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?
>>>>>>
>>>>>> I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up
>>>>>> with a control state that isn't reachable just by configuring the legacy schema.
>>>>>
>>>>> Could you please highlight the scenario you refer to?
>>>>
>>>> My expectation here was that for any value of MB, X, the value for each of the regions would be the
>>>> same, Y. (Not sure if this is actually the case.)
>>>>
>>>> MB: X
>>>> # MB_REGION0_MAX:Y
>>>> # MB_REGION1_MAX:Y
>>>> # MB_REGION2_MAX:Y
>>>> # MB_REGION3_MAX:Y
>>>>
>>>> So, if one of the regions was to be changed individually then it would be in a state not reachable
>>>> by just changing the legacy control, MB. Potentially this complicated switching mode as well as
>>>> uncommenting and writing a schemata.
>>>
>>> Indeed. This scenario was highlighted in slide 8 of
>>> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
>>> and discussed between Tony and Dave Martin in the thread starting at
>>> https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/
>>>
>>> Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions.
>>> On a high level I understood that the underlying controls will always, as you state below, "show the
>>> full story" and if user space interacts with them (instead of using the legacy control) then the
>>> control values associated with the legacy control cannot be relied on to represent accurate state to
>>> the point that it may even be better to not be visible. 
>>> You are right, this is a good motivation to not allow user to modify underlying controls when in
>>> "legacy" mode. 
>>>
>>> The remaining open is whether resctrl should display the (read-only) underlying control
>>> values in schemata file when in "legacy" mode. By default this does not work for region-aware since
>>> it will at least initially use different hardware interfaces for the different controls, essentially
>>> this means that there is no actual emulation and the legacy and region aware control are both
>>> legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".
>>>
>>> The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a
>>> control to start with underlying controls disabled so that they are not displayed in schemata file
>>> when legacy mode is enabled.
>>>
>>> If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer
>>> be needed for region-aware. RISC-V also considered using this "status" that I think may also be
>>> solved with this approach, I am not sure though.
>>>
>>>> I was thinking that in legacy mode that the region values would always be kept the same but perhaps
>>>> the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the
>>>> full story. 'best estimate' could be difficult to choose but allows switching mode to be
>>>> non-destructive.
>>>>
>>>> For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting
>>>> that MPAM would only use emulation for exposing finer grade controls to the user and not allowing
>>>> different underlying hardware to be used for the emulation. To me, it seems reasonable that a new
>>>> control would involve a new schema but I do appreciate the benefit of being able to continue to use
>>>> the old interface without stopping a new interface being used. The danger is that there are
>>>> unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different
>>>> from the user expectations.
>>>
>>> I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level
>>> problem to be that user space may have expectation that for every "resource" represented by a directory in
>>> info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user
>>> space tools is not obvious to me though: there is already an exception since there is the
>>> "L3_MON" "resource" that does not have a schemata file entry.
>>>
>>> To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via
>>> an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl
>>> is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB
>>> then resctrl should not be forced to always provide such emulation.
>>>
>>> I would appreciate your thoughts here.
>>
>> Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for
>> resource emulation.
>>
>> As emulation adds complication and has the potential to break other user expectations I hope we can
>> manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a
>> resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains
>> backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the
>> NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.
>>
>> Can we get around this by by choosing a different organization of the info/ directory? We already
>> have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION)
>> and move the scope to be a property at info/<resource>/scope rather than
>> info/<resource>/resource_schemata/ctrl/scope.
> 
> Good point that L2 and L3, which is the same resource at different scope, are treated as different
> resources. With that view it seems reasonable to also consider MB and MB_NODE as different resources.
> The point here is that it is at a different scope so more specifically what a "domain ID" in the
> schemata file represents. With this view, MB_REGION is L3 scope that matches to the MB resource.

Ok. This makes sense to me. I had confused myself about what MB_REGION is.

> 
> You will notice I get a bit lost in the discussion below so getting back here I would like to
> clarify if you propose that info/ contains a directory for each allocation scope of each resource with that
> directory containing descriptions of all the controls for that resource at the indicated scope or do
> you propose that info/ contains a directory for each control?

The former, info/ contains a directory for each allocation scope of each resource.



info
├── L2
│   ├── resource_schemata
│   │   ├── L2
│   │   ├── L2_CMAX
│   │   └── L2_CMIN
│   └── scope : L2
├── L3
│   ├── resource_schemata
│   │   ├── L3
│   │   ├── L3_CMAX
│   │   └── L3_CMIN
│   └── scope : L3
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   │   └── MB_MAX
│   │   ├── MB_MIN
│   │   ├── MB_PBM
│   │   └── MB_PROP
│   └── scope : L3
└── MB_NODE
    ├── resource_schemata
    │   ├── MB_NODE_MAX
    │   ├── MB_NODE_MIN
    │   ├── MB_NODE_PBM
    │   └── MB_NODE_PROP
    └── scope : NUMA NODE

> 
>>
>> Can MB_REGION be considered an orthogonal new control or does using it require that the traditional
>> intel MB (delay) not be configured?
> 
> The Intel systems that support both MSR ("traditional") and ACPI (region aware MBA) cannot use both
> concurrently. Not sure if this answers your question.

Thanks, this answers my question.

> 
> Just to clarify, there is no single "MB_REGION" control. When considering the "region aware MBA"
> feature I am currently aware of the following possible controls:
> MB_REGION0_MIN
> MB_REGION0_MAX
> MB_REGION0_OPT
> MB_REGION1_MIN
> MB_REGION1_MAX
> MB_REGION1_OPT
> MB_REGION2_MIN
> MB_REGION2_MAX
> MB_REGION2_OPT
> MB_REGION3_MIN
> MB_REGION3_MAX
> MB_REGION3_OPT
> 
> 
>> On an MPAM system:
>>
>> info
>> ├── MB
>> │   ├── resource_schemata
>> │   │   ├── MB
>> │   │   └── MB_MAX
>> │   └── scope
>> └── MB_NODE
>>     ├── resource_schemata
>>     │   └── MB_NODE
>>     └── scope
> 
> Above MB and MB_MAX are represented on the same level. My understanding is that MB_MAX is
> used by driver as the underlying hardware control for the percentage based MB exposed to
> user space. Thus, when user space changes the "MB" control via schemata file it is expected
> to also impact the underlying "MB_MAX" control. By representing them as above this relationship
> is lost. So far I understood that the "MB_MAX" control will be shown as a child of the "MB" control
> to show this relationship. Looks like you plan to change this but from what I understand this
> emulation is still relevant?

Please just consider this a mistake. MB_MAX should be a child of MB.

info/
├── MB
│   ├── resource_schemata
│   │   └── MB
│   │       └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE_MAX (Changed from MB_NODE)
    └── scope

> 
> When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
> from above that it would either start out by itself having the properties of the underlying
> "MAX" control or is the plan to have it be a percentage based control backed by the
> underlying "MAX" hardware control?

The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
but at a different location in the SoC, at the memory controller rather than in the L3.

To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
be MB_NODE_MAX.

My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
pattern for control names and the pattern for resource names being <resource_type>_<scope>.
(allowing for <scope> or <control_type> being missing to match existing naming.)

I don't think we should introduce more percent based controls and for new controls we can introduce
a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/
as discussed previously. Although, I have been pondering on whether we can do a bit better.

We could use hexadecimal point based format for controls which are a proportion of a resource and
have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
independent of the granularity of the control (number of parts).

0 is represented as 0x0
1 as                0x1
1/2 as              0x0.8
7/256 as            0x0.07
1/2**28             0x0.00000001
etc

This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
hope this would be useful for other architectures too. I am preparing some RFC patches on top of
your PoC for consideration of this idea and to explore some of the proposals discussed relating to
generic schemata and how they land in practice from the MPAM side.

> 
> I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
> Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
> info/MB_NODE/resource_schemata/<control>?

Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
complications that it is a property of the MBW_MAX control and that it may be configurable for each
PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
except the meaning of the bit is reversed. We can just use a consistent name in s/w though.

> 
>>
>> On an x86 system:
>>
>> info
>> ├── MB
>> │   ├── resource_schemata
>> │   │   ├── MB
>> │   │   └── MB_MAX (Finer grained MB, more below)
>> │   └── scope
>> └── MB_REGION
>>     ├── resource_schemata
>>     │   └── MB_REGION
>>     └── scope
>>
>>
>> Do you think this helps?
> 
> "REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
> Combine that with up to (currently) four regions each with three controls I find an interface like above
> potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
> a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
> scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
> 
> Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
> way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
> that is not presented in above hierarchy.

OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
previously seem s better fit than what I suggested.

> 
> I think I am missing quite a bit here as I try to navigate an interface so different from what we have
> discussed so far.
> I would like to explore with more detail how this interface can handle the different scenarios we have
> discussed so far.

Certainly, I don't think we have got to the bottom of this yet.

> 
> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
> except for one configuration where it is "SYSTEM"(?) scope.
> Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/

Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
the CPUs and so L3 instances in the group are online. Am I correct here?

Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
then a task running on a CPU affine to that L3 group won't be throttled unless that particular
domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
domain.
> 
>>
>> This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
>> but it exposed a different interface to the user. If I understand correctly intel have an option to
>> have finer grained control of MB (delay) as well and so it would make sense to use a common name
>> rather than just going for the MPAM centric name of MB_MAX.
> 
> Apologies but I was not able to parse above.

Ok, let me try to explain again (although it's probably not what we want to do). My intent here was
to try and explore whether we can reuse naming and controls across architectures in the same way we
already have for the L2/L3 cache portion bitmap and the existing MB control.

To quote from a previous mail of yours:
https://lore.kernel.org/lkml/a84af037-6439-4362-be07-d45143e06309@xxxxxxxxx/
"""
  For example, on an MPAM system (if I understand correctly) the user may see:
  info/
  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_MAX/
          └── MB_MIN/

  Compared with a possible implementation on Intel that looks like:
  info/
  └── MB/
      └── resource_schemata/
          ├── MB/
          │   └── MB_OPT/
          ├── MB_MAX/
          └── MB_MIN/
"""

In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the
legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't
make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.

Thanks,

Ben

> 
> Reinette
>