Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Ben Horgan]

Hi Reinette,

On 7/17/26 17:00, Reinette Chatre wrote:
> Hi Ben,
> 
> On 7/17/26 5:20 AM, Ben Horgan wrote:
>> On 7/16/26 18:07, Reinette Chatre wrote:
>>> On 7/16/26 9:44 AM, Ben Horgan wrote:
>>>> On 7/16/26 17:04, Reinette Chatre wrote:
>>>>> Hi Ben, Chenyu, and Tony,
>>>>>
>>>>> On 7/16/26 7:59 AM, Ben Horgan wrote:
>>>>>> On 7/15/26 16:41, Reinette Chatre wrote:
>>>>>>> On 7/15/26 1:34 AM, Ben Horgan wrote:
>>>>>>>> On 7/14/26 23:06, Reinette Chatre wrote:
> 
> ...
> 
>>>>>>>> Alternatively, could the "mode" file be used to switch between "MB" and "MB_MAX" and the two never
>>>>>>>> need be shown at the same time. The user opts in to using the new interface, "MB_MAX" by setting
>>>>>>>> "mode" and can just toggle back if they want to use "MB" directly again.
>>>>>>> Interesting. So far the "mode" options have been "legacy" and "native" where "legacy" would show the
>>>>>>> legacy as well as emulated controls in the schemata file and "native" will only show the new controls.
>>>>>>> resctrl could make the default of "legacy" mean that *only* the legacy control is shown without insight
>>>>>>> into the controls it is being emulated with. The only insight to this would continue to be via the
>>>>>>> hierarchy in the info directory, which would show relationship but not the actual control values.
>>>>>>> Do you think there could be a need for users (excluding validation?) that may want to see the underlying
>>>>>>> control values used to emulate a legacy control?
>>>>>>
>>>>>> They may want to see the underlying values to be able to migrate there legacy configuration to the
>>>>>> native configuration but they can just write their legacy configuration and then toggle the mode to
>>>>>> see the values in the native mode.
>>>>>>
>>>>>>>
>>>>>>> I assume for backward compatibility that "legacy" would remain the default so a possible inconvenience
>>>>>>> here would be that users familiar with the new controls would forever need to switch the mode before being
>>>>>>> able to use them.
>>>>>>
>>>>>> This does also make the new controls slightly harder to discover.
>>>>>>
>>>>> What if resctrl combines the two suggestions? Specifically, for backward compatibility the mode will be
>>>>> "legacy" and if there are emulated controls then resctrl displays them in schemata file with "#" prefix
>>>>> but not(*) support any changes from user space to the underlying hardware controls. This will make new controls
>>>>> easier to discover and let user space see the underlying values, but not break a user space that
>>>>> may read schemata file, change a few values, and write entire file back.
>>>>
>>>> My initial impression is that this would work but is probably unnecessary.
>>>
>>> ok. My goal was to present ideas to address the issues raised so far. Ideally this would result in discussion of
>>> pros/cons. If you find this unnecessary, could you please expand with some insight into which parts you find
>>> unnecessary or how you would prefer this solved? I am finding it difficult to interpret above response.
>>
>> Sorry, yes. I was a bit lazy in my reply yesterday. If the mode is easy and non-destructive to
>> change then doesn't the user get the same information with just the burden of toggling the mode and
>> then reading the schemata a second time. This does rely on the user being aware of the new behaviour
>> though.
>>
>> If the emulation isn't a, one to one, bijection then is changing mode expected to be destructive?
>> More on this below.
> 
> There may be some corner cases (region aware may have some as you highlight below), but in general I do not expect
> changing mode to be destructive. The idea behind giving insight into underlying control values while legacy interface
> is in use is to address your earlier point that not doing so will make new controls slightly harder to discover.
> (more below)
> 
>>>>> How the "enabled" vs "disabled" state of a control works with this needs some confirmation since region-aware
>>>>> MBA support has this extra caveat of supporting MSR and ACPI interfaces which results in the relationship
>>>>> between "mode" and "status" of underlying controls not being consistent. This may be ok but please consider
>>>>> example below.
>>>>>
>>>>> Thinking through this with examples as I understand MPAM and RDT region aware so far. Could this work for
>>>>> MPAM and region aware MBA?
>>>>>
>>>>> (*) Should resctrl allow user space to change underlying control value of an emulated control in "legacy" mode?
>>>>
>>>> I don't think this causes problems for MPAM but for RDT region aware controls couldn't you end up
>>>> with a control state that isn't reachable just by configuring the legacy schema.
>>>
>>> Could you please highlight the scenario you refer to?
>>
>> My expectation here was that for any value of MB, X, the value for each of the regions would be the
>> same, Y. (Not sure if this is actually the case.)
>>
>> MB: X
>> # MB_REGION0_MAX:Y
>> # MB_REGION1_MAX:Y
>> # MB_REGION2_MAX:Y
>> # MB_REGION3_MAX:Y
>>
>> So, if one of the regions was to be changed individually then it would be in a state not reachable
>> by just changing the legacy control, MB. Potentially this complicated switching mode as well as
>> uncommenting and writing a schemata.
> 
> Indeed. This scenario was highlighted in slide 8 of
> https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
> and discussed between Tony and Dave Martin in the thread starting at
> https://lore.kernel.org/lkml/aPf0OKwDZ4XbmVRB@agluck-desk3/
> 
> Tony and Dave discussed a few scenarios and how resctrl could behave with different user interactions.
> On a high level I understood that the underlying controls will always, as you state below, "show the
> full story" and if user space interacts with them (instead of using the legacy control) then the
> control values associated with the legacy control cannot be relied on to represent accurate state to
> the point that it may even be better to not be visible. 
> You are right, this is a good motivation to not allow user to modify underlying controls when in
> "legacy" mode. 
> 
> The remaining open is whether resctrl should display the (read-only) underlying control
> values in schemata file when in "legacy" mode. By default this does not work for region-aware since
> it will at least initially use different hardware interfaces for the different controls, essentially
> this means that there is no actual emulation and the legacy and region aware control are both
> legitimate hardware controls. Specifically, considering your example, there is no mapping from "X" to "Y".
> 
> The way region-aware is planning to address this is to use the "enabled" vs "disabled" status of a
> control to start with underlying controls disabled so that they are not displayed in schemata file
> when legacy mode is enabled.
> 
> If resctrl instead only displays legacy controls when in "legacy" mode then the "status" may no longer
> be needed for region-aware. RISC-V also considered using this "status" that I think may also be
> solved with this approach, I am not sure though.
> 
>> I was thinking that in legacy mode that the region values would always be kept the same but perhaps
>> the legacy (MB) value could just be the 'best estimate' and the native (MB_REGION0) values show the
>> full story. 'best estimate' could be difficult to choose but allows switching mode to be
>> non-destructive.
>>
>> For MPAM, at least until Fenghua sent his series today on MBA control emulation [1], I was expecting
>> that MPAM would only use emulation for exposing finer grade controls to the user and not allowing
>> different underlying hardware to be used for the emulation. To me, it seems reasonable that a new
>> control would involve a new schema but I do appreciate the benefit of being able to continue to use
>> the old interface without stopping a new interface being used. The danger is that there are
>> unexpected user visible side effects, e.g. domain lifetime, and that control behaviour is different
>> from the user expectations.
> 
> I have not looked at Fenghua's series in detail but from earlier discussion I understood the high level
> problem to be that user space may have expectation that for every "resource" represented by a directory in
> info/ there is a matching entry in the schemata file. Whether this is an actual expectation from user
> space tools is not obvious to me though: there is already an exception since there is the
> "L3_MON" "resource" that does not have a schemata file entry.
> 
> To ensure backward compatibility the safest would be for resctrl to always provide a "MB" control via
> an entry in the schemata file when the "MB" resource is exposed via info/. If on the other hand resctrl
> is not expected to provide a "MB" entry in schemata file when there is a "MB" resource described in info/MB
> then resctrl should not be forced to always provide such emulation.
> 
> I would appreciate your thoughts here.

Ok, thanks for pointing this out, I hadn't previously properly understood the motivation for
resource emulation.

As emulation adds complication and has the potential to break other user expectations I hope we can
manage to keep it to only simple cases. For instance as I mentioned before, the lifetime of a
resctrl domain lifetime is necessary different (when there are multiple NUMA nodes) for domains
backed by MSC at the memory to MSC at the cache. At the memory they would be inaccessible when the
NUMA node is powered off. Also, any partitioning is happening at a different point in the topology.

Can we get around this by by choosing a different organization of the info/ directory? We already
have separate resource for L2 and L3. Perhaps, the same for MB, MB_NODE, (not sure about MB_REGION)
and move the scope to be a property at info/<resource>/scope rather than
info/<resource>/resource_schemata/ctrl/scope.

Can MB_REGION be considered an orthogonal new control or does using it require that the traditional
intel MB (delay) not be configured?

On an MPAM system:

info
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX
│   └── scope
└── MB_NODE
    ├── resource_schemata
    │   └── MB_NODE
    └── scope

On an x86 system:

info
├── MB
│   ├── resource_schemata
│   │   ├── MB
│   │   └── MB_MAX (Finer grained MB, more below)
│   └── scope
└── MB_REGION
    ├── resource_schemata
    │   └── MB_REGION
    └── scope


Do you think this helps?

This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
but it exposed a different interface to the user. If I understand correctly intel have an option to
have finer grained control of MB (delay) as well and so it would make sense to use a common name
rather than just going for the MPAM centric name of MB_MAX.

Thanks,

Ben


> 
>> [1] https://lore.kernel.org/linux-arm-kernel/cover.1784217438.git.fenghuay@xxxxxxxxxx/
>>
>>>>
>>>>>
>>>>> MPAM
>>>>> ====
>>>>> 1) System default
>>>>>
>>>>> info hierarchy
>>>>> --------------
>>>>> info/
>>>>> └── MB/
>>>>>     └── schemata/
>>>>>         ├── MB/
>>>>>         │   ├── MB_MAX/
>>>>>         │   │   └── status:enabled
>>>>>         │   └── status:enabled
>>>>>         └── mode:[legacy] native
>>>>>
>>>>> schemata file
>>>>> -------------
>>>>> MB:...
>>>>> # MB_MAX:...
>>>>>
>>>>> - In above, user space attempting to change MB_MAX via schemata file with or without "#" prefix will have no effect.
>>>>>   TBD: there should not be obstacles to resctrl supporting user space changes by removing the "#" prefix.
>>>>
>>>> Perhaps a different prefix for read only lines as opposed to commented lines.
>>>
>>> Which prefix do you have in mind?
>>
>> I was avoiding that.. but perhaps for read only lines perhaps they could be in brackets as it hints
>> at it being an extra explanation.
>>
>> (MB_MAX:...)
> Sure (although it seems we are moving away from this approach).
> 
> Reinette
>