Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre]

Hi Fenghua,

On 7/10/26 1:59 PM, Fenghua Yu wrote:
> On 6/25/26 08:43, Reinette Chatre wrote:
>> On 6/24/26 6:26 PM, Fenghua Yu wrote:
>>> On 6/24/26 15:22, Reinette Chatre wrote:
>>>> On 6/24/26 12:08 PM, Fenghua Yu wrote:
>>>>> On 5/29/26 11:06, Reinette Chatre wrote:
>>>>>
>>>>> As Shaopen and Ben mentioned earlier, we are working on two MPAM
>>>>> features that may need to change schemata interface. The CPU-less
>>>>> feature was discussed on LPC (although the interfaces will be
>>>>> slightly different from the LPC).
>>>>
>>>> I know. Here is where I tried to engage with you on needed interfaces after LPC:
>>>> https://lore.kernel.org/lkml/fb1e2686-237b-4536-acd6-15159abafcba@xxxxxxxxx/
>>>
>>> MPAM ACPI defines MSC (Memory System Control) is defined in one of two ways (not both) on one platform:
>>> 1. L3 and memory together on each processor MSC
>>> 2. L3 in processor MSC and memory control/monitoring in different memory MSCs.
> 
> Ben said there is type 3 platform:
> 3. L3 cache and memory bandwidth in processor MSCs and memory bandwidth in different memory MSCs.
> 
>>>
>>> On type 1 platform, schemata is legacy:
>>> MB:1=100;2=100  <-- cache id 1 and 2 as domain id
>>>
>>> On type 2 platform, I will not reuse "MB:" name. Instead, define new resource name "MBN:" for numa node and schemata is:
>>> MBN:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10, 18,
>>>                             26 as domain id
>>> On type 2 platform, there won't be "MB:" line. Numa 0 and 1
>>> are for mbm allocation on socket 0 and 1. 2,10, 18 and 26 are for GPU
>>> memory nodes allocation.
> 
> On type 3 platform, there could be "MB:" line for L3 cache and "MB_NODE:" for numa node. Example schemata is:
> 
>      MB:1=100;2=100                 <-- cache id 1 and 2 as domain id
> MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100 <-- numa id 0, 1, 2, 10,
>                                                    18, 26 as domain id
>>
>> (to help make things explicit I will refer to what you call "MBN" as "MB_NODE" to make it
>> explicit that it is memory bandwidth allocation at node scope)
>>
>> I am trying to consider how this can be accomplished while also considering all the other
>> new hardware features that resctrl need to support. Consider, for example, AMD's "Global
>> MBA" (https://lore.kernel.org/lkml/cover.1776980182.git.babu.moger@xxxxxxx/) that throttles
>> memory bandwidth at L3 scope but the user configures allocations at NODE scope. At this time
>> the plan is to support this with a second control associated with the MB resource that can
>> allocate memory bandwidth at node scope. See
>> https://lore.kernel.org/lkml/430ffb48-29f4-44d9-9164-9f8b743b2739@xxxxxxx/
>>
>> If resctrl creates a new resource for node scoped memory bandwidth allocations to support these
>> "type 2" systems then that will result in an inconsistent interface between architectures that
>> we should avoid.
>>
>> Have you been listening in on the discussions surrounding emulated controls? Considering that,
>> would it be possible to support the "MB" control on a type "2" system but have it be backed by
>> (emulated by) the underlying "MB_NODE" control?
>>
>> resctrl could expose both controls on these "type 2" systems but make it clear that "MB"
>> is emulated by "MB_NODE". For example:
>>
>> info/
>> └── MB/
>>      └── resource_schemata/
>>          └── MB/
>>              └── MB_NODE/
>>
>> User will see both controls in schemata file but when changes are made to "MB" control it
>> will show in the "MB_NODE" control and vice-versa. User could also disable the "MB" control
>> that will establish familiarity with the interface at which point resctrl can drop the
>> "MB" control from the schemata file on these "type 2" systems.
>>
>> Having the MB resource available with an MB control will keep resctrl backward compatible
>> if there are any tools that expect that. If backward compatibility is not of concern then
>> resctrl could initialize with the emulated control disabled by default. See discussion at
>> https://lore.kernel.org/lkml/5e575bc2-e67f-4696-9332-33c54023c057@xxxxxxxxx/
>> that describes a new resctrl capability in support of RISC-V and RDT.
>> With this resctrl could initialize with:
>>
>> info/
>> └── MB/
>>      └── resource_schemata/
>>          ├── MB/
>>          │   ├── MB_NODE/
>>          │   │   └── status:enabled
>>          │   └── status:disabled
>>          └── mode:legacy [native]
>>
>> With above a "type 2" system will boot with its schemata file just containing the "MB_NODE"
>> control while info/MB describes the memory bandwidth resource.
>>
> 
> On type 3 machine, schemata has both MB in legacy mode with cache id as domain id and MB_NODE with numa id as domain id.
> 
> Is this directory OK?
> 
>  info/
>  └── MB/
>       └── resource_schemata/
>           ├── MB/
>           │   ├── MB_NODE/
>           │   │   └── status:disabled
>           │   └── status:enabled
>           ├── MB_NODE/
>           └── mode:node
> 
> 1. MB and MB_NODE are shown in parallel in inf/MB/resource_schemata/

I do not think there is a need to expose an emulated MB_NODE control if the actual MB_NODE
hardware control exists. 

> 2. mode is set as "node" meaning "MB" is for L3 and "MB_NODE" is for numa node

I assume you mean "scope" instead of "mode"? (more below)

> 3. Emulation "MB_NODE" is disabled (or should the "MB_NODE" sub-dir be invisible?)

Right, I do not think emulation is needed here. No need to make it invisible since it should not exist.

>From what I understand these "type 3" machines could be simplified to:

info/
└── MB/
    └── resource_schemata/
        ├── MB/
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

Beyond this I believe that MPAM currently emulates the MB control with its "MB_MAX" control and users may want
to make bandwidth allocations at the fine granularity that it supports. Taking this into account the interface
may end up looking like:

info/
└── MB/
    └── resource_schemata/
        ├── MB/
        │   ├── MB_MAX/
        │   │   └── scope:L3
        │   └── scope:L3
        └── MB_NODE/
            └── scope:NODE

A system like above will thus have three schemata file entries:
MB
MB_MAX
MB_NODE

Three schemata file entries would be unnecessary for users familiar with the finer granularity MB_MAX control
so that is where the "mode" file can be used to disable the legacy MB control to just expose MB_MAX and MB_NODE
on these systems.

Would that work for these systems?

... 

>>>>> There is another MPAM feature called MBW Max hardlimit which sets
>>>>> "MB:" allocation as hardlimit (i.e. MBW throttling percentage must
>>>>> be satisfied) per domain. Adding a new "MB_HLIM:" line in schemata.
>>>>> It's 1:1 mapped to "MB:" to control hardlimit of MB throttling
>>>>> percentage on each domain. By default hardlimit is off (0) and can
>>>>> be turned on to set MBW Max hardlimit on a domain.
>>>>
>>>> ack. This sounds like a new control associated with the MB resource.
>>>> This is a boolean control as Dave highlighted in previous discussion so
>>>> resctrl would need to know its properties.
>>>> See https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@xxxxxxxxxxxxxxx/
>>>>
>>>
>>> Right. ("MB_HLIM" name may be adjusted accordingly when "MB_MAX" is available.)
>>>
>>>>> For exmple:
>>>>> MB_HLIM: 0=0;1=0;2=1;10=0;18=0;26=0
>>>>> MB:0=100;1=100;2=80;10=100;18=100;26=100
>>>>>
>>>>> On GPU memory numa node 2: cannot use more than 80% of total max mbw even if there is still idle mem bandwidth on this node).
>>>>>
>>>>> MBW allocations on all other domains are soft limited, meaning MBW can be used more than specified if mem is idle.
>>>>>
>>>>
>>>> ack.
>>>>
>>>>>>            L3:0=fff;1=fff
>>>>>> # echo 'MB_MIN:0=50' > schemata
>>>>>> # cat schemata
>>>>>>            MB_MAX:0=100;1=100
>>>>>>            MB_MIN:0=50;1=100
>>>>>>            MB:0=100;1=100
>>>>>>            L3:0=fff;1=fff
>>>>>>
>>>>>> Writing to the dummy control will call a dummy callback that just prints to the
>>>>>> kernel log:
>>>>>> "resctrl: Updata temporary MIN control on domain 0 with user value 50"
>>>>>>
>>>>>>
>>>>>> Example output of info/MB/:
>>>>>> /sys/fs/resctrl/info/MB/thread_throttle_mode:max
>>>>>> /sys/fs/resctrl/info/MB/num_closids:15
>>>>>> /sys/fs/resctrl/info/MB/delay_linear:1
>>>>>> /sys/fs/resctrl/info/MB/min_bandwidth:10
>>>>>
>>>>> Add two new MB info RO files:
>>>>> 1. /sys/fs/resctrl/info/MB/domain_id
>>>>> It shows "numa" for using numa id in "MB:" or "cache" for using legacy cache id.
>>>>
>>>> This proposal introduces a *global* property to the MB *resource*? It does not seem as though
>>>> this takes into account *anything* about how resctrl can support new hardware that has been
>>>> discussed before, during, or after LPC. You have not participated in these discussions and
>>>> now make an orthogonal proposal that does not take into account *any* of the requirements
>>>> that we have been struggling with for months.
>>>>
>>>> Why should this proposal be taken seriously? In your absence folks have been trying to
>>>> accommodate how these upcoming products and be supported and the "scope" file associated with
>>>> a control is intended to communicate to user space how the domain ID should be interpreted.
>>>>
>>>> Why are you proposing something entirely different here without even acknowledging current
>>>> approach and explaining why it does not work for you?
>>>>
>>>
>>> So can I change this part to adding the following files in info dirctory?
>>>
>>> 1. For numa memory bw allocation (MBN):
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/resolution:100
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/tolerance:5
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/type:scalar
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/min:10
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scale:1
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/scope:NUMA
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/unit:all
>>> /sys/fs/resctrl/info/MBN/resource_schemata/MBN/max:100
>>
>> This is not just about adding files to the info directory. The files, directories, their relationships,
>> and content have meaning. All I see from these proposals is an attempt to slap some new files into
>> resctrl without any consideration to present consistent interface to users and without consideration of
>> other architectures that need to be supported by resctrl.
>>
>> resctrl needs to provide a generic and consistent interface to user space irrespective of the
>> underlying architecture. Architectures cannot just slap some new files for their convenience.
>>
>>>
>>>>> 2. /sys/fs/resctrl/info/MB/max_lim
>>>>> It shows number 0-3 for MPAM MBW max limit behaviors: 0 for supporting both softlimit and hardlimit, etc.
>>>>
>>>> Again this adds another *global* property to the MB resource but then above you
>>>> describe the new "MB_HLIM" schemata file entry that implies that it is a new control
>>>> for the MB resource. Having it be a new control for the MB resource matches earlier
>>>> discussions. To support this I thus expect it to be exposed as a new control with
>>>> potentially a new type if any of the existing planned types do not suffice.
>>>>
>>>
>>> How about adding these MB_HLIM dir and files in info?
>>>
>>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/type: boolean
>>> /sys/fs/resctrl/info/MB_HLIM/resource_schemata/MB_HLIM/max_lim: 0
>>
>> This presents "MB_HLIM" as a *resource* to user space. It is not a resource
>> but a *control* of a resource, no? I thus expect it to instead look something like
>> below that makes it clear that MB_HARDMAX is a control of the MB resource.
>>
>> info
>> └── MB
>>      └── resource_schemata
>>          ├── MB
>>          └── MB_HARDMAX
> 
> Yes, this makes sense. I have changed to this hierarchy.

Thank you very much for considering this approach.

Reinette