Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept
From: Ben Horgan
Date: Tue Aug 25 2026 - 11:20:43 EST
Hi Reinette,
On 24/08/2026 19:07, Reinette Chatre wrote:
> Hi Ben,
>
> On 8/24/26 7:40 AM, Ben Horgan wrote:
>> On 07/08/2026 23:53, Reinette Chatre wrote:
>>> On 8/7/26 11:33 AM, Moger, Babu wrote:
>>>> On 8/7/2026 10:35 AM, Reinette Chatre wrote:
>>>>> On 8/7/26 7:14 AM, Moger, Babu wrote:
>>>>>> On 8/6/2026 12:11 PM, Reinette Chatre wrote:
>>>>>>> On 8/6/26 10:04 AM, Luck, Tony wrote:
>>>>>>>> On Wed, Aug 05, 2026 at 04:57:04PM -0700, Reinette Chatre wrote:
>>>>>>>>> On 8/5/26 9:59 AM, Ben Horgan wrote:
>>>>>>>>>> Just given a go at running this on a model with MPAM and I can mount resctrl with this
>>>>>>>>>> small patch to initialise the emulated_by lists so that list_empty() behaves.
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
>>>>>>>>>> index 21dcbf5764cc..638095649151 100644
>>>>>>>>>> --- a/drivers/resctrl/mpam_resctrl.c
>>>>>>>>>> +++ b/drivers/resctrl/mpam_resctrl.c
>>>>>>>>>> @@ -1018,6 +1018,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res)
>>>>>>>>>> case RDT_RESOURCE_L3:
>>>>>>>>>> mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_BITMAP;
>>>>>>>>>> mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF;
>>>>>>>>>> + INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);
>>>>>>>>>> INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains);
>>>>>>>>>> __set_bit(RESCTRL_BITMAP_FLAG_SPARSE, mpam_ctrl->r_ctrl.bitmap.flags);
>>>>>>>>>> mpam_ctrl->r_ctrl.bitmap.cbm_len = class->props.cpbm_wd;
>>>>>>>>>> @@ -1048,6 +1049,7 @@ static int mpam_resctrl_control_init(struct mpam_resctrl_res *res)
>>>>>>>>>> r->ctrl_scope = RESCTRL_L3_CACHE;
>>>>>>>>>> mpam_ctrl->r_ctrl.type = RESCTRL_CTRL_SCALAR;
>>>>>>>>>> mpam_ctrl->r_ctrl.name = RESCTRL_CTRL_NAME_DEF;
>>>>>>>>>> + INIT_LIST_HEAD(&mpam_ctrl->r_ctrl.emulated_by);
>>>>>>>>>> INIT_LIST_HEAD_RCU(&mpam_ctrl->r_ctrl.domains);
>>>>>>>>>>
>>>>>>>>>> r->bw_throttle_mode = THREAD_THROTTLE_UNDEFINED;
>>>>>>>>>
>>>>>>>>> Thank you for this. Added this and it is now available in branch resctrl/controls_rfc_v2.1
>>>>>>>>
>>>>>>>> Is same needed for x86? I don't see any initialization of the
>>>>>>>> "r_ctrl.emulated_by" lists in similar initialization functions.
>>>>>>> x86 "emulated_by" list initialization should be in both branches.
>>>>>>>
>>>>>>> Branch resctrl/controls_rfc_v2.1 combined all "emulated_by" list initialization
>>>>>>> (for x86 and MPAM) into commit:
>>>>>>> 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode")
>>>>>>>
>>>>>>> In the original resctrl/controls_rfc_v2 the x86 "emulated_by" list initialization can
>>>>>>> be found in commit fbed64f80515 ("x86/resctrl: SAMPLE: Emulated controls")
>>>>>>>
>>>>>>
>>>>>> Looking at the commit:
>>>>>>
>>>>>> commit 3407e523c988 ("fs/resctrl: Introduce emulated controls and control mode")
>>>>>>
>>>>>> Based on the patch description, emulated controls are intended to be
>>>>>> used only when there is a difference between the native and legacy
>>>>>> controls. If no such difference exists, both modes should operate
>>>>>> identically.
>>>>>
>>>>> The original AMD MBA enabling did not follow the original percentage based MBA
>>>>> control so resctrl essentially has *two* "legacy" MB controls today: one for AMD
>>>>> and one for Intel, MPAM, and RISC-V (planned afaik). That cannot be changed now.
>>>>> AMD would continue to expose the MB control that is not the percentage based "legacy"
>>>>> control but actually AMD's native control. In short, yes, on AMD's "MB" control
>>>>> the "legacy" and "native" control modes should operate identically.
>>>>> At least now users could use the files in info/MB/schemata/MB/* to learn the
>>>>> properties of the control.
>>>>>> For MBA(AMD) and GMBA, there does not appear to be any difference
>>>>>> between the two modes. Is that understanding correct?
>>>>> It is not clear to me how resctrl should support AMD's GLBE. Note I am intentionally
>>>>> not using GMBA since that already makes an assumption on how resctrl will support this.
>>>>> Do you perhaps have an answer for Ben's question in
>>>>> https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@xxxxxxx/ ?
>>>>
>>>> Yes. We are already at RFC v2. Let me respond here. Please see my response below.
>>>>
>>>>>
>>>>> GLBE claims to enable users to allocate memory bandwidth at node scope but the memory
>>>>> is managed at L3. This results in scenarios where, for example, a NUMA node can be
>>>>> online and used but resctrl cannot expose it for bandwidth allocation when all the CPUs
>>>>> at that NUMA node scope are offline. GLBE is thus not actually allocating memory bandwidth
>>>>> at the NUMA node.
>>>>
>>>> Yea. That is correct.
>>>>
>>>> There are a couple of key differences when compared to the pure NUMA scope.
>>>>
>>>> 1. In some cases, a NUMA node is treated as the entire system.
>>>>
>>>>
>>>> https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/
>>>>
>>>> 2. When a user updates the settings on a GLBE for a specific node, we need to update the MSRs within that node at the L3 scope:
>>>>
>>>> https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@xxxxxxx/
>>>>
>>>>
>>>> Considering these differences, I think we should probably treat GLBE
>>>> separate from MB_NODE scope. What do you think?
>>>
>>> I think so too, but to Ben's point we need to be clear on terminology here. There are two usages
>>> of "scope" to consider:
>>> 1) The scope of the *resource* being allocated.
>>> Here "scope" applies to the resource. For cache there is L2 and L3 that indicates the
>>> scope of the cache resource. There is also now two different scope to consider for
>>> memory bandwidth allocation, "L3 scope" for memory bandwidth at egress from L3 and "NUMA/node"
>>> scope for memory bandwidth at ingress to NUMA node.
>>>
>>> 2) The scope of the *control* used to allocate the resource.
>>> So far the scope of the resource has been assumed that to be the same as the scope of the control.
>>> That is, a resource at particular scope is allocated at that same scope. GLBE taught us this is not
>>> always the case - memory bandwidth at L3 scope can be allocated at NUMA(with some caveat as you point out)
>>> scope.
>>>
>>> We have gone back and forth on this. Currently upstream resctrl has control scope as a property of
>>> the resource with expectation that the resource and its controls have the same scope. As a change
>>> to this RFC v1 of this PoC had "scope" as a property of the control to support a control to have
>>> different scope as resource. After deciding to treat MB_NODE as a new resource to support the CPU-less
>>> nodes, RFC v2 changed this back to have scope a property of the resource.
>>>
>>> First, it looks to me as though the RFC v1 separating control scope from resource scope needs to
>>> return.
>>>
>>> Second, how to name these resources/controls needs to be decided. This is what prompted my question
>>> about future considerations to you because my original proposal in
>>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@xxxxxxxxx/
>>> was able to convey the resource and allocation scope via info hierarchy but it used the same
>>> name in the schemata file that Ben highlighted could be problematic.
>>>
>>> I believe there is agreement that the resource should include the resource scope in its name. This
>>> is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have
>>> "L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE"
>>> in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource*
>>> scope.
>>>
>>> Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does
>>> not allocate memory bandwidth *resource* at node scope.
>>>
>>> We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@xxxxxxxxx/ )
>>> of having the control's scope part of the control's name.
>>>
>>> We could have a rule of thumb to include both resource scope and control scope in the
>>> control name but only one instance is displayed if they are the same.
>>>
>>> For example, what do you think of something like below as an alternate proposal of what I mentioned in
>>> https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@xxxxxxxxx/:
>>>
>>> GLBE (booted with NPS < 4):
>>> info/
>>> └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
>>> └── schemata/
>>> ├── MB/ /* control scope = L3 */
>>> └── MB_L3NODE/ /* control scope = node ("L3" resource scope + "node" control scope) */
>>>
>>> GLBE (booted with NPS = 4):
>>> info/
>>> └── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
>>> └── schemata/
>>> ├── MB/ /* control scope = L3 */
>>> └── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/
>>
>> I think this separation of control scope and resource scope works. My understanding is that, as the
>> scope of the resource is L3, we can infer that the lifecycle of the resctrl domain is linked to the
>> L3 cache and so the cpu hotplug lifecycle. The new part is that the size of the domain is determined
>> by the control scope, a single L3 (L3 control scope as per current resctrl) or a collection of L3
>> instances as identified by a NUMA node id (NODE) or the whole system (SYSTEM).
>
> Thank you very much. I agree. An additional way I use to simplify this is that the new "control scope"
> is how resctrl communicates to user space what the domain ID used in, for example, schemata file
> represents.
>
> One caveat is that the above layout is not something that the filesystem can enforce so for this
> interface to remain consistent will depend on the support from all architectures.
>
>>>
>>> MPAM MSC:
>>> info/
>>> ├── MB/ /* memory bandwidth allocation at L3 scope */
>>> │ └── schemata/
>>> │ └── MB/ /* control scope = L3 */
>>> └── MB_NODE/ /* memory bandwidth allocation at node scope */
>>> └── schemata/
>>> └── MB_NODE/ /* control scope = node */
>>
>> Yes to the general structure. You may have just omitted this for clarity but I don't think we should
>> introduce a control just called MB_NODE, it should be specify the type too, so one of MB_NODE_MIN,
>> MB_NODE_MAX, MB_NODE_PROP etc.
>
> I agree. One uncertainty here is how the MB_NODE resource's control mode should be presented to
> user space. I think it may be most intuitive to present "native" as the one and only control mode
> option?
Yes, "native" being the only option here makes sense to me.
Thanks,
Ben
> In the current implementation the "native" and "legacy" control mode of a resource without
> any emulated controls will result in the same controls enabled.
>
> Reinette
>
>