Re: [RFC v2] arm,x86,fs/resctrl: Generic schema description Proof of Concept
From: Babu Moger
Date: Tue Aug 11 2026 - 10:59:50 EST
Hi Reinette,
On 8/10/26 23:06, Reinette Chatre wrote:
Hi Babu,
On 8/10/26 11:52 AM, Babu Moger wrote:
On 8/7/26 17:53, Reinette Chatre wrote:
...
Second, how to name these resources/controls needs to be decided. This is what prompted my question
about future considerations to you because my original proposal in
https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@xxxxxxxxx/
was able to convey the resource and allocation scope via info hierarchy but it used the same
name in the schemata file that Ben highlighted could be problematic.
I believe there is agreement that the resource should include the resource scope in its name. This
is currently done for L2 and L3, and planned to be done for MB_NODE (the "MB" resource does not have
"L3" in its name, this cannot be changed now, but "MB" resource is implicitly "L3" scope). The "NODE"
in MB_NODE is thus memory bandwidth allocation at NUMA node scope - "NODE" in MB_NODE is the *resource*
scope.
Considering this I do not think that GLBE should use MB_NODE as you also state above. Since it does
not allocate memory bandwidth *resource* at node scope.
We also discussed before (https://lore.kernel.org/lkml/c78169bc-e2d6-4583-96ec-09fa6dd6653a@xxxxxxxxx/ )
of having the control's scope part of the control's name.
We could have a rule of thumb to include both resource scope and control scope in the
control name but only one instance is displayed if they are the same.
For example, what do you think of something like below as an alternate proposal of what I mentioned in
https://lore.kernel.org/lkml/f5b6cec4-03d8-4a11-884d-d4579dab6b22@xxxxxxxxx/:
GLBE (booted with NPS < 4):
info/
└── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
└── schemata/
├── MB/ /* control scope = L3 */
└── MB_L3NODE/ /* control scope = node ("L3" resource scope + "node" control scope) */
GLBE (booted with NPS = 4):
info/
└── MB/ /* memory bandwidth allocation implicitly/legacy at L3 scope, think of this as "MB_L3" */
└── schemata/
├── MB/ /* control scope = L3 */
└── MB_L3SYSTEM/ /* control scope = system ("L3" resource scope + "system" control scope*/
Can the names here MB_L3NODE and MB_L3SYSTEM be simplified to
MB_NODE and MB_SYSTEM? The L3 scope is already implied by the higher-
level hierarchy.
This would be the alternate proposal I mention above but Ben highlighted in
https://lore.kernel.org/lkml/1444b515-752e-4167-87f5-30ace189e05c@xxxxxxx/ that such naming would result in
a conflict if GLBE is ever combined with some actual node based MBA that would result in two controllers named
"MB_NODE" that need to be accommodated by schemata file. This is what prompted my question to
you with the conclusion that we cannot predict the future. The above proposal prepares resctrl for some
hypothetical scenario where a GLBE style system also supports actual node MBA.
Oh, okay. As the hypothetical case doesn't exist at this time, I'll leave the decision to you and Ben. I'm fine either way.
ii
MPAM MSC:
info/
├── MB/ /* memory bandwidth allocation at L3 scope */
│ └── schemata/
│ └── MB/ /* control scope = L3 */
└── MB_NODE/ /* memory bandwidth allocation at node scope */
└── schemata/
└── MB_NODE/ /* control scope = node */
The GLBE controls make it clear that the *same* resource is allocated using two
different controls that have different scope. This creates an implicit dependency
between the two that is not quite captured but having two controls for the same
resource would already give user space some insight that their control values
need to be considered with care.
I'd appreciate your, Ben's, and anybody else's thoughts on this.
Looks good to me.
There is one more point I'd like to bring up regarding GLBE.
Consider the following example with 16 CPUs, 2 NUMA domains, and 2 L3 domains per NUMA node:
NUMA0
├─ L3_0: CPUs 0-3
└─ L3_1: CPUs 4-7
NUMA1
├─ L3_2: CPUs 8-11
└─ L3_3: CPUs 12-15
When updating NUMA0, we need to update one CPU from each L3 domain.
However, this relationship is not currently represented in our
domain data structure. The domain's hdr.cpumask simply contains
CPUs 0-7 and does not provide any information about the underlying
L3-domain boundaries.
I expect that the domain associated with a control that has node scope would include all
CPUs associated with that node in is cpumask. In your example I thus expect there to
only be two domains, domain ID 0 would have CPUs 0 - 7 in its cpumask and domain ID 1
would have CPUs 8 to 15 in its cpumask.
Yes, that's correct. However, when updating the NUMA nodes, we cannot call smp_call_function_any(0-7) and smp_call_function_any(8-15) as we are currently doing.
Instead, we need to perform the operation at the L3 cache domain level, for example:
NUMA0
├─ L3_0: CPUs 0-3 -> smp_call_function_any(0-3)
└─ L3_1: CPUs 4-7 -> smp_call_function_any(4-7)
NUMA1
├─ L3_2: CPUs 8-11 -> smp_call_function_any(8-11)
└─ L3_3: CPUs 12-15 -> smp_call_function_any(12-15)
In other words, the target CPU should be selected from each L3 cache group rather than from the entire NUMA node. The current approach chooses one CPU from the full NUMA domain, which does not guarantee coverage across all L3 cache domains within that NUMA node.
This was the reason we needed additional processing while programming the MSRs:
https://lore.kernel.org/lkml/a2a06bd290e68f902be9e7cc3ad35f0a2211b950.1776980182.git.babu.moger@xxxxxxx/
I'm wondering whether this relationship should be represented
explicitly in the control data structure, or whether that would be
exposing details that are too specific to a particular hardware
implementation.
From what I understand this is already accommodated. What am I missing? Reinette
Please see the response above. Hope it clears the explanation.
Thanks
Babu