Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre]

Hi Everybody,

On 8/6/26 2:24 AM, Ben Horgan wrote:
> On 05/08/2026 07:06, Reinette Chatre wrote:
>> On 8/4/26 8:11 AM, Ben Horgan wrote:
>>> On 8/4/26 15:09, Babu Moger wrote:
>>>> On 8/4/26 04:11, Ben Horgan wrote:
>>>>> On 7/22/26 18:02, Babu Moger wrote:
>>>>>> On 7/22/26 05:47, Ben Horgan wrote:
>>>>>>> On 7/21/26 21:02, Babu Moger wrote:
>>>>>>>> On 7/21/26 12:30, Reinette Chatre wrote:
>>>>>>>>> On 7/21/26 6:23 AM, Ben Horgan wrote:
>>>>>>>>>> On 7/20/26 23:54, Reinette Chatre wrote:
>>>>>>>>>>> On 7/20/26 6:30 AM, Ben Horgan wrote:
>>>>>>>>> ...>
>>>>>>>>>> The former, info/ contains a directory for each allocation scope of each resource.
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory
>>>>>>>>>>> bandwidth allocation
>>>>>>>>>>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains
>>>>>>>>>>> controlled at L3 scope),
>>>>>>>>>>> except for one configuration where it is "SYSTEM"(?) scope.
>>>>>>>>>>> Ref.: https://lore.kernel.org/
>>>>>>>>>>> lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/
>>>>>>>>>>
>>>>>>>>>> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems
>>>>>>>>>> to be
>>>>>>>>>> accidental that it maps to the NUMA node but really the scope is just a grouping of L3
>>>>>>>>>> instances.
>>>>>>>>>> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync
>>>>>>>>>> with
>>>>>>>>>> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether
>>>>>>>>>> any of
>>>>>>>>>> the CPUs and so L3 instances in the group are online. Am I correct here?
>>>>>>>>>>
>>>>>>>>>> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
>>>>>>>>>> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3
>>>>>>>>>> group
>>>>>>>>>> then a task running on a CPU affine to that L3 group won't be throttled unless that particular
>>>>>>>>>> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to
>>>>>>>>>> that
>>>>>>>>>> domain.
>>>>>>>>>
>>>>>>>>> I'll defer to Babu for accurate answers about this hardware capability.
>>>>>>>>>
>>>>>>>>
>>>>>>>> To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may
>>>>>>>> appear
>>>>>>>> as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire
>>>>>>>> system. In such cases, there is only one schemata entry controlling the whole system.
>>>>>>>>
>>>>>>>> Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still
>>>>>>>> performed at the L3 level, but that implementation detail can be hidden from users and does not
>>>>>>>> need
>>>>>>>> to be exposed through the interface.
>>>>>>>
>>>>>>> We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl
>>>>>>> user interface but the behaviour required for an MPAM system and an AMD system appears different
>>>>>>> from the point of view of lifecycle of the resctrl domain.
>>>>>>>
>>>>>>> For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so
>>>>>>> goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it
>>>>>>> looks
>>>>>>> to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA
>>>>>>> node. To me it does seem odd that a control with a domain associated with an offline NUMA node can
>>>>>>> continue to throttle (cross-NUMA) traffic.
>>>>>>>
>>>>>>> Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is
>>>>>>> this just implicitly the NUMA id?
>>>>>>>
>>>>>> Yes, the GLBE Control Domain ID is exposed to the user. It is essentially equivalent to the NUMA ID.
>>>>>
>>>>> What's the on/off lifecycle of these nodes? Does it follow the NUMA lifecycle as managed by the NUMA
>>>>> node notifiers documented in Documentation/core-api/memory-hotplug.rst or is it just linked to the
>>>>> cpu hotplug as is done currently for the resctrl cache based domains.
>>>>>
>>>> It will follow the CPU hotplug lifecycle, similar to how it is currently handled for the resctrl
>>>> cache-based domains.
>>>
>>> Ok. MPAM MSC are associated with the memory controllers and so MPAM controls that have NUMA scope
>>> will follow the NUMA memory hotplug lifecycle. This points to them being different resources. Do you
>>> have any thoughts on how we would handle this difference?
>> If I understand correctly, on an AMD GLBE system, even though it is "NUMA node scoped" it does not
>> support memory bandwidth allocation for a NUMA node that is online but all its CPUs are offline. This is
>> because GLBE is essentially L3 MBA that allows to set limits across multiple L3 domains. I think this
>> is a good match for a control associated with the legacy MB resource that just has a different scope
>> of allocation but how to do so without creating confusion with a "real" NUMA bandwidth allocation is
>> not clear to me since it may end up looking like:
>>
>> GLBE (make clear allocation is at L3 scope but domains are node scoped):
>> info/
>> └── MB/
>>     └── schemata/
>>         ├── MB/      /* scope of domain ID = L3   */
>>         └── MB_NODE/ /* scope of domain ID = node */
>>
>> MPAM MSC (make clear allocation is at node scope:
>> info/
>> ├── MB/
>> │   └── schemata/
>> │       └── MB/      /* scope of domain ID = L3   */
>> └── MB_NODE/
>>     └── schemata/
>>         └── MB_NODE/ /* scope of domain ID = node  */
> 
> These info/ trees do express the difference between the two, GLBE and MPAM MSC at the memory. These
> both result in the same schemata file though.
> MB:<cache_id>=...
> MB_NODE:<node_id>...
> 
> This gives a potential conflict if GLBE is paired with memory allocation controls at the memory. I
> don't know if this is likely though.
> 
>>
>> Any suggestions?
>>
>> I do not see how these can be considered different resources though. To me it looks like different
>> controls operating at different scope for the same ("memory bandwidth") resource?
> 
> This is likely just a terminology issue. My thoughts are that, in a similar way to the L2 and L3
> both being cache resources but a different level of the hierarchy, you can think of memory bandwidth
> at egress from L3 as a different resource from memory bandwidth at ingress to the memory
> controllers. Sure, they are interdependent but on an MPAM system you could have MSC at both
> locations. On a system with one L3 and one NUMA node and no other caches in between they are the
> same resource as they are just opposite ends of the same link but that's not the case when you can
> have cross NUMA traffic.
> 
> So in your diagram above info/MB and info/MB_NODE are different "resources" and info/MB/MB_NODE is
> node based scoping of the info/MB "resource". Do you have a good way of referring to these resource
> concepts so that we can be more precise in our language?
> 
This thread continues at https://lore.kernel.org/lkml/81f465d7-d12f-4414-9824-85d3cba5a36b@xxxxxxx/

Reinette