Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Ben Horgan]

Hi Babu,

On 7/21/26 21:02, Babu Moger wrote:
> Hi Ben/Reinette,
> 
> On 7/21/26 12:30, Reinette Chatre wrote:
>> Hi Ben,
>>
>> On 7/21/26 6:23 AM, Ben Horgan wrote:
>>> On 7/20/26 23:54, Reinette Chatre wrote:
>>>> On 7/20/26 6:30 AM, Ben Horgan wrote:
>> ...>
>>> The former, info/ contains a directory for each allocation scope of each resource.
>>>
>>>
>>>
>>> info
>>> ├── L2
>>> │   ├── resource_schemata
>>> │   │   ├── L2
>>> │   │   ├── L2_CMAX
>>> │   │   └── L2_CMIN
>>> │   └── scope : L2
>>> ├── L3
>>> │   ├── resource_schemata
>>> │   │   ├── L3
>>> │   │   ├── L3_CMAX
>>> │   │   └── L3_CMIN
>>> │   └── scope : L3
>>> ├── MB
>>> │   ├── resource_schemata
>>> │   │   ├── MB
>>> │   │   │   └── MB_MAX
>>> │   │   ├── MB_MIN
>>> │   │   ├── MB_PBM
>>> │   │   └── MB_PROP
>>> │   └── scope : L3
>>> └── MB_NODE
>>>      ├── resource_schemata
>>>      │   ├── MB_NODE_MAX
>>>      │   ├── MB_NODE_MIN
>>>      │   ├── MB_NODE_PBM
>>>      │   └── MB_NODE_PROP
>>>      └── scope : NUMA NODE
>>>
>>
>> At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA"
>> can fit
>> in here still needs to be worked out. One thing that resctrl may need to highlight when
>> documenting this
>> new capability is that while historically a resource had a matching schemata entry, this is no
>> longer the
>> case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I
>> expect that
>> "Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control.
>>
>>  
>>> Please just consider this a mistake. MB_MAX should be a child of MB.
>>>
>>> info/
>>> ├── MB
>>> │   ├── resource_schemata
>>> │   │   └── MB
>>> │   │       └── MB_MAX
>>> │   └── scope
>>> └── MB_NODE
>>>      ├── resource_schemata
>>>      │   └── MB_NODE_MAX (Changed from MB_NODE)
>>>      └── scope
>>
>> ack.
>>
>>>
>>>>
>>>> When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
>>>> from above that it would either start out by itself having the properties of the underlying
>>>> "MAX" control or is the plan to have it be a percentage based control backed by the
>>>> underlying "MAX" hardware control?
>>>
>>> The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
>>> but at a different location in the SoC, at the memory controller rather than in the L3.
>>>
>>> To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
>>> be MB_NODE_MAX.
>>
>> ack.
>>
>>>
>>> My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
>>> pattern for control names and the pattern for resource names being <resource_type>_<scope>.
>>> (allowing for <scope> or <control_type> being missing to match existing naming.)
>>
>> This is where discussions are from my view also.
>>
>>>
>>> I don't think we should introduce more percent based controls and for new controls we can introduce
>>> a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/
>>
>> No, we should not introduce more percentage based controls per se. With the new schema format a
>> percentage based control is just a variant of a proportional scalar control.
>>
>>> as discussed previously. Although, I have been pondering on whether we can do a bit better.
>>>
>>> We could use hexadecimal point based format for controls which are a proportion of a resource and
>>> have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
>>> independent of the granularity of the control (number of parts).
>>>
>>> 0 is represented as 0x0
>>> 1 as                0x1
>>> 1/2 as              0x0.8
>>> 7/256 as            0x0.07
>>> 1/2**28             0x0.00000001
>>> etc
>>>
>>> This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
>>> values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
>>> mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
>>> hope this would be useful for other architectures too. I am preparing some RFC patches on top of
>>> your PoC for consideration of this idea and to explore some of the proposals discussed relating to
>>> generic schemata and how they land in practice from the MPAM side.
>>
>> I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in
>> resctrl input and output.
>> Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally
>> created to *avoid* rounding errors (and parsing complexity).
>>
>> [1] https://lore.kernel.org/lkml/aNFliMZTTUiXyZzd@xxxxxxxxxxxxxxx/
>>
>>>> I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM",
>>>> etc.).
>>>> Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
>>>> info/MB_NODE/resource_schemata/<control>?
>>>
>>> Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
>>> complications that it is a property of the MBW_MAX control and that it may be configurable for each
>>> PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
>>> form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
>>> name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
>>> an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.
>>
>> ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new
>> control
>> files need to be decided on? Sounds like you are headed in this direction and already have one
>> control file
>> in mind for this new type.
>>
>> With this in mind the control could be built on top of what is being developed at the moment,
>> possibly
>> be presented to user space following Dave Martin's suggestion in
>> https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@xxxxxxxxxxxxxxx/:
>>
>>   | MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]
>>
>>   or
>>
>>   | MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]
>>
>>>
>>> For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
>>> except the meaning of the bit is reversed. We can just use a consistent name in s/w though.
>>>
>>>>
>>>>>
>>>>> On an x86 system:
>>>>>
>>>>> info
>>>>> ├── MB
>>>>> │   ├── resource_schemata
>>>>> │   │   ├── MB
>>>>> │   │   └── MB_MAX (Finer grained MB, more below)
>>>>> │   └── scope
>>>>> └── MB_REGION
>>>>>      ├── resource_schemata
>>>>>      │   └── MB_REGION
>>>>>      └── scope
>>>>>
>>>>>
>>>>> Do you think this helps?
>>>>
>>>> "REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at
>>>> L3 scope.
>>>> Combine that with up to (currently) four regions each with three controls I find an interface
>>>> like above
>>>> potentially confusing to document in an intuitive way. Unless you are perhaps saying that we
>>>> should introduce
>>>> a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at
>>>> the same
>>>> scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
>>>>
>>>> Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would
>>>> still need a
>>>> way for user space to switch from one to the other which implies a dependency between "MB" and
>>>> "MB_REGION"
>>>> that is not presented in above hierarchy.
>>>
>>> OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
>>> previously seem s better fit than what I suggested.
>>
>> ok, I'll keep following that.
>>
>>>
>>>>
>>>> I think I am missing quite a bit here as I try to navigate an interface so different from what
>>>> we have
>>>> discussed so far.
>>>> I would like to explore with more detail how this interface can handle the different scenarios
>>>> we have
>>>> discussed so far.
>>>
>>> Certainly, I don't think we have got to the bottom of this yet.
>>>
>>>>
>>>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory
>>>> bandwidth allocation
>>>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains
>>>> controlled at L3 scope),
>>>> except for one configuration where it is "SYSTEM"(?) scope.
>>>> Ref.: https://lore.kernel.org/
>>>> lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/
>>>
>>> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
>>> accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
>>> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
>>> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
>>> the CPUs and so L3 instances in the group are online. Am I correct here?
>>>
>>> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
>>> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
>>> then a task running on a CPU affine to that L3 group won't be throttled unless that particular
>>> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
>>> domain.
>>
>> I'll defer to Babu for accurate answers about this hardware capability.
>>
> 
> To me, Global MBA should be considered a NODE-scoped resource. In some configurations it may appear
> as SYSTEM-scoped, but that is effectively equivalent to a single-node encompassing the entire
> system. In such cases, there is only one schemata entry controlling the whole system.
> 
> Yes, multiple L3 instances are grouped together to form a NODE. Internally, programming is still
> performed at the L3 level, but that implementation detail can be hidden from users and does not need
> to be exposed through the interface.

We seem to have two things that can both, somewhat reasonably, be called NODE scope in the resctrl
user interface but the behaviour required for an MPAM system and an AMD system appears different
from the point of view of lifecycle of the resctrl domain.

For MPAM NUMA scope the MSC instance (MPAM hardware interface) is at the memory controller and so
goes on and offline based on whether the NUMA node is offline or online. For AMD NUMA scope it looks
to me that the lifecycle of the resctrl domains would be tied to the CPUs associated with the NUMA
node. To me it does seem odd that a control with a domain associated with an offline NUMA node can
continue to throttle (cross-NUMA) traffic.

Is there any GLBE Control Domain ID or similar that is exposed to the user, e.g. is sysfs, or is
this just implicitly the NUMA id?

Thanks,

Ben