Re: [RFC] mpam,x86,fs/resctrl: Generic schema description Proof of Concept

[Reinette Chatre]

Hi Ben,

On 7/21/26 6:23 AM, Ben Horgan wrote:
> On 7/20/26 23:54, Reinette Chatre wrote:
>> On 7/20/26 6:30 AM, Ben Horgan wrote:
...> 
> The former, info/ contains a directory for each allocation scope of each resource.
> 
> 
> 
> info
> ├── L2
> │   ├── resource_schemata
> │   │   ├── L2
> │   │   ├── L2_CMAX
> │   │   └── L2_CMIN
> │   └── scope : L2
> ├── L3
> │   ├── resource_schemata
> │   │   ├── L3
> │   │   ├── L3_CMAX
> │   │   └── L3_CMIN
> │   └── scope : L3
> ├── MB
> │   ├── resource_schemata
> │   │   ├── MB
> │   │   │   └── MB_MAX
> │   │   ├── MB_MIN
> │   │   ├── MB_PBM
> │   │   └── MB_PROP
> │   └── scope : L3
> └── MB_NODE
>     ├── resource_schemata
>     │   ├── MB_NODE_MAX
>     │   ├── MB_NODE_MIN
>     │   ├── MB_NODE_PBM
>     │   └── MB_NODE_PROP
>     └── scope : NUMA NODE
> 

At first glance this looks good to me. As you highlight below the nuances of how "Global S/MBA" can fit
in here still needs to be worked out. One thing that resctrl may need to highlight when documenting this
new capability is that while historically a resource had a matching schemata entry, this is no longer the
case. As you show above some systems may have MB_NODE resource but no MB_NODE control while I expect that
"Global MBA" (if it adopts this) may indeed have a MB_NODE resource with a MB_NODE control. 

 
> Please just consider this a mistake. MB_MAX should be a child of MB.
> 
> info/
> ├── MB
> │   ├── resource_schemata
> │   │   └── MB
> │   │       └── MB_MAX
> │   └── scope
> └── MB_NODE
>     ├── resource_schemata
>     │   └── MB_NODE_MAX (Changed from MB_NODE)
>     └── scope

ack.

> 
>>
>> When thinking about MPAM, what would the underlying hardware control of "MB_NODE" be? It looks
>> from above that it would either start out by itself having the properties of the underlying
>> "MAX" control or is the plan to have it be a percentage based control backed by the
>> underlying "MAX" hardware control?
> 
> The underlying hardware of MB_NODE would be essentially the same hardware as that backing MB_MAX,
> but at a different location in the SoC, at the memory controller rather than in the L3.
> 
> To correct myself slightly, I don't think we should have a control called MB_NODE, rather, it should
> be MB_NODE_MAX.

ack.

> 
> My understanding of previous discussions is that <resource_type>_<scope>_<control_type> it the
> pattern for control names and the pattern for resource names being <resource_type>_<scope>.
> (allowing for <scope> or <control_type> being missing to match existing naming.)

This is where discussions are from my view also.

> 
> I don't think we should introduce more percent based controls and for new controls we can introduce
> a new format to describe them. Perhaps just the positive integer with a resolution supplied in info/

No, we should not introduce more percentage based controls per se. With the new schema format a
percentage based control is just a variant of a proportional scalar control.

> as discussed previously. Although, I have been pondering on whether we can do a bit better.
> 
> We could use hexadecimal point based format for controls which are a proportion of a resource and
> have a resolution which is a power of 2. The advantage of this is that the meaning of the value is
> independent of the granularity of the control (number of parts).
> 
> 0 is represented as 0x0
> 1 as                0x1
> 1/2 as              0x0.8
> 7/256 as            0x0.07
> 1/2**28             0x0.00000001
> etc
> 
> This maps well to the MPAM fixed-point fraction point format without having the weirdness of having
> values forced to 1 or 0 not being really 0. These MPAM h/w oddities can be hidden just by using the
> mbw_min mbw_max of a control. In MPAM this could be used in CMIN, CMAX, MB_MAX, MB_MIN and I would
> hope this would be useful for other architectures too. I am preparing some RFC patches on top of
> your PoC for consideration of this idea and to explore some of the proposals discussed relating to
> generic schemata and how they land in practice from the MPAM side.

I'm going to stand with Dave Martin [1] on this point with a preference to avoid floating point in
resctrl input and output. 
Could you please elaborate where values are forced to 1 or 0? The new schema format was intentionally
created to *avoid* rounding errors (and parsing complexity).

[1] https://lore.kernel.org/lkml/aNFliMZTTUiXyZzd@xxxxxxxxxxxxxxx/

>> I also understand MPAM to support more memory bandwidth controls ("MIN", "HARDMAX"/"HARDLIM", etc.).
>> Do you envision them to exist within info/MB/resource_schemata/<control> as well as within
>> info/MB_NODE/resource_schemata/<control>?
> 
> Yes, at least for MIN, see the info/ tree above. For HARDLIM, perhaps, but HARDLIM has the added
> complications that it is a property of the MBW_MAX control and that it may be configurable for each
> PARTID or a fixed property of the h/w. When HARDLIM is configurable the control name could be of the
> form <resource_type>_<scope>_<control>_<control_param> where <control_param> is HARDLIM and the full
> name for the HARDLIM configuration on the MB_NODE resource is MB_NODE_MAX_HARDLIM. There can also be
> an info/<resource>/resource_schemata/<control>/lim file which has values, soft, hard, configurable.

ack. HARDLIM sounds like it would be a new control type. Perhaps a "boolean" type for which new control
files need to be decided on? Sounds like you are headed in this direction and already have one control file
in mind for this new type.

With this in mind the control could be built on top of what is being developed at the moment, possibly
be presented to user space following Dave Martin's suggestion in
https://lore.kernel.org/lkml/aO0Oazuxt54hQFbx@xxxxxxxxxxxxxxx/:

 | MB_HARDMAX: 0=0, 1=1, 2=1, 3=0 [...]

 or

 | MB_HARDMAX: 0=off, 1=on, 2=on, 3=off [...]

> 
> For CMAX, maximum cache capacity, there is an equivalent control SOFTLIM, which behaves as HARDLIM
> except the meaning of the bit is reversed. We can just use a consistent name in s/w though.
> 
>>
>>>
>>> On an x86 system:
>>>
>>> info
>>> ├── MB
>>> │   ├── resource_schemata
>>> │   │   ├── MB
>>> │   │   └── MB_MAX (Finer grained MB, more below)
>>> │   └── scope
>>> └── MB_REGION
>>>     ├── resource_schemata
>>>     │   └── MB_REGION
>>>     └── scope
>>>
>>>
>>> Do you think this helps?
>>
>> "REGION" is not a new scope but instead region-aware MBA is controlled and manages bandwidth at L3 scope.
>> Combine that with up to (currently) four regions each with three controls I find an interface like above
>> potentially confusing to document in an intuitive way. Unless you are perhaps saying that we should introduce
>> a new separate  "MB_REGION" L3 scope resource (so let resctrl support multiple "MB" resources at the same
>> scope?) and then *it* contains the twelve new controls within its resource_schemata directory?
>>
>> Since the region-aware controls are orthogonal to the MSR based legacy control resctrl would still need a
>> way for user space to switch from one to the other which implies a dependency between "MB" and "MB_REGION"
>> that is not presented in above hierarchy.
> 
> OK. The /sys/fs/resctrl/info/MB/schemata/mode and the MB_REGION controls a child of MB you described
> previously seem s better fit than what I suggested.

ok, I'll keep following that.

> 
>>
>> I think I am missing quite a bit here as I try to navigate an interface so different from what we have
>> discussed so far.
>> I would like to explore with more detail how this interface can handle the different scenarios we have
>> discussed so far.
> 
> Certainly, I don't think we have got to the bottom of this yet.
> 
>>
>> The other x86 feature to consider is AMD's upcoming "Global" MBA/SMBA that exposes memory bandwidth allocation
>> in "groups of L3" that I understand could usually be mapped to NODE scope (but it remains controlled at L3 scope),
>> except for one configuration where it is "SYSTEM"(?) scope.
>> Ref.: https://lore.kernel.org/lkml/8f77f498b1c77fa8fd8f5d5687f03ae598068544.1776980182.git.babu.moger@xxxxxxx/
> 
> Hmmm, I'm not sure that the scope can be considered to be NODE scope for GMBA. To me it seems to be
> accidental that it maps to the NUMA node but really the scope is just a grouping of L3 instances.
> For a control to NUMA scope I would expect the resctrl domains to go offline and online in sync with
> the NUMA nodes. For GMBA it looks like it would just going offline/online based on whether any of
> the CPUs and so L3 instances in the group are online. Am I correct here?
> 
> Assuming the domains are on L3 groups rather than NUMA also changes which end of the link the
> traffic is regulated and so how cross-NUMA traffic behaves differently. If the domain is an L3 group
> then a task running on a CPU affine to that L3 group won't be throttled unless that particular
> domain is throttled but with NUMA node domains it may be throttled if it has traffic going to that
> domain.

I'll defer to Babu for accurate answers about this hardware capability.

>>
>>>
>>> This also brings another question. On MPAM systems the 'MB_MAX' is backed by the same MSC h/w as MB
>>> but it exposed a different interface to the user. If I understand correctly intel have an option to
>>> have finer grained control of MB (delay) as well and so it would make sense to use a common name
>>> rather than just going for the MPAM centric name of MB_MAX.
>>
>> Apologies but I was not able to parse above.
> 
> Ok, let me try to explain again (although it's probably not what we want to do). My intent here was
> to try and explore whether we can reuse naming and controls across architectures in the same way we
> already have for the L2/L3 cache portion bitmap and the existing MB control.
> 
> To quote from a previous mail of yours:
> https://lore.kernel.org/lkml/a84af037-6439-4362-be07-d45143e06309@xxxxxxxxx/
> """
>   For example, on an MPAM system (if I understand correctly) the user may see:
>   info/
>   └── MB/
>       └── resource_schemata/
>           ├── MB/
>           │   └── MB_MAX/
>           └── MB_MIN/
> 
>   Compared with a possible implementation on Intel that looks like:
>   info/
>   └── MB/
>       └── resource_schemata/
>           ├── MB/
>           │   └── MB_OPT/
>           ├── MB_MAX/
>           └── MB_MIN/
> """
> 
> In the two setups MPAM MB_MAX and intel MB_OPT play the same role, a finer grained control of the
> legacy MB control. I was thinking these could share a name (MB-PRECISE), but it probably doesn't
> make sense as MB_OPT and MB_MAX have different relationships to MB_MIN.
> 
Apologies, I neglected to follow up on this after I clarified internally which RDT control should
actually be used to emulate the percentage based control. My original thoughts that you highlight
above is not correct the RDT also plans to use the "MAX" underlying hardware control for the
percentage based MB control. With this RDT and MPAM should look more similar with RDT simplified
(by dropping the region-aware terms) as:

   info/
   └── MB/
       └── resource_schemata/
           ├── MB/
           │   └── MB_MAX/
           ├── MB_OPT/
           └── MB_MIN/


Reinette