Re: [PATCH 1/2] x86/resctrl, Documentation: Keep mbm_assign_mode "default" on boot
From: Moger, Babu
Date: Fri Jul 24 2026 - 16:58:42 EST
Hi Reinette,
On 7/23/2026 4:46 PM, Reinette Chatre wrote:
Hi Babu,
On 7/23/26 11:03 AM, Babu Moger wrote:
On 7/21/26 15:54, Reinette Chatre wrote:
On 7/20/26 12:15 PM, Babu Moger wrote:
On 7/20/26 13:27, Reinette Chatre wrote:I see the current behavior (return accurate data when available and zero when no
On 7/20/26 10:00 AM, Babu Moger wrote:
On 7/17/26 17:56, Reinette Chatre wrote:
On 7/17/26 2:13 PM, Babu Moger wrote:
The kernel currently enables the ABMC-based "mbm_event" mode by default on
hardware that supports it. However, this can cause bandwidth monitoring
failures with existing userspace tools such as pqos.
The pqos tool mounts the resctrl filesystem and creates 16 or more resctrl
groups by default. On systems with 32 or fewer ABMC counters, this default
configuration can consume all available counters, since each group requires
one counter for local MBM and another for total MBM. If additional
monitoring groups are created, counter resources are exhausted and pqos
tool reports memory bandwidth counters as zero for those groups.
It is not obvious to me that this is a problem. If I understand correctly
there are two scenarios possible with this pqos behavior:
- ABMC is not in use ("mbm_assign_mode" is set to "default")
- pqos can create 16 or more monitor groups
- hardware still supports a limited number of counters with consequence that
underlying counters reset at any time as the different monitoring groups
need to be tracked.
- pqos can read monitoring data of all 16 monitor groups, sometimes reading the
events would return "Unavailable", sometimes reading the events return data.
- *None* of the monitoring numbers returned are guaranteed to be accurate.
- ABMC is in use ("mbm_assign_mode" is set to "mbm_event"):
- pqos can create 16 or more monitor groups
- only a subset of monitoring groups have counters assigned and these counters
are guaranteed to only track the monitor groups/events they are assigned to
- pqos can read monitoring data of all 16 monitor groups with two possibilities:
- monitor group/event has counter assigned: monitoring numbers are guaranteed to be accurate
- monitor group/event does not have counter assigned: monitoring numbers return 0
If my understanding is correct then the preference is to rather have wrong data than
see 0? This does not sound right. What am I missing?
This hardware can monitor up to 64 RMIDs without any counter resets.
How many RMIDs does the hardware claim to support via CPUID that ends up being shown to user
space via "num_rmids"?
#cat /sys/fs/resctrl/info/L3_MON/num_rmids
4096
I understood from original ABMC enabling that the underlying hardware counters of "default"
and "mbm_event" mode on AMD are the same. That is, in "default" mode the hardware does a
"best effort" assignment of hardware counters to events while "mbm_event" mode lets the user
control the assignment. It instead sounds like this is not the case and there are actually
two distinct underlying hardware counter mechanisms?
That is correct. They are two different counters.
As you know, the pqos tool creates COS1 through COS15 regardless of
the command-line options used, resulting in a total of 16 groups
including the default group. With ABMC enabled, this consumes all
available ABMC counters(32 counters, 2 counters for each group).
When pqos is invoked with the -m option, it creates additional monitoring groups. For example:
pqos -m all:0 -> creates 1 monitoring group
pqos -m all:0,1 -> creates 2 monitoring groups
Since all ABMC counters have already been allocated to the default
set of groups, no counters remain for these additional monitoring
groups. As a result, the monitoring commands report zero values,
effectively making monitoring unusable.
In contrast, the default monitoring mode can still support up to 48
additional monitoring groups (64 total RMIDs minus the 16 default
groups created by pqos).
For this reason, I still believe keeping the default monitoring mode as the default is the better option.
It is not clear to me where the "64" number comes from. Even if resctrl sets the "default"
The count of 64 is known from internal information. It can also be determined by allocating monitoring counters in a loop until the hardware starts returning an "unavailable" response, which occurs after all 64 counters have been assigned. This information is not documented.
mode as default, what will happen to the scenario you describe and 49, instead of 48,
additional groups are created? From what I understand the moment the 65th group is created user will
transition from "accurate per monitor group monitoring data for all 64 monitoring groups" to "inaccurate
per monitor group monitoring data for all 65 monitoring groups" with no indication that this is happening?
Yes. That is correct.
I believe AMD supports more than 64 RMIDs and in this case there seems to be three ranges:
[supported by ABMC, depends on events but lets say RMID 0 to 16] < [RMID 17 to 64] < [RMID 65 to total number of RMIDs supported]
Current default "mbm_event" mode uses ABMC so as you state this always results in:
- accurate counts for 16 monitor groups
- zero for all other monitor groups up to total number of RMIDs supported
That is correct. With ABMC, there are 32 available counters, allowing up to 32 monitoring events to be tracked simultaneously. Since each monitoring group requires two counters—one for local MBM and one for total MBM—the system can support only 16 monitoring groups at a time.
As I see it switching to the "default" mode would result in:
Scenario 1, 64 or fewer monitor groups are created:
- accurate counts for all monitor groups
Scenario 2, 65 or more monitor groups are created:
- inaccurate counts for all monitor groups
That is true.
I agree, it's not an ideal situation, but that's how it has always worked. At the moment, I'm not sure about a better alternative.
I do not believe there is any way for user space to know when or if system switches from "scenario 1" to
"scenario 2" on these systems and this unpredictable behavior that results in wrong data does not
sound ideal to me.
accurate data available) as the "better alternative". Intentionally returning wrong
data when it can be avoided does not sound right to me.
The changelog states:
The kernel currently enables the ABMC-based "mbm_event" mode by default on
hardware that supports it. However, this can cause bandwidth monitoring
failures with existing userspace tools such as pqos.
Could another view be that mbm_event mode is the *only* reliable accurate bandwidth
monitoring and any other mode (AMD hardware not supporting assignable counters or
AMD hardware with assignable counters choosing to use "default" mode) can cause
bandwidth monitoring failures (with existing userspace tools such as pqos)?
Yes, that is correct. However, the bandwidth counters become
inaccurate only when users start monitoring more than a certain
number of groups (64 in this case). In typical deployments, users do
not create more than 64 groups, so the counters remain accurate for
the vast majority of use cases. We do not want to introduce
oh ... hmmm ... I have a different view of deployments since the message I
keep hearing is "we need more RMIDs".
unnecessary complexity or inconvenience for those users.
This sounds like a request to (temporarily) change resctrl user interface to
accommodate users that create between 16 and 64 monitor groups (where 64 is
a magic number not exposed to users) at the expense of users that create between
65 (magic number + 1) and 4096 monitor groups?
Additionally, changing the current behavior could disrupt existing
tools and workflows, which would likely result in a significant
increase in support requests.
resctrl should aim to maintain a consistent user interface and changes to
that interface should be done with a lot of care and with very good motivation.
Above motivation sounds vague to me. You are concerned about support
requests from users that create between 16 and 64 monitor groups but not
concerned about support requests from users that create between 65 and
4096 monitor groups?
One could instead argue that this patch would cause more support requests
since users will start seeing unavailable and wrong data the moment they
create more than (the magic) 64 monitor groups.
At the same time, we are not taking away the option for users who
require strictly accurate measurements while monitoring a large
number of groups. That capability remains available. We also plan to
add more counters and enhancements over time, and it is possible
that the mbm_event option could become the default in the future.
Similarly, the default capability remains for users that prefer to
create between 16 and 64 monitor groups and do not want to assign
counters.
Without exposing the magic 64 resctrl needs to provide a good default
and I continue to find the accurate data the better option.
For now, however, our goal is for the default mode to remain the
boot-time default. This avoids unexpected behavior changes and
prevents users from being unnecessarily alarmed by issues that do
not affect their typical usage scenarios.
At the same time this commit states that, essentially, this commit is
planned to be reverted in the future. Users cannot be expected to upgrade
their hardware when upgrading the kernel so this behavior is already planned
to be reverted on the hardware that it claims to support. Why not just
keep existing behavior?
I wish I could agree with you on this, but I'm receiving significant pushback internally.
The primary argument is that, according to longstanding Linux principles, we should never break existing userspace tools.
Before ABMC was introduced:
# sudo pqos -m all:[0-191]
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
0-191 0.01 108184k 385184.0 34100.8 22.1
After ABMC was introduced and made the default mode:
# sudo pqos -m all:[0-191]
CORE IPC MISSES LLC[KB] MBL[MB/s] MBR[MB/s]
0-191 0.01 109937k 385600.0 0.0 0.0
As you can see, both MBL (Memory Bandwidth Local) and MBR (Memory Bandwidth Remote) are reported as zero when ABMC is enabled by default.
From a user perspective, this represents a regression in functionality. A tool that previously provided valid memory bandwidth monitoring data now reports zeros unless the user manually changes the monitoring mode. That effectively breaks existing workflows and monitoring applications, which is difficult to justify from a Linux userspace compatibility standpoint.
While ABMC offers advantages and is the direction we want to move toward, it is still relatively new and the ecosystem has not fully adapted to it. Making it the default before key userspace tools support it creates compatibility concerns.
I'd appreciate your thoughts on how best to address this situation.
Thanks,
Babu