Re: [RESEND PATCH v4 05/15] x86,fs/resctrl: Introduce architecture hooks to program kernel-mode

[Reinette Chatre]

Hi Babu,

On 7/7/26 2:50 PM, Babu Moger wrote:
> Kernel-mode policies defined by enum resctrl_kernel_mode must be applied to

"Kernel-mode policies" -> "Kernel modes"?

> each affected CPU whenever a policy is selected or its scope changes.

What is an "affected CPU"?


policy -> "mode"? or rather: "whenever a policy is selected or its scope changes" ->
"whenever user space switches the kernel mode or changes which CPUs are associated
with the active kernel mode"?

> Generic resctrl therefore requires an architecture-specific interface to
> program allocation and monitoring associations in hardware across a given
> CPU mask.
> 
> Introduce a helper, resctrl_arch_configure_kmode(),  to handle kernel-mode
> programming. On x86/AMD systems, this helper programs the
> MSR_IA32_PQR_PLZA_ASSOC register on all online CPUs in the specified mask
> via on_each_cpu_mask(). Also provide a no-op stub for MPAM systems.

No need to describe the code details, please just make it high level of what
the code accomplished as opposed to describing the code self.

> 
> Generic resctrl does not invoke this hook yet; it will be used when user
> space selects a kernel-mode policy or updates the associated CPU set.

Looking ahead how this arch helper is used it really is a "one size fits all"
based on what AMD requires. Specifically, as I see it this architecture helper
is called under three very different scenarios:
- A new kernel mode is activated
- CPUs are added/removed from an active kernel mode
- A kernel mode is de-activated.

There is no way for an architecture to distinguish these three scenarios. An architecture
that, for example, needs to do some arch-specific init to support a particular mode will
not know when it should do this.

This "one size fits all" may be ok for an initial approach until we learn what other
architectures require, but the API needs to be clear on when and how architecture can
expect it to be called from resctrl fs.

Consider, for example, the API description containing text/contract like:
- If a per-cpu kernel mode is active when user space switches to a new per-cpu kernel
  mode then resctrl_arch_configure_kmode() will first be called to de-activate the
  active kernel mode on all CPUs that the kernel mode is active on.
- When user space switches to a new per-cpu kernel mode then resctrl_arch_configure_kmode() is
  called with cpu_online_mask. 
- When user space adds a CPU to an active per-cpu kernel mode ...
- When user space removes a CPU from to an active per-cpu kernel mode ...
- resctrl fs will always provide the same closid, rmid, and "assign_mon" parameters when
  activating a kernel mode, all interactions (adding/removing CPU) while the kernel mode is
  active, as well as when de-activating the kernel mode.

> 
> Signed-off-by: Babu Moger <babu.moger@xxxxxxx>
> ---
> v4: Added assign_mon parameter in resctrl_arch_configure_kmode() to program the RMID
>     as discussed in below.
>     https://lore.kernel.org/lkml/20260605100642.1103628-1-qinyuntan@xxxxxxxxxxxxxxxxx/
>     Changed cpumask type to "const struct cpumask *cpu_mask".
>     Added MPAM stub to avoid any linking issues when resctrl_arch_configure_kmode()
>     is called from FS layer. Thanks to Qinyun.
>     Re-wrote the changelog to be generic.
>     Updated code comments.
> 
> v3: Removed task based PLZA implementation so related changes are removed.
>     Removed handling of rmid_en as it is not required. The group type assigned
>     will be different so the monitoring part is already taken care.
>     Updated the change log with details.
>     Removed resctrl_arch_set_kmode() as arch only provides the modes supported.
>     It is FS which decided which mode to apply.
> 
> v2: Updated the commit message to include the sequence of steps to enable PLZA.
>     Added mode code comments for clarity.
>     Added kmode to functin names to be generic.
> ---
>  arch/x86/kernel/cpu/resctrl/ctrlmondata.c | 36 +++++++++++++++++++++++
>  drivers/resctrl/mpam_resctrl.c            |  5 ++++
>  include/linux/resctrl.h                   | 15 ++++++++++
>  3 files changed, 56 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> index b20e705606b8..025f139434f2 100644
> --- a/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> +++ b/arch/x86/kernel/cpu/resctrl/ctrlmondata.c
> @@ -131,3 +131,39 @@ int resctrl_arch_io_alloc_enable(struct rdt_resource *r, bool enable)
>  
>  	return 0;
>  }
> +
> +static void resctrl_kmode_set_one_amd(void *arg)
> +{
> +	union msr_pqr_plza_assoc *plza = arg;
> +
> +	wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, plza->full);
> +}
> +
> +/*
> + * Program Privilege Level Zero Association (PLZA) on @cpu_mask.
> + *
> + * When @enable is true, CPL 0 allocation traffic on the targeted CPUs uses
> + * @closid from MSR_IA32_PQR_PLZA_ASSOC instead of the CLOSID from
> + * MSR_IA32_PQR_ASSOC. Monitoring is redirected to @rmid only when
> + * @assign_mon is true; otherwise kernel-mode monitoring continues to use the
> + * RMID associated with the current task.
> + *
> + * @cpu_mask:	CPUs whose PLZA MSR should be updated.
> + * @closid:	CLOSID to use for kernel-mode allocation when PLZA is enabled.
> + * @rmid:	RMID to use for kernel-mode monitoring when @assign_mon is true.
> + * @assign_mon:	Whether PLZA should provide the kernel-mode RMID.
> + * @enable:	Whether PLZA should provide the kernel-mode association.
> + */
> +void resctrl_arch_configure_kmode(const struct cpumask *cpu_mask, u32 closid, u32 rmid,
> +				  bool assign_mon, bool enable)
> +{
> +	union msr_pqr_plza_assoc plza = { 0 };
> +
> +	plza.split.rmid = rmid;
> +	plza.split.rmid_en = assign_mon;
> +	plza.split.closid = closid;
> +	plza.split.closid_en = 1;
> +	plza.split.plza_en = enable;
> +
> +	on_each_cpu_mask(cpu_mask, resctrl_kmode_set_one_amd, &plza, 1);
> +}
> diff --git a/drivers/resctrl/mpam_resctrl.c b/drivers/resctrl/mpam_resctrl.c
> index 226ff6f532fa..630b6cfc0269 100644
> --- a/drivers/resctrl/mpam_resctrl.c
> +++ b/drivers/resctrl/mpam_resctrl.c
> @@ -158,6 +158,11 @@ bool resctrl_arch_get_io_alloc_enabled(struct rdt_resource *r)
>  	return false;
>  }
>  
> +void resctrl_arch_configure_kmode(const struct cpumask *cpu_mask, u32 closid,
> +				  u32 rmid, bool assign_mon, bool enable)
> +{
> +}
> +
>  void resctrl_arch_pre_mount(void)
>  {
>  }
> diff --git a/include/linux/resctrl.h b/include/linux/resctrl.h
> index c7abed51cd5f..47db34dd167e 100644
> --- a/include/linux/resctrl.h
> +++ b/include/linux/resctrl.h
> @@ -734,6 +734,21 @@ enum resctrl_kernel_mode {
>  
>  #define RESCTRL_NUM_KERNEL_MODES (RESCTRL_KMODE_LAST + 1)
>  
> +/**
> + * resctrl_arch_configure_kmode() - Program kernel-mode association on CPUs
> + * @cpu_mask:	CPUs to update; the architecture applies the change on the
> + *		online subset of this mask.

Could architecture not expect cpu_mask to only contain online CPUs ?

> + * @closid:	Allocation class for kernel-mode traffic. On x86 this is the

"Allocation class" ? Care should only be taken when using "closid" outside of
kernel. Please see all the other examples of arch API in this file that uses closid.

> + *		CLOSID programmed when allocation is assigned for kernel work.
> + * @rmid:	Monitoring context for kernel-mode traffic. On x86 this is the

"Monitoring context" ? At this time it is quite clear to architectures how to do
needed mapping.

Some archs will use closid/rmid separately, others will consider them together.

> + *		RMID programmed when monitoring is assigned for kernel work.
> + * @assign_mon:	true to assign @rmid for kernel work; false to inherit

This is where the only distinction is required when considering other architectures.
Note that, from user space perspective, it is a monitoring group, potentially identified
with both closid/rmid  that is assigned, not just an rmid.
This is thus not a request to "assign @rmid for kernel work" but instead something like
"kernel work should be monitored by resource group identified by @rmid, or both @closid
and @rmid, depending on the architecture"  

> + *		monitoring from the user task.
> + * @enable:	true to enable kernel-mode association on the targeted CPUs.

"the targeted CPUs" -> CPUs in @cpu_mask? 
Please be explicit what is expected from architecture when "false" is provided.

> + */
> +void resctrl_arch_configure_kmode(const struct cpumask *cpu_mask, u32 closid, u32 rmid,
> +				  bool assign_mon, bool enable);
> +
>  extern unsigned int resctrl_rmid_realloc_threshold;
>  extern unsigned int resctrl_rmid_realloc_limit;
>  

Reinette