Re: [PATCH v7 13/19] x86,fs/resctrl: Add CPU hotplug support for kernel mode

From: Babu Moger

Date: Mon Nov 16 2026 - 15:08:33 EST




On 11/11/26 16:44, Babu Moger wrote:
The RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode allows the CPU's CLOSID
and/or RMID association to change when transitioning from user mode to
kernel mode. The mode is enabled or disabled on a per-CPU basis through a
backing resource group. CPUs that come online are brought into a state
consistent with the active kernel mode, while CPUs that go offline are
removed from its CPU mask.

Integrate RESCTRL_GLOBAL_ENABLE_PER_CPU mode with CPU hotplug handling so
that CPUs automatically join the active kernel mode when they come online
and are removed from it when they go offline. Also, ensure the register
state remains in a consistent reset state across kexec boot transitions.
This ensures that kernel mode behavior remains consistent across CPU
hotplug events and kexec transitions.

Introduce resctrl_kmode_global_online_cpu() and
resctrl_kmode_global_offline_cpu(), and hook them into the existing
resctrl_online_cpu() and resctrl_offline_cpu() paths. Also update
clear_closid_rmid() to reset MSR_IA32_PQR_PLZA_ASSOC register across kexec
transitions.

Signed-off-by: Babu Moger <babu.moger@xxxxxxx>
---
v7: Changelog update.
Code comment update.
Added changes to reset the PLZA register during the bootup.

v6: Update the changelog in non-bugfix form.
Update the code to remove rdtgroup->kmode.
Introduce new wrapper resctrl_configure_kmode_global() to consolidate
architecture calls.

v5: Added resctrl_kmode_offline_cpu() to disable the kernel mode
association and update kmode_cpu_mask when an associated CPU goes
offline.
This addresses sashiko comments:
https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=11
https://sashiko.dev/#/patchset/cover.1783461016.git.babu.moger%40amd.com?part=10
Also moved the patch earlier than patch 10.
[RESEND PATCH v4 10/15] fs/resctrl: Reset the kernel mode binding when an rdtgroup is removed
That seemed much more clear.

v4: New patch in the series. Patch taken from:
https://lore.kernel.org/lkml/20260611111706.1981788-5-qinyuntan@xxxxxxxxxxxxxxxxx/
Updated the code to enable kernel mode by default whenever a CPU comes
online when one of global-assign mode is enabled.
---
arch/x86/kernel/cpu/resctrl/core.c | 3 ++
fs/resctrl/rdtgroup.c | 56 ++++++++++++++++++++++++++++++
2 files changed, 59 insertions(+)

diff --git a/arch/x86/kernel/cpu/resctrl/core.c b/arch/x86/kernel/cpu/resctrl/core.c
index 14530c11bbd7..d7faf30c2885 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -704,6 +704,9 @@ static void clear_closid_rmid(int cpu)
state->cur_closid = RESCTRL_RESERVED_CLOSID;
state->cur_rmid = RESCTRL_RESERVED_RMID;
wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
+
+ if (boot_cpu_has(X86_FEATURE_PLZA))
+ wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
}

Sashiko reported:

https://sashiko.dev/#/patchset/cover.1794437052.git.babu.moger%40amd.com?part=13

=============================================================
commit 87e9e9cf13d544344f9797647860bcba9ffab37d
Author: Babu Moger <babu.moger@xxxxxxx>

x86,fs/resctrl: Add CPU hotplug support for kernel mode

This commit integrates RESCTRL_GLOBAL_ENABLE_PER_CPU kernel mode with CPU
hotplug handling and resets MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid().

diff --git a/arch/x86/kernel/cpu/resctrl/core.c
b/arch/x86/kernel/cpu/resctrl/core.c
index fc59d66f77de1..33965b760aed0 100644
--- a/arch/x86/kernel/cpu/resctrl/core.c
+++ b/arch/x86/kernel/cpu/resctrl/core.c
@@ -735,6 +735,9 @@ static void clear_closid_rmid(int cpu)
state->cur_closid = RESCTRL_RESERVED_CLOSID;
state->cur_rmid = RESCTRL_RESERVED_RMID;
wrmsrq(MSR_IA32_PQR_ASSOC, val.q);
+
+ if (boot_cpu_has(X86_FEATURE_PLZA))
+ wrmsrq(MSR_IA32_PQR_PLZA_ASSOC, 0);
}

Does zeroing MSR_IA32_PQR_PLZA_ASSOC in clear_closid_rmid() clobber the
domain's active PLZA configuration during CPU hotplug?

When a CPU goes offline while RESCTRL_GLOBAL_ENABLE_PER_CPU is active,
resctrl_arch_offline_cpu() first runs resctrl_offline_cpu() ->
resctrl_kmode_global_offline_cpu(), which clears plza_en while preserving rmid, rmid_en, closid, and closid_en in MSR_IA32_PQR_PLZA_ASSOC (or leaves the MSR untouched if the CPU was not in kmode_cpu_mask). Immediately afterward, resctrl_arch_offline_cpu() calls clear_closid_rmid() and writes 0 to MSR_IA32_PQR_PLZA_ASSOC:

arch/x86/kernel/cpu/resctrl/core.c:resctrl_arch_offline_cpu() {
...
resctrl_offline_cpu(cpu);

mutex_lock(&domain_list_lock);
for_each_capable_rdt_resource(r)
domain_remove_cpu(cpu, r);
mutex_unlock(&domain_list_lock);

clear_closid_rmid(cpu);
...
}

That overwrites the MSR value just programmed by
resctrl_kmode_global_offline_cpu() and zeroes rmid, rmid_en, closid, and
closid_en on the offlined CPU while other CPUs in the same QoS domain
still have the active kernel mode configuration programmed, violating the hardware requirement documented for union msr_pqr_plza_assoc that all fields except plza_en must be programmed identically across every CPU in a QoS domain. During CPU online, resctrl_arch_online_cpu() likewise calls clear_closid_rmid() after domain_add_cpu() and before resctrl_online_cpu() takes rdtgroup_mutex to program the active configuration.
=======================================================================

Yes. This is an issue. We cannot write 0 to MSR_IA32_PQR_PLZA_ASSOC in hotplug handler.

Couple of options:

1. Introduce the reboot notifier for resctrl and clear MSR_IA32_PQR_PLZA_ASSOC. Reboot notifier is called during kexec reboot.

or

2. Just clear the PLZA_EN bit in clear_closid_rmid().
if (boot_cpu_has(X86_FEATURE_PLZA))
msr_clear_bit(MSR_IA32_PQR_PLZA_ASSOC, PLZA_EN);

Thoughts?

Thanks
Babu