Re: [PATCH v6 5/6] fs/resctrl: Release the limbo RMIDs when the CDP state changes

From: Ben Horgan

Date: Thu Nov 19 2026 - 11:49:56 EST


Hi Jinjie,

On 19/11/2026 09:48, Jinjie Ruan wrote:
> On MPAM the counter of an RMID is selected by the PARTIDs of the CLOSID
> that owned it. Without CDP, CLOSID c is PARTID c; with CDP it is
> PARTIDs 2c and 2c+1. An RMID that stays in limbo across an unmount and
> a mount that changes the CDP state is read through the PARTIDs of the
> new state, which are not the PARTIDs its cache lines are tagged with.
>
> With 5 PARTIDs:
>
> mount CDP off, CLOSID c -> PARTID c
> mkdir g1..g4 CLOSIDs 1..4
> umount RMIDs enter limbo with CLOSIDs 1..4
> mount -o cdp CDP on, CLOSID c -> PARTIDs 2c, 2c+1
> limbo handler reads CLOSID 2 as PARTID 5 > PARTID_MAX 4
> MSC error interrupt, mpam_disable()
>
> The other direction is worse because nothing reports it. If resctrl is
> mounted with CDP and mounted again without it, CLOSID c is read as
> PARTID c, which is in range, so the limbo handler silently tests the
> occupancy of the PARTID of a different CLOSID and frees or keeps the
> RMIDs based on it.
>
> Where the RMID depends on the CLOSID, as on MPAM, release the RMIDs in
> limbo without reading them when a mount changes the CDP state, instead
> of restarting the limbo handler. Cache lines that still carry those
> RMIDs are accounted to the next user of each RMID. Where the RMID does
> not depend on the CLOSID, as on x86, the CDP state does not change how
> an RMID is read and the limbo handler is restarted as before.
>
> Compare the CDP state of every resource, not just the L3: on MPAM, CDP
> enabled on any resource gives every CLOSID two PARTIDs in all MSCs, so
> mounting with -o cdpl2 alone also changes how the L3 monitors read a
> CLOSID, and cache levels beyond the L3 would too.
>
> Reproduced on QEMU with the sequence above:
>
> mpam:__mpam_irq_handler: error irq from msc:1 'MSMONCFG_ID_RANGE', partid:5, pmg: 0, ris: 0
> mpam:mpam_disable: MPAM disabled due to hardware error interrupt
>
> With the change the sequence completes without an error.
>
> Fixes: 6789fb99282c ("arm_mpam: resctrl: Add CDP emulation")
> Suggested-by: Ben Horgan <ben.horgan@xxxxxxx>
> Suggested-by: Reinette Chatre <reinette.chatre@xxxxxxxxx>
> Signed-off-by: Jinjie Ruan <ruanjinjie@xxxxxxxxxx>
> Cc: <stable@xxxxxxxxxxxxxxx>
> ---
> v6:
> - Drop resctrl_arch_mon_cdp_enabled() and compare the CDP state of
> every resouce's cdp state, with a comment on why they are
> checked (Reinette, Ben).
> - Restrict the release to the systems where the RMID depends on the
> CLOSID, in the changelog and the comments, and keep the hardware IDs
> out of the file system comments (Reinette).
> - Reword the limbo_cdp_enabled comment (Reinette).
> - Drop the stable version range, the Fixes tag already gives it.
> v5:
> - Split the change of the limbo handler lifetime into the previous
> patch. The limbo handler now stops while resctrl is unmounted on every
> architecture, and only the release of the RMIDs depends on
> CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Drop the cqm_work_cpu changes. resctrl_limbo_resume() cancels the
> worker before it queues it again or releases the RMIDs.
> - Move the rdt_disable_ctx() reordering to a separate patch.
> - Add resctrl_arch_mon_cdp_enabled() instead of comparing the CDP state
> of L2 in the file system.
> - Rename limbo_cdp to limbo_cdp_enabled and document that it is only set
> with CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID.
> - Check QOS_L3_OCCUP_EVENT_ID first and drop the rmid_busy_llc check.
> - Use lockdep_assert_cpus_held() at the start of resctrl_limbo_resume().
> - Describe the reverse sequence, from CDP on to CDP off.
> - Link to v4: https://lore.kernel.org/all/20261105070934.2258613-4-ruanjinjie@xxxxxxxxxx/
> v4:
> - Use resctrl_mounted instead of a separate limbo_paused flag.
> - Record the CDP state once when resctrl is mounted.
> - Add resctrl_limbo_resume(), called once resctrl_mounted is set.
> - Previous Link: https://lore.kernel.org/all/20261031094616.1691026-1-ruanjinjie@xxxxxxxxxx/
> ---
> fs/resctrl/monitor.c | 35 +++++++++++++++++++++++++++++++++--
> 1 file changed, 33 insertions(+), 2 deletions(-)
>
> diff --git a/fs/resctrl/monitor.c b/fs/resctrl/monitor.c
> index ede1708b920e..2c8c60138768 100644
> --- a/fs/resctrl/monitor.c
> +++ b/fs/resctrl/monitor.c
> @@ -70,6 +70,13 @@ static u32 *closid_num_dirty_rmid;
> */
> static unsigned int rmid_limbo_count;
>
> +/*
> + * @limbo_cdp_enabled - True if the RMIDs in limbo were placed there while CDP
> + * was enabled. Only used when CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID is
> + * defined.
> + */
> +static bool limbo_cdp_enabled;
> +
> /*
> * @rmid_entry - The entry in the limbo and free lists.
> */
> @@ -1044,12 +1051,15 @@ int setup_rmid_lru_list(void)
> * cqm_resume_limbo_handler() - Restart the limbo handler after a mount.
> *
> * The limbo handler stops while resctrl is unmounted. Restart it on every
> - * domain that still has RMIDs in limbo.
> + * domain that still has RMIDs in limbo. Where the RMID depends on the CLOSID
> + * and the mount changed the RMID index a CLOSID is monitored with, the RMIDs
> + * in limbo can no longer be read: release them instead.
> */
> void cqm_resume_limbo_handler(void)
> {
> struct rdt_resource *r = resctrl_arch_get_resource(RDT_RESOURCE_L3);
> struct rdt_l3_mon_domain *d;
> + bool release = false;
>
> lockdep_assert_cpus_held();
> lockdep_assert_held(&rdtgroup_mutex);
> @@ -1057,6 +1067,24 @@ void cqm_resume_limbo_handler(void)
> if (!r->mon_capable || !resctrl_is_mon_event_enabled(QOS_L3_OCCUP_EVENT_ID))
> return;
>
> + /*
> + * Where the RMID depends on the CLOSID, the RMIDs in limbo can only
> + * be read with the CDP state they were freed under. Check every
> + * resource because CDP enabled on any resource changes how every
> + * CLOSID is monitored, so CDP on another resource alone also changes
> + * how the L3 monitors are read.
> + */
> + if (IS_ENABLED(CONFIG_RESCTRL_RMID_DEPENDS_ON_CLOSID)) {
> + struct rdt_resource *res;
> + bool cdp_enabled = false;
> +
> + for_each_rdt_resource(res)
> + cdp_enabled |= resctrl_arch_get_cdp_enabled(res->rid);
> +
> + release = cdp_enabled != limbo_cdp_enabled;
> + limbo_cdp_enabled = cdp_enabled;
> + }
> +
> list_for_each_entry_rcu(d, &r->mon_domains, hdr.list,
> lockdep_is_cpus_held()) {
> if (!has_busy_rmid(d))
> @@ -1067,7 +1095,10 @@ void cqm_resume_limbo_handler(void)
> * it is queued again on a CPU picked now.
> */
> cancel_delayed_work(&d->cqm_limbo);
> - cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
> + if (release)
> + release_limbo_rmids(d);

Rather than release all the rmids don't we want to check them all to see which are busy?

Thanks,

Ben

> + else
> + cqm_setup_limbo_handler(d, 0, RESCTRL_PICK_ANY_CPU);
> }
> }
>