Re: [PATCH v4 4/7] fs,x86/resctrl: Add architecture hooks for every mount/unmount

[Reinette Chatre]

Hi Tony,

On 4/10/26 11:59 AM, Luck, Tony wrote:
> On Fri, Apr 10, 2026 at 08:16:50AM -0700, Reinette Chatre wrote:
>> On 4/9/26 1:35 PM, Luck, Tony wrote:
>>> On Mon, Apr 06, 2026 at 02:16:46PM -0700, Reinette Chatre wrote:
>>>> On 4/6/26 1:35 PM, Luck, Tony wrote:
>>>>> On Fri, Apr 03, 2026 at 05:52:30PM -0700, Reinette Chatre wrote:
>>>>>> On 3/30/26 2:43 PM, Tony Luck wrote:
>>
>>>>>>> @@ -2900,6 +2893,30 @@ static int rdt_get_tree(struct fs_context *fc)
>>>>>>>  	return ret;
>>>>>>>  }
>>>>>>>  
>>>>>>> +static int rdt_get_tree_wrapper(struct fs_context *fc)
>>>>>>> +{
>>>>>>> +	int ret;
>>>>>>> +
>>>>>>> +	mutex_lock(&resctrl_mount_lock);
>>>>>>> +
>>>>>>> +	/*
>>>>>>> +	 * resctrl file system can only be mounted once.
>>>>>>> +	 */
>>>>>>> +	if (resctrl_mounted) {
>>>>>>> +		mutex_unlock(&resctrl_mount_lock);
>>>>>>> +		return -EBUSY;
>>>>>>> +	}
>>>>>>> +
>>>>>>
>>>>>> This does not look right. Here too is resctrl_mounted accessed without rdtgroup_mutex
>>>>>> held. This change implies that resctrl_mounted is now protected by resctrl_mount_lock
>>>>>> but resctrl is not changed to respect this throughout resulting in unsafe access of
>>>>>> resctrl_mounted.
>>>>>>
>>>>>> Does this new resctrl_mount_lock need to be in resctrl fs? It really seems as though the
>>>>>> needed synchronization belongs in the architecture. Could this instead be accomplished
>>>>>> with a private mutex within the AET code?
>>>>>
>>>>> If you dig in lore for the v3 of this patch, you'll see I had the mutex in the
>>>>> AET code. But there were some complications.
>>>>>
>>>>> 1) Need to acquire in intel_aet_pre_mount() and release in intel_aet_mount_result()
>>>>> which is legal, but makes code more complex when call chains need to be compared to
>>>>> check that the mutex is being released correctly.
>>>>
>>>> Why was it needed to hold mutex for so long? I cannot find explanation here or in changelog
>>>> of v3. I did not remember correctly and considered the AET code to be doing the domain
>>>> addition. Even so, I do think a mutex internal to the arch code can be used to manage
>>>> the synchronization. Could you please elaborate why this cannot be done?
>>>
>>> I tried to move the locks into architecture code. But main problem is still
>>> handling when a user tries to mount an already mounted resctrl file system
>>> and gets -EBUSY.
>>>
>>> In that case file system calls resctrl_arch_pre_mount() with the file system
>>> mounted. You suggested that the AET code could detect and ignore a repeat
>>> enumeration by noting that the event_group "(*peg)->pfg" is non-NULL, set by
>>
>> It would be great if resctrl only needs to enumerate once but I do not see how that
>> is possible since there is no clear indication from PMT driver whether (all) its 
>> data is ready or not.
> 
> That was the root problem. But allowing INTEL_PMT_TELEMETRY to be a
> module adds the additional constraint that the module cannot be unloaded
> while resctrl is mounted. If that happens, the mappings to the MMIO
> space disappear. The existing intel_pmt_get_regions_by_feature() and
> intel_pmt_put_feature_group() do not do module_{get,put}() to prevent
> that. New series under development addresses this.

ack.

> 
>>> the original enumeration. But that fails in this scenario:
>>>
>>> 	# rmmod pmt_telemetry
>>> 	# mount -t resctrl resctrl /sys/fs/resctrl
>>> 		... succeeds, but without AET present
>>> 	# modprobe pmt_telemetry
>>> 	# mount -t resctrl resctrl /sys/fs/resctrl
>>> 		... enumeration success, but now calls resctrl_enable_mon_event()
>>> 		... with the file system mounted
>>
>> Thank you for catching this.
>>
>>>
>>> I think the bast solution for this is to change definition of resctrl_arch_pre_mount()
>>> from "called on every mount attempt" to "called only when resctrl is NOT mounted".
>>> This is because architecture code cannot tell whether the file system is mounted.
>>
>> Does this mean the addition of the extra locking in resctrl fs? I am not comfortable with
>> that asymmetrical locking.
> 
> Conceptually the change here is from "architecture must do all
> enumeration before file system code starts" to "architecture is allowed
> to make changes when the file system is not mounted".

This does not seem to be something that resctrl should enforce in general. Architecture
could theoretically make changes any time it wants and stage them for resctrl fs to
consume when it is able to do so.

> There are currently several limits to what is safe to do because file
> system only does a partial cleanup on unmount. But general development
> direction has been to move initialization code into mount path. For AET
> things need minimal tweaks today. For some future feature more
> refactoring from "init" to "mount" might be needed.

Right ... and we need to consider how the architecture and filesystem boundaries are consistent
so that this is managed well. Just adding a lock for convenience does not seem appropriate to me.

> 
>> I think a problem here is that resctrl_enable_mon_event() gives the architecture code the
>> capability of manipulating internal resctrl fs state directly. This is a consequence
>> of the original design before the arch/fs split. I see the additional resctrl fs locking
>> as an attempt to make it easier for the arch to manipulate fs state but I do not think we
>> should go in that direction, instead I think it is better to improve the arch/fs boundaries
>> here.
>>
>> To that end, what if resctrl uses its familiar "capable" vs "enable" separation here?
>> That is, the architecture informs resctrl fs whether it is *capable* of a feature and
>> resctrl fs, based on interactions with user space, determines whether the feature is
>> *enabled*?
> 
> For the AET features user space could look at /sys/class/intel_pmt/*/guid to see
> which events could be upgraded from "capable" to "enabled". But how can that
> be conveyed to "resctrl fs"? Adding more mount options would seem to be

User space should not be involved here and I do not see any need to add new mount
options. It is the architecture that marks events as capable if it supports the events.

> the only choice. Config files under "info/" are only available after
> mount. But by then domains have been created and the top level mon_data
> directory populated.

Why would this be needed? 

Here is how I understand the relationship:

- arch enumerates a new feature at any time and calls resctrl_capable_mon_event()
  to mark any event discovered during enumeration as "capable".
- arch is responsible for domain creation and is trusted to do so *after*
  marking any events as "capable", any needed per-domain state is created for
  the "capable" events.

- when resctrl fs is mounted, before creating any files, resctrl fs automatically
  sets all "capable" events to "enabled". resctrl fs will create needed files only for
  "enabled" events. 

- if an arch discovers new events after resctrl is mounted then it can still
  enumerate the events and mark them as "capable" - resctrl fs will pick that up
  on remount.

- arch can only disable an event as part of the unmount handler, this will clear
  "capable" as well as "enabled". This can be enforced with a check in the
  callback where only rdtgroup_mutex should be needed to access resctrl_mounted.

> 
> This also runs into problems if INTEL_PMT_TELEMETRY is unloaded between
> telling the file system that some set of events are "capable" and the
> file system asking to enable them.

This is existing problem and sounds like you are solving this with the module_get()
and module_put(). 

> 
>> For a simpler addition resctrl could obtain a new helper, for example,
>> resctrl_capable_mon_event() that only marks an event as "capable". resctrl_enable_mon_event()
>> could remain as the "early" call that marks events as capable
>> and enabled but may be deprecated. Both of these *have* to be called before any domains
>> are created since per-domain state would now depend on whether the system is "capable"
>> of an event or not. resctrl fs can assume that all "capable" events needs to be enabled
>> and it can mark them so at the beginning of each mount, resctrl will only expose
>> "enabled" events.
>>
>> There are likely some simplifications on top of current implementation to not make
>> this too invasive.
>>
>> What do you think?
> 
> I think it opens a new can of worms. Maybe the most challenging is for
> the file system to add a "hold" to the INTEL_PMT_TELEMETRY module when
> enabling an event.

This is a different problem that needs to be solved also and it sounds as though you
have a solution for that.

>>>>> 2) The "only mounted once" case meant extra state (AET_PRESENT, which you note
>>>>> in next patch may be redundant) because intel_aet_pre_mount() is called, but
>>>>> needs to do nothing.
>>>>
>>>> Right, I do not see need for extra state. In fact, since it is not clear to me that
>>>> PMT enumeration will be complete when intel_pmt_get_regions_by_feature() is called it
>>>> seemed worthwhile to only rely on event_group::pfg - if PMT enumeration was not complete
>>>> during mount N it may be complete on mount N+1? This creates a poor user interface
>>>> though since user would need an alternate way to know if AET is supported and then
>>>> a "remount until it works" approach.
>>>
>>> The race remains, and is lost if resctrl is auto-mounted at boot from /etc/fstab.
>>>
>>> The user can tell if AET is supported with:
>>>
>>> 	$ grep ^ /sys/class/intel_pmt/*/guid
>>>
>>> and checking if any of the RMID based event guids are present on the system.
>>>
>>> Delta T for the race is small enough that delaying the mount to some other
>>> startup script should be sufficient. Users are likely to have such a script
>>> to create the CTRL_MON directories and configure schemata for their workload.
>>> So annoying, but easily solved.
>>
>> I do not have a clear understanding of how the new implementation with registration
>> function will look to understand the races involved.
> 
> I think I'm ready with v5 of the series. I can post (as "RFC") so you

You consider it ready without completing this discussion? I interpret this to mean that
I am wasting my time trying to discuss. 

> can see, and comment, on the details.

... and it is already posted ... before this discussion is complete ... unless you feel
that the hour between your response here and the posting of the new version is sufficient
grace for me to respond.
The first four versions of this work were all sent within one week. I tried to
collaborate with you on v4 but now you sent v5 without completing the discussion.
I cannot collaborate with such rapid attempts at throwing stuff to see what sticks.

>>>>> Adding resctrl_mount_lock to the file system code made things simpler. The
>>>>
>>>> Adding complications to resctrl fs to make things simpler for x86?
>>>
>>> I believe it is necessary, since architecture cannot tell if the file system
>>> is mounted.
>>>
>>>>> pre-mount code can't be called with rdtgroup_mutex held because it needs to
>>>>> build the domains. That needs cpus_read_lock() + mutex_lock(&domain_list_lock);
>>>>
>>>> ack. Can an arch-specific mutex be used instead?
>>>
>>> See above.
>>>
>>>>> I need to add more comments on locking. resctrl_mounted is only modified when both
>>>>> resctrl_mount_lock AND rdtgroup_mutex are held. I believe that makes it safe to
>>>>> read the value of resctrl_mounted with just rdtgroup_mutex held.
>>>>
>>>> ...but not to read it with only resctrl_mount_lock held as in snippet above.
>>>
>>> Holding either of resctrl_mount_lock or rdtgroup_mutex makes it safe to
>>> read the value of resctrl_mounted as it can only be modified when both
>>> mutexes are held.
>>
>> I am concerned about this approach due to it not being symmetrical and how resctrl fs
>> now adds additional locking to accommodate drivers.
> 
> I'm not sure I see the asymmetry. File system code already calls architecture
> code with some set of locks held. This just adds a new lock (at the top of
> the locking hierarchy) that is held across calls to resctrl_arch_pre_mount()
> and resctrl_arch_unmount().

The asymmetry is in the inconsistency which locks are used to interact with
resctrl_mounted ... in this version at least.

Reinette