Re: [RFC PATCH] x86,resctrl: Read event counters with an IPI when the read cannot block

[Peter Newman]

On Thu, Oct 22, 2026 at 12:58 AM Reinette Chatre
<reinette.chatre@xxxxxxxxx> wrote:
>
> Hi Shakeel,
>
> On 10/20/26 8:13 AM, Shakeel Butt wrote:
> > In the Meta fleet, readers of the resctrl mon_data files often sleep
> > while holding rdtgroup_mutex. Over several hours, a lock profiler on
> > several thousand hosts (almost all AMD) caught:
> >
> >   - 96,368 reads that held rdtgroup_mutex and cpu_hotplug_lock for
> >     17 ms on average and up to several seconds, mostly sleeping.
> >   - Three different userspace applications [1, 2, 3] doing these reads.
>
> This is very demanding usage of an interface that cannot and has never
> intended to be fast.
>
> >   - Other readers and the MBM overflow worker waiting for the mutex.
> >
> > These long stalls put gaps in the monitoring data for these
> > applications.
> >
> > The read has to run on a CPU in the L3 domain. mon_event_read() uses
> > smp_call_on_cpu() for that. It queues work on that CPU and sleeps until
> > a kworker runs it. When that CPU is busy, this takes a long time, and
> > the mutex stays held all along.
> >
> > Commit 09909e098113 ("x86/resctrl: Queue mon_event_read() instead of
> > sending an IPI") did this for MPAM, whose reads may sleep. On x86 a
> > read is just one WRMSR and one RDMSR and it don't need to sleep.
> >
> > Add CONFIG_RESCTRL_MON_READ_IRQSAFE, which x86 selects. With it, read
> > the counter with interrupts off: on the current CPU if it is a
> > housekeeping (not nohz_full) CPU in the domain, otherwise with an IPI
> > to one. If every CPU in the domain is nohz_full, keep the existing
> > smp_call_function_any() fallback. MPAM and other architectures keep
> > the old path. Peter Newman posted a similar change in 2024 [4].
>
> Did you consider the thread that followed that proposal to learn why it did
> not land?
>
> >
> > Results on a 316-CPU AMD EPYC with busy target CPUs, and readers that
> > copy the fleet agents, per 60 s run:
> >
> >                                        before    after
> >   hold time per read, average          350 us    4 us
> >   hold time per read, worst            3.4 ms    109 us
> >   time off CPU while holding           9 s       0
> >   one pass over 750 files, average     360 ms    7.4 ms
>
> The numbers make the contention clear. If I got the math right the "after"
> is still about 10us per file and with 4us spent holding rdtgroup_mutex there
> is still about 6us per file spent outside resctrl. I expect this is
> taken up by the syscalls, the various kernfs allocations to manage the read
> cycle, also a seq_buf allocation of a page (just to return a few bytes of data).
>
> I do not think the 750 file example accurately captures how "bad" things
> can get. From what I can tell this proposal only *slightly* adjusts the limit
> of how many monitor groups this deployment can support.
>
> My guess is that the 750 file example looks like possibly about 25 monitor
> groups with three events each monitored across 10 domains.
>
> The "after" value indicates that this deployment can now support more monitoring
> groups. For example, 250 monitor groups with a pass over the increased number
> of files taking about 74ms.
>
> AMD can support 4096 monitor groups though so while this change is undeniably
> an improvement, it does not address the foundational limitations of this
> interface and does not enable this deployment to use full capability of
> the systems.
>
> If the goal is indeed to obtain accurate monitoring data at regular
> intervals when using the stated hardware at its full capacity then
> I would suggest that you consider the per-group rate files that Peter
> settled on per
> https://lore.kernel.org/all/CALPaoCgpnVORZfbKVLXDFUZvv8jhpShHPzB3cwdLTZQH1o9ULw@xxxxxxxxxxxxxx/
>
> I actually cannot remember seeing an update on that work after he posted that
> message and my assumption has been that that is what they have been using since
> then.

These changes have been available within Google for a while, but I
don't believe they've ever gone into use. They've had a lot of other
issues with the MBM implementations on AMD and have been estimating
jobs' bandwidth using the offcore PMU instead.

Shakeel, I can share the changes for you to try out, but they won't be
up to the standard for code review. If it works for you, that would be
one confirmed user and I could put more time into it.

>
> Essentially the solution builds on the monitoring data already collected by the
> per-domain overflow handler, it turns that data into a rate that user space can
> read on demand without any IPI.
>
> > The cost: interrupts are off on the reading CPU for about 0.5 us per
> > counter, and up to 131 us for a root group read with 128 MON groups.
>
> 128 MON groups sound low - it looks like AMD has 4096 possible MON groups.
>
> Interrupting a monitored application with interrupts off for so long
> is significant for a fleet whose product is the monitored application.

Interrupts are reenabled between counter reads, so the cumulative time
interrupts are masked doesn't impact latency.

The time the customer's workload wasn't scheduled was a bigger issue,
though. On the higher-availability products, this was feature would
need to be disabled by an entire domain running in nohz_full mode.

-Peter