Re: [PATCH V2] perf/x86/rapl: fix deadlock in rapl_pmu_event_stop

From: duoming
Date: Wed Sep 21 2022 - 05:53:32 EST


Hello,

On Wed, 21 Sep 2022 09:53:04 +0200 Ingo Molnar wrote:

> * Duoming Zhou <duoming@xxxxxxxxxx> wrote:
>
> > There is a deadlock in rapl_pmu_event_stop(), the process is
> > shown below:
> >
> > (thread 1) | (thread 2)
> > rapl_pmu_event_stop() | rapl_hrtimer_handle()
> > ... | if (!pmu->n_active)
> > raw_spin_lock_irqsave() //(1) | ...
> > ... |
> > hrtimer_cancel() | raw_spin_lock_irqsave() //(2)
> > (block forever)
> >
> > We hold pmu->lock in position (1) and use hrtimer_cancel() to wait
> > rapl_hrtimer_handle() to stop, but rapl_hrtimer_handle() also need
> > pmu->lock in position (2). As a result, the rapl_pmu_event_stop()
> > will be blocked forever.
> >
> > This patch extracts hrtimer_cancel() from the protection of
> > raw_spin_lock_irqsave(). As a result, the rapl_hrtimer_handle()
> > could obtain the pmu->lock.
> >
> > Fixes: 65661f96d3b3 ("perf/x86: Add RAPL hrtimer support")
> > Signed-off-by: Duoming Zhou <duoming@xxxxxxxxxx>
> > ---
> > Changes in v2:
> > - Move hrtimer_cancel() to the end of rapl_pmu_event_stop() function.
> >
> > arch/x86/events/rapl.c | 7 +++++--
> > 1 file changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
> > index 77e3a47af5a..7c110092c83 100644
> > --- a/arch/x86/events/rapl.c
> > +++ b/arch/x86/events/rapl.c
> > @@ -281,8 +281,6 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode)
> > if (!(hwc->state & PERF_HES_STOPPED)) {
> > WARN_ON_ONCE(pmu->n_active <= 0);
> > pmu->n_active--;
> > - if (pmu->n_active == 0)
> > - hrtimer_cancel(&pmu->hrtimer);
> >
> > list_del(&event->active_entry);
> >
> > @@ -300,6 +298,11 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode)
> > hwc->state |= PERF_HES_UPTODATE;
> > }
> >
> > + if (!pmu->n_active) {
> > + raw_spin_unlock_irqrestore(&pmu->lock, flags);
> > + hrtimer_cancel(&pmu->hrtimer);
> > + return;
> > + }
>
> Looks racy now: AFAICS now it's possible for rapl_hrtimer_handle() to
> execute at an arbitrary moment after pmu->lock is dropped - which could be
> use-after-free after cleanup_rapl_pmus() executes and the PMU is freed,
> right?
>
> There's also the quality-of-implementation issue of the hrtimer executing
> in a delayed fashion for the *next* event that may have been added, leading
> to possibly unexpected results.

Thank your for your suggestions! In order to solve the above problems,
I come up with the following solution.

diff --git a/arch/x86/events/rapl.c b/arch/x86/events/rapl.c
index 77e3a47af5a..a526a08ee6e 100644
--- a/arch/x86/events/rapl.c
+++ b/arch/x86/events/rapl.c
@@ -219,11 +219,13 @@ static enum hrtimer_restart rapl_hrtimer_handle(struct hrtimer *hrtimer)
struct perf_event *event;
unsigned long flags;

- if (!pmu->n_active)
- return HRTIMER_NORESTART;
-
raw_spin_lock_irqsave(&pmu->lock, flags);

+ if (!pmu->n_active) {
+ raw_spin_unlock_irqrestore(&pmu->lock, flags);
+ return HRTIMER_NORESTART;
+ }
+
list_for_each_entry(event, &pmu->active_list, active_entry)
rapl_event_update(event);

@@ -282,7 +284,7 @@ static void rapl_pmu_event_stop(struct perf_event *event, int mode)
WARN_ON_ONCE(pmu->n_active <= 0);
pmu->n_active--;
if (pmu->n_active == 0)
- hrtimer_cancel(&pmu->hrtimer);
+ hrtimer_try_to_cancel(&pmu->hrtimer);

list_del(&event->active_entry);

Firstly, the deadlock could be mitigated. Because if the timer callback function
is running, the hrtimer_try_to_cancel() will directly return.

Secondly, the race could be avoided. Because we use pmu->lock to synchronize and
move the check "if (!pmu->n_active)" into the protection scope of pmu->lock.
If the rapl_pmu_event_stop() has finished, the "pmu->n_active" equals to 0 and
the rapl_hrtimer_handle() will return "HRTIMER_NORESTART".

Thirdly, this solution will not cause quality-of-implementation issue of the hrtimer.

Best regards,
Duoming Zhou