Re: [PATCH] sched_ext: Fix missing rq lock in scx_bpf_cpuperf_set()
From: Andrea Righi
Date: Thu Mar 27 2025 - 05:56:40 EST
On Thu, Mar 27, 2025 at 08:56:12AM +0100, Andrea Righi wrote:
> On Wed, Mar 26, 2025 at 02:24:16PM -1000, Tejun Heo wrote:
> > Hello, Andrea.
> >
> > On Tue, Mar 25, 2025 at 03:00:21PM +0100, Andrea Righi wrote:
> > > @@ -7114,12 +7114,22 @@ __bpf_kfunc void scx_bpf_cpuperf_set(s32 cpu, u32 perf)
> > >
> > > if (ops_cpu_valid(cpu, NULL)) {
> > > struct rq *rq = cpu_rq(cpu);
> > > + struct rq_flags rf;
> > > + bool rq_unlocked;
> > > +
> > > + preempt_disable();
> > > + rq_unlocked = (rq != this_rq()) || scx_kf_allowed_if_unlocked();
> > > + if (rq_unlocked) {
> > > + rq_lock_irqsave(rq, &rf);
> >
> > I don't think this is correct:
> >
> > - This is double-locking regardless of the locking order and thus can lead
> > to ABBA deadlocks.
> >
> > - There's no guarantee that the locked rq is this_rq(). e.g. In wakeup path,
> > the locked rq is on the CPU that the wakeup is targeting, not this_rq().
> >
> > Hmm... this is a bit tricky. SCX_CALL_OP*() always knows whether the rq is
> > locked or not. We might as well pass it the currently locked rq and remember
> > that in a percpu variable, so that scx_bpf_*() can always tell whether and
> > which cpu is rq-locked currently. If unlocked, we can grab the rq lock. If
> > the traget cpu is not the locked one, we can either fail the operation (and
> > trigger ops error) or bounce it to an irq work.
>
> Hm... that's right, it looks like this requires a bit more work than
> expected, but saving the currently locked rq might be helpful also for
> other kfuncs, I'll take a look at this.
What if we lock the rq in the scx_kf_allowed_if_unlocked() case, and for
all the other cases we ignore locking if rq == this_rq(). If we need to
operate on a different rq than the current one we could either defer the
work or just trigger an ops error. Something like:
if (scx_kf_allowed_if_unlocked()) {
rq_lock_irqsave(rq, &rf);
update_rq_clock(rq);
} else if (rq != this_rq()) {
// defer work or ops error
return;
}
lockdep_assert_rq_held(rq);
rq->scx.cpuperf_target = perf;
cpufreq_update_util(rq, 0);
if (scx_kf_allowed_if_unlocked())
rq_unlock_irqrestore(rq, &rf);
AFAICS all the current scx schedulers call scx_bpf_cpuperf_set() from
ops.running(), ops.tick() or ops.init(), so even with the ops error we
should cover all the existent cases.
The only unsupported scenario is calling scx_bpf_cpuperf_set() from
ops.enqueue() / ops.select_cpu(), but maybe we could add the deferred work
later to handle that if needed.
Thanks,
-Andrea