Re: [RFC PATCH v2 0/3] disable optimistic spinning for ftrace_lock

[Yafang Shao]

On Wed, Mar 11, 2026 at 8:53 PM David Laight
<david.laight.linux@xxxxxxxxx> wrote:
>
> On Wed, 11 Mar 2026 12:54:26 +0100
> Peter Zijlstra <peterz@xxxxxxxxxxxxx> wrote:
>
> > On Wed, Mar 11, 2026 at 07:52:47PM +0800, Yafang Shao wrote:
> > > Recently, we resolved a latency spike issue caused by concurrently running
> > > bpftrace processes. The root cause was high contention on the ftrace_lock
> > > due to optimistic spinning. We can optimize this by disabling optimistic
> > > spinning for ftrace_lock.
> > >
> > > While semaphores may present similar challenges, I'm not currently aware of
> > > specific instances that exhibit this exact issue. Should we encounter
> > > problematic semaphores in production workloads, we can address them at that
> > > time.
> > >
> > > PATCH #1: introduce slow_mutex_[un]lock to disable optimistic spinning
> > > PATCH #2: add variant for rtmutex
> > > PATCH #3: disable optimistic spinning for ftrace_lock
> > >
> >
> > So I really utterly hate this.
>
> Yep...
> Adding the extra parameter is likely to have a measurable impact
> on everything else.
>
> The problematic path is obvious:        find_kallsyms_symbol+142
>         module_address_lookup+104
>         kallsyms_lookup_buildid+203
>         kallsyms_lookup+20
>         print_rec+64
>         t_show+67
>         seq_read_iter+709
>         seq_read+165
>         vfs_read+165
>         ksys_read+103
>         __x64_sys_read+25
>         do_syscall_64+56
>         entry_SYSCALL_64_after_hwframe+100
>
> The code needs to drop the ftrace_lock across t_show.

It's unclear whether we can safely release ftrace_lock within t_show —
doing so would probably necessitate a major redesign of the current
implementation.

>
> Although there is a bigger issue of why on earth the code is reading the
> list of filter functions at all - never mind all the time.

bpftrace reads the complete list of available functions into
userspace, then performs matching against the target function to
determine if it is traceable.

> I'll do it by hand when debugging, but I'd have though anything using bpf
> will know exactly where to add its hooks.


-- 
Regards
Yafang