Re: [EXT.MSG] Re: [PATCH RFC bpf-next 0/2] bpf: Do not let preemption cause false recursion drops in trampoline progs

[Martin Kelly]

We have noticed this issue independently in testing of our own security
agent and have hit the same false recursion skips on 7.0 and 7.1
kernels. More details and measurements below.

On Tue, Nov 3, 2026 at 6:24 PM Alexei Starovoitov
<alexei.starovoitov@xxxxxxxxx> wrote:
> Today a fentry prog never runs twice at the same time on one cpu.
> Existing progs rely on that for their per-cpu maps.
>
>   task A, cpu 0                     task B, cpu 0
>   buf = lookup(&percpu_scratch)
>   fill buf
>   <preempted>
>                                     buf = lookup(&percpu_scratch)
>                                     fill buf
>                                     output(buf)
>   output(buf)   <- data of B
>
> A plain (*cnt)++ in a per-cpu map loses updates the same way.
> Today B is skipped and recursion_misses counts it.
> With the per-task check A sends a corrupted event and nothing counts it.

Our programs avoid writing to per-CPU arrays because of corruption on
PREEMPT_RT kernels, where trampolines have always been preemptible and
thus can corrupt these structures. So for us, an opt-in flag that allows
preemption (instead of skipping) would work well and not regress current
programs that don't opt in. A verifier scan for per-CPU arrays could
also work, but it would need to distinguish read-only vs write access to
those maps.

> Add bpf_preempt_disable().
> Your test forces the preemption in the middle of the prog.
> Explicit bpf_preempt_disable() will fix it.
> And you'll have an easy way to see why disabling preemtion
> for the whole program might hurt your own production.

We tried bpf_preempt_disable()/bpf_preempt_enable() around our hooks on
a 2-vCPU 7.1.5 VM with HRTICK under an open/close stress test load.
recursion_misses for the locks_remove_file fentry actually increased
slightly 29M to 31M (this may be within noise). As far as we can tell
from the code, the trampoline takes prog->active before the program's
first instruction and releases it in __bpf_prog_exit_recur() after the
program finishes. bpf_preempt_enable() calls preempt_enable(), which is
a reschedule point. That reschedule happens with prog->active still
held.

There are a few recent changes that make preemption much more likely
than it used to be. These were done for valid reasons but have made
tracing programs noticeably less reliable (see below test results).
Although newer program types like BPF-LSM are useful, they're not
broadly available on every distro, nor do they support every function
needed for security use cases, so we still depend on tracing programs.
We'd like to see if there's a way forward for allowing tracing program
preemption while minimizing recursion_misses.

Changes that increased preemption
===
7dadeaa6e851 ("sched: Further restrict the preemption modes") introduced
in kernel 7.0: on architectures with lazy preemption support, this
removes PREEMPT_NONE and PREEMPT_VOLUNTARY as options, leaving only full
and lazy. Distributions that previously shipped voluntary preemption
(where non-sleepable BPF programs were never involuntarily preempted)
now get BPF program preemption when PREEMPT_RCU is enabled.

a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL()
with SRCU-fast") introduced in kernel 7.0: results in RAW_TRACEPOINT
hooks being preemptible. This means the false skips are not only in
fentry/fexit hooks.

9213aa4784cf ("sched: Default enable HRTICK when deferred rearming is
enabled") introduced in kernel 7.1: this enables HRTICK and HRTICK_DL by
default when CONFIG_HRTIMER_REARM_DEFERRED is enabled. We think this
means that even fast BPF hooks are more likely to be preempted since it
makes the scheduler more granular.

Recursion miss results
===
In a process-heavy workload on a 2-vCPU VM (~1M fork+exec of
short-lived processes, ~606/second) we recorded the following number of
recursion_misses on a 7.0.9 and a 7.1.5 kernel. Both kernels use lazy
preemption with PREEMPT_RCU enabled. The miss rate has increased
significantly, which we suspect is due to the HRTICK change.

                                  7.0.9 (HRTICK off) 7.1.5 (HRTICK on)
                                  ================== =================
security_file_permission fentry   1 in 61,580        1 in 1,568
security_inode_getattr fentry     1 in 59,242        1 in 1,707
do_file_open fexit                1 in 813,983       1 in 2,503
sched_process_exec raw_tracepoint 1 in 1,000,816     1 in 500,408

The sched_process_exec raw_tp had few misses (1-2) but is included just
to show that raw_tp can have misses as well since tracepoints became
preemptible.

Possible approaches
===
We don't have a preferred mechanism and are generally open to any
approach that would help reduce these recursion_misses. Whatever the
mechanism, it would be ideal to cover raw_tp programs too, since
a46023d5616e made those preemptible.

We'd be happy to help test or develop patches if needed.

Thanks,
Martin