Re: [EXT.MSG] Re: [PATCH RFC bpf-next 0/2] bpf: Do not let preemption cause false recursion drops in trampoline progs

[Alexei Starovoitov]

On Thu Nov 5, 2026 at 1:12 AM UTC, Martin Kelly wrote:
> We have noticed this issue independently in testing of our own security
> agent and have hit the same false recursion skips on 7.0 and 7.1
> kernels. More details and measurements below.
>
> On Tue, Nov 3, 2026 at 6:24 PM Alexei Starovoitov
> <alexei.starovoitov@xxxxxxxxx> wrote:
>> Today a fentry prog never runs twice at the same time on one cpu.
>> Existing progs rely on that for their per-cpu maps.
>>
>>   task A, cpu 0                     task B, cpu 0
>>   buf = lookup(&percpu_scratch)
>>   fill buf
>>   <preempted>
>>                                     buf = lookup(&percpu_scratch)
>>                                     fill buf
>>                                     output(buf)
>>   output(buf)   <- data of B
>>
>> A plain (*cnt)++ in a per-cpu map loses updates the same way.
>> Today B is skipped and recursion_misses counts it.
>> With the per-task check A sends a corrupted event and nothing counts it.
>
> Our programs avoid writing to per-CPU arrays because of corruption on
> PREEMPT_RT kernels, where trampolines have always been preemptible and
> thus can corrupt these structures. 

what do you mean? the reentrance prevention counter works on PREEMPT_RT too.
per-cpu array are not corrupted there.

> So for us, an opt-in flag that allows
> preemption (instead of skipping) would work well and not regress current
> programs that don't opt in. A verifier scan for per-CPU arrays could
> also work, but it would need to distinguish read-only vs write access to
> those maps.
>
>> Add bpf_preempt_disable().
>> Your test forces the preemption in the middle of the prog.
>> Explicit bpf_preempt_disable() will fix it.
>> And you'll have an easy way to see why disabling preemtion
>> for the whole program might hurt your own production.
>
> We tried bpf_preempt_disable()/bpf_preempt_enable() around our hooks on
> a 2-vCPU 7.1.5 VM with HRTICK under an open/close stress test load.
> recursion_misses for the locks_remove_file fentry actually increased
> slightly 29M to 31M (this may be within noise). As far as we can tell
> from the code, the trampoline takes prog->active before the program's
> first instruction and releases it in __bpf_prog_exit_recur() after the
> program finishes. bpf_preempt_enable() calls preempt_enable(), which is
> a reschedule point. That reschedule happens with prog->active still
> held.

pls try _no_resched() as I suggested in the other reply.

> In a process-heavy workload on a 2-vCPU VM (~1M fork+exec of
> short-lived processes, ~606/second) we recorded the following number of
> recursion_misses on a 7.0.9 and a 7.1.5 kernel. Both kernels use lazy
> preemption with PREEMPT_RCU enabled. The miss rate has increased
> significantly, which we suspect is due to the HRTICK change.
>
>                                   7.0.9 (HRTICK off) 7.1.5 (HRTICK on)
>                                   ================== =================
> security_file_permission fentry   1 in 61,580        1 in 1,568
> security_inode_getattr fentry     1 in 59,242        1 in 1,707
> do_file_open fexit                1 in 813,983       1 in 2,503
> sched_process_exec raw_tracepoint 1 in 1,000,816     1 in 500,408

Thanks for the numbers.
Are you saying turning off HRTICK is not an option for your production?

> We don't have a preferred mechanism and are generally open to any
> approach that would help reduce these recursion_misses. Whatever the
> mechanism, it would be ideal to cover raw_tp programs too, since
> a46023d5616e made those preemptible.

agree. let's come up with a design then.