[PATCH bpf-next v3 0/4] bpf: Check recursion per task for preemptible programs
From: Mike Bazov via B4 Relay
Date: Tue Nov 17 2026 - 06:29:38 EST
fentry, fexit, raw_tp and syscall tracepoint programs can be preempted
while they run, but their recursion check is a per-CPU counter:
task A enters prog on CPU 0 prog->active = 1
task A is preempted inside prog
task B enters prog on CPU 0 dropped, counted as recursion miss
Nothing recursed, but B's event is lost. Since 7.0, x86 and arm64 have
only lazy and full preemption, so this happens on stock kernels. Our
fentry/fexit probe on inet_sendmsg() and inet_recvmsg() loses 9,254 of
8 million runs under HTTP load on a 4-CPU VM. With the series it loses
none.
The series checks recursion per task instead: a program is recursing
only if the current task is already inside it.
v2 did that for every program, which breaks a program that uses a
per-CPU map as scratch space: the second task overwrites its buffer,
and nothing counts that. So in v3 the verifier marks the programs that
keep per-CPU state, and those keep the per-CPU counter and behave
exactly as today:
- per-CPU maps, bpf_this_cpu_ptr(), bpf_per_cpu_ptr()
- tail calls: the target has no recursion check of its own
- timed may_goto: its budget is per CPU
- a private stack
Trade-offs:
- Marked programs are not fixed. A program with a per-CPU map or a
tail call loses runs to preemption as it does today.
- The verifier can't see everything. A program that keeps per-CPU
slots of its own, like array[bpf_get_smp_processor_id()], is not
marked. Today the counter keeps its runs apart. With the series two
runs can write one slot, and nothing counts that.
- raw_tp is about 2% slower in bench, see the table below. fentry and
fexit are within noise.
- Tracing, raw_tp and tracepoint programs lose the private stack
(patch 1). It is already unsafe for them: two raw_tp programs that
tail-call one handler let two tasks share its stack today. A program
with two 2000-byte frames no longer loads.
- A marked freplace program can't be attached to an unmarked raw_tp or
tracepoint program: -EINVAL.
One question: the marking is automatic. A flag that the program's
author sets would avoid the second trade-off, but existing programs
would keep losing runs until they are rebuilt. Which do you prefer?
1. bpf: Avoid per-CPU private stack in preemptible tracing progs
2. bpf: Mark programs whose runs must not overlap on a CPU
3. bpf: Check program recursion per task via bpf_ctx
4. selftests/bpf: Add preemption recursion test for tracing programs
Testing, x86-64 VM with 4 vCPUs:
The selftest pins two tasks to one CPU and makes B enter the program
while A is preempted inside it, 20 rounds per subtest. With the series
all 19 subtests pass. On bpf-next 7 fail: B is dropped in every round
(fentry, fentry_s, raw_tp, raw_tp_s, tp_syscall), A's stack is
overwritten (two_dispatchers), and the freplace attach is not rejected
(ext_attach).
bench -w2 -d12 -a trig-<x>, M/s, median of 10 runs [min-max], lazy
preemption, in boot order, on bpf-next 0fb18ce8a2e9:
trig-fentry trig-fexit trig-rawtp
bpf-next 41.65 [40.81-42.14] 38.89 [37.58-39.41] 42.62 [41.95-42.80]
series 42.40 [41.87-43.04] 39.33 [35.24-41.68] 41.31 [40.48-41.74]
bpf-next #2 41.64 [40.87-41.74] 39.29 [35.18-40.62] 42.05 [41.24-42.35]
series #2 42.34 [42.12-42.78] 39.87 [39.46-41.38] 41.53 [40.79-41.75]
Changes in v3:
- Programs with per-CPU state keep the per-CPU counter.
- One pair of helpers does the check for all runners; the syscall
tracepoint patch is folded into patch 3.
- raw_tp: walk the chain before linking the frame, 5-7% -> 2% slower.
- A marked extension can't be attached to an unmarked target.
v2: https://lore.kernel.org/bpf/20261111-bpf-preempt-v2-r2-v2-0-9d8f6b952ad2@xxxxxxxxxxxxxxx/
v1: https://lore.kernel.org/bpf/20261025-b4-bpf-trampoline-preempt-v1-0-198dcdc0c6ac@xxxxxxxxxxxxxxx/
---
Mike Bazov (4):
bpf: Avoid per-CPU private stack in preemptible tracing progs
bpf: Mark programs whose runs must not overlap on a CPU
bpf: Check program recursion per task via bpf_ctx
selftests/bpf: Add preemption recursion test for tracing programs
include/linux/bpf.h | 98 +++++-
include/linux/bpf_verifier.h | 19 ++
kernel/bpf/fixups.c | 1 +
kernel/bpf/syscall.c | 10 +
kernel/bpf/trampoline.c | 26 +-
kernel/bpf/verifier.c | 53 +++-
kernel/trace/bpf_trace.c | 11 +-
net/bpf/test_run.c | 9 +-
net/ipv4/bpf_tcp_ops.c | 6 +-
.../selftests/bpf/prog_tests/may_goto_priv_stack.c | 6 +-
.../selftests/bpf/prog_tests/timed_may_goto.c | 6 +-
.../selftests/bpf/prog_tests/trampoline_preempt.c | 352 +++++++++++++++++++++
.../selftests/bpf/progs/may_goto_priv_stack.c | 14 +-
.../selftests/bpf/progs/test_global_func3.c | 9 +
.../selftests/bpf/progs/timed_may_goto_user.c | 16 +-
.../selftests/bpf/progs/trampoline_preempt.c | 258 +++++++++++++++
.../selftests/bpf/progs/verifier_private_stack.c | 81 ++---
.../testing/selftests/bpf/progs/verifier_unwind.c | 33 +-
18 files changed, 899 insertions(+), 109 deletions(-)
---
base-commit: 32b66b94c86f009b49668de7340f2ed65aad2dc5
change-id: 20261111-bpf-preempt-v2-r2-f9f9a0dbd5d4
Best regards,
--
Mike Bazov <mikeba@xxxxxxxxxxxxxxx>