Re: [PATCH 0/6] rcu,srcu: Make call_rcu()/call_srcu() safe from any context

[Paul E. McKenney]

On Wed, Jul 29, 2026 at 09:21:59AM -0700, Puranjay Mohan wrote:
> call_rcu() and call_srcu() only ever touch their per-CPU callback lists
> with interrupts disabled: the enqueue runs under local_irq_save() (and the
> nocb locks when offloaded), and so do callback invocation and grace-period
> work.  That is fine as long as call_rcu() itself is invoked with
> interrupts enabled, but it is not always.  An NMI handler can call
> call_rcu(), and instrumentation can reenter it.  The case that prompted
> this is a BPF program attached to rcu_segcblist_enqueue() that frees an
> object: the free reaches call_rcu_tasks_trace(), which is call_srcu()
> under the hood, back on the same CPU with the srcu_data lock already held,
> and it deadlocks on that lock.  Either way, enqueuing directly can corrupt
> the list or deadlock.
> 
> Rather than scatter context checks through the enqueue, make it defer
> whenever interrupts are disabled: stage the callback on a per-CPU lockless
> list and re-issue it from an irq_work once interrupts are back on, going
> straight to the enqueue helper so the re-issue cannot defer again.  Only
> the drain side takes a lock; the staging is a bare llist_add() and stays
> safe from NMI.  This is behind a new hidden CONFIG_RCU_DEFER, which is set
> wherever a reentrant enqueue is possible (HAVE_NMI, KPROBES,
> FUNCTION_TRACER or TRACEPOINTS); without it call_rcu() enqueues exactly as
> before.
> 
> CPU offline is the awkward part.  A callback can be deferred very late in
> the outgoing CPU's teardown -- from do_idle() or cpuhp_ap_report_dead(),
> past the CPUHP_AP_SMPCFD_DYING flush that would otherwise run the irq_work
> -- so the irq_work can no longer run there to re-issue it.  rcu_barrier()
> and srcu_barrier() therefore drain the deferred lists themselves before
> they wait: for online CPUs they wait the irq_work out, and for offline
> ones they drain the list directly, since that irq_work may never run
> again.  rcutree_migrate_callbacks() drains the outgoing CPU's list too, so
> a late deferral still lands on a callback list even when nobody calls a
> barrier.  To keep those three drainers from stepping on each other, the
> drain holds a per-CPU raw lock across the llist_del_all() and the
> re-issue, so a drainer never returns having pulled callbacks off the
> deferred list but not yet put them on a callback list.  Every lock the
> re-issue touches (nocb, rcu_node, srcu_data) is already raw, so the
> nesting is fine.
> 
> The irq_work is IRQ_WORK_INIT_HARD.  It is not needed for correctness, but
> a non-HARD irq_work runs from a kthread on PREEMPT_RT and can be delayed
> under load, letting deferred callbacks pile up; running the re-issue in
> hard-irq context keeps that from turning into an OOM.
> 
> Patches 1 and 2 do Tree and Tiny RCU, 3 and 4 Tree and Tiny SRCU.  Patch 5
> teaches rcutorture to issue ->call() from a perf-overflow NMI -- the
> nmi_calls parameter, on by default -- on the flavors that advertise it,
> and checks that every callback issued from NMI is later invoked.  Patch 6
> adds the BPF reentry reproducer described above.

Nice!  I applied this series to -rcu for testing and review.  Patch 6
might want to go up a different path, but let's see how it goes.
If it goes via -rcu, it will need an appropriate ack.

							Thanx, Paul

> Puranjay Mohan (6):
>   rcu: Make call_rcu() safe to call from any context
>   rcu: Make Tiny call_rcu() safe to call from any context
>   srcu: Make call_srcu() safe to call from any context
>   srcu: Make Tiny call_srcu() safe to call from any context
>   rcutorture: Exercise ->call() from NMI context
>   selftests/bpf: Add a call_srcu() re-entry reproducer
> 
>  include/linux/srcutiny.h                      |  11 +-
>  include/linux/srcutree.h                      |   4 +
>  kernel/rcu/Kconfig                            |   6 +
>  kernel/rcu/rcu.h                              |  18 +++
>  kernel/rcu/rcutorture.c                       | 115 +++++++++++++++
>  kernel/rcu/srcutiny.c                         |  53 ++++++-
>  kernel/rcu/srcutree.c                         | 138 +++++++++++++++++-
>  kernel/rcu/tiny.c                             | 101 ++++++++++---
>  kernel/rcu/tree.c                             | 122 ++++++++++++++--
>  kernel/rcu/tree.h                             |   5 +
>  .../selftests/bpf/prog_tests/rcu_reentry.c    |  58 ++++++++
>  .../testing/selftests/bpf/progs/rcu_reentry.c |  45 ++++++
>  12 files changed, 636 insertions(+), 40 deletions(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/rcu_reentry.c
>  create mode 100644 tools/testing/selftests/bpf/progs/rcu_reentry.c
> 
> 
> base-commit: 9dc303e69bcd49f9668ca090ae45325269531fbb
> -- 
> 2.53.0-Meta
>