Re: [PATCH 03/13] srcutree: Add reader-free fastpath to synchronize_srcu_atomic()

[Paul E. McKenney]

On Mon, Sep 07, 2026 at 03:58:19PM +0800, Kunwu Chan wrote:
> From: Kunwu Chan <kunwu.chan@xxxxxxxxx>
> 
> synchronize_srcu_atomic() is restricted to srcu_read_lock_atomic() and
> srcu_read_unlock_atomic(), whose read-side critical sections disable
> preemption.  In the common case where there are no readers at all, the
> grace period therefore need not do the index flip.  Add a fastpath
> that sums both ranks of the per-CPU ->srcu_ctrs[] counters and, if the
> lock counts match the unlock counts on both ranks, ends the grace
> period immediately, skipping the srcu_advance_state() scans, mirroring
> the similar Tiny SRCU fastpath.
> 
> Correctness requires the counter-sum proof to follow the grace-period
> anchor written by srcu_gp_start(); placing it before the anchor could
> let this grace period miss a pre-existing reader and return without
> waiting for it.  The smp_mb() between the unlock and lock sums pairs
> with the smp_mb() in __srcu_read_lock().  The grace period is ended
> manually under ->lock and ->srcu_atomic_gp_flag.
> 
> Signed-off-by: Kunwu Chan <kunwu.chan@xxxxxxxxx>

Adding David Woodhouse on CC.

David, does this provide appropriate performance in your use case?

							Thanx, Paul

> ---
>  kernel/rcu/srcutree.c | 48 +++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 48 insertions(+)
> 
> diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
> index 7dd705eec573..533607de5728 100644
> --- a/kernel/rcu/srcutree.c
> +++ b/kernel/rcu/srcutree.c
> @@ -2113,6 +2113,8 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  {
>  	unsigned long srcu_state;
>  	struct srcu_usage *sup = ssp->srcu_sup;
> +	unsigned long rdm0, rdm1;
> +	unsigned long unlocks0, unlocks1;
>  
>  	// Initialize.	Either init_srcu_struct() was invoked or
>  	// DEFINE_SRCU() or similar was used.  Therefore, no allocation
> @@ -2149,6 +2151,52 @@ void synchronize_srcu_atomic(struct srcu_struct *ssp)
>  	srcu_gp_start(ssp);
>  	raw_spin_unlock_irq_rcu_node(sup);
>  
> +	//
> +	// Fastpath:  If there are no readers at all, neither grace-period
> +	// scan need wait, so both can be satisfied at once without doing
> +	// the index flip.  The counter-sum proof is the same as that of
> +	// srcu_readers_active_idx_check(), but spanning both indices.
> +	// Atomic SRCU guarantees that all readers are of
> +	// SRCU_READ_FLAVOR_ATOMIC, so the SLOWGP check never triggers and
> +	// the ->srcu_reader_flavor masks returned by
> +	// srcu_readers_unlock_idx() are unused.
> +	//
> +	// This proof must follow the grace-period anchor written by the
> +	// srcu_gp_start() above, never precede it.  With the anchor first,
> +	// a reader whose lock increment is missed by the sums below cannot
> +	// have incremented its lock counter before the anchor, and therefore
> +	// cannot be a pre-existing reader of this grace period.  Placing the
> +	// proof before the anchor would let this grace period miss a
> +	// pre-existing reader and return without waiting for it.
> +	//
> +	// The smp_mb() pairs with the smp_mb() in __srcu_read_lock()
> +	// (store-buffering pattern), which guarantees that a lock is always
> +	// counted if the corresponding unlock is counted, the same
> +	// memory-ordering guarantee as is provided by
> +	// srcu_readers_active_idx_check().
> +	//
> +	unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
> +	unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
> +	smp_mb(); /* A */
> +	if (srcu_readers_lock_idx(ssp, 0, false, unlocks0) &&
> +	    srcu_readers_lock_idx(ssp, 1, false, unlocks1)) {
> +		// No readers, so end this grace period manually, skipping
> +		// the index flip.  Advancing the sequence number via
> +		// rcu_seq_start() in srcu_gp_start() above and rcu_seq_end()
> +		// below keeps get_state_synchronize_srcu() and
> +		// poll_state_synchronize_srcu() working, all under ->lock
> +		// and ->srcu_atomic_gp_flag, which excludes concurrent
> +		// sequence-number updates.
> +		raw_spin_lock_irq_rcu_node(sup);
> +		rcu_seq_end(&sup->srcu_gp_seq);
> +		raw_spin_unlock_irq_rcu_node(sup);
> +		WARN_ON_ONCE(!poll_state_synchronize_srcu(ssp, srcu_state));
> +		atomic_set_release(&sup->srcu_atomic_gp_flag, 0);
> +		preempt_enable();
> +		non_block_end();
> +		return;
> +	}
> +
>  	// Wait for it to complete, helping it along.
>  	while (!poll_state_synchronize_srcu(ssp, srcu_state)) {
>  		cpu_relax();
> -- 
> 2.43.0
>