Re: [PATCH v1 00/11] RCU: Enable callbacks to benefit from expedited grace periods

[Paul E. McKenney]

On Fri, Jul 24, 2026 at 02:47:22PM +0100, Puranjay Mohan wrote:
> On Fri, Jul 24, 2026 at 1:52 PM Puranjay Mohan <puranjay12@xxxxxxxxx> wrote:
> >
> > On Fri, Jul 24, 2026 at 1:45 PM Frederic Weisbecker <frederic@xxxxxxxxxx> wrote:
> > >
> > > Le Thu, Jul 23, 2026 at 05:31:17PM -0700, Paul E. McKenney a écrit :
> > > > On Wed, Jun 24, 2026 at 06:23:42AM -0700, Puranjay Mohan wrote:
> > > > > This series lets call_rcu() callbacks be reclaimed as soon as either a
> > > > > normal or an expedited grace period that covers them has elapsed, rather
> > > > > than always waiting for a normal grace period.
> > > > >
> > > > > Motivation
> > > > > ==========
> > > > > Today there is an asymmetry: synchronize_rcu_expedited() callers get fast
> > > > > reclaim, but call_rcu() callers never benefit from those same expedited
> > > > > grace periods, even though an expedited GP proves exactly the same thing
> > > > > as a normal one -- all pre-existing readers are done.  When expedited GPs
> > > > > are running on the system (driven by other subsystems), call_rcu()
> > > > > callbacks that could already be freed instead sit in RCU_WAIT_TAIL until
> > > > > the next normal GP.  This series treats a grace period as a grace period
> > > > > regardless of how it was driven, so memory is reclaimed sooner.
> > > > >
> > > > > Design
> > > > > ======
> > > > > Callback segments now record both the normal and expedited grace-period
> > > > > sequence in struct rcu_gp_seq, and rcu_segcblist_advance() releases a
> > > > > segment as soon as poll_state_synchronize_rcu_full() reports that either
> > > > > has completed.  Three notification paths are taught about expedited
> > > > > completion so the advance actually happens: the NOCB rcuog kthreads,
> > > > > the rcu_pending() tick gate, and rcu_core().
> > > > >
> > > > > Changelog:
> > > > > RFC: https://lore.kernel.org/all/20260417231203.785172-1-puranjay@xxxxxxxxxx/
> > > > > Changes in v1:
> > > > >  - New prep patch 1 renames struct rcu_gp_oldstate to struct rcu_gp_seq
> > > > >    and its fields rgos_norm/rgos_exp to norm/exp tree-wide (Frederic).
> > > > >  - The rcu_segcblist segment field stays named gp_seq; only its type
> > > > >    changes (Frederic).
> > > > >  - Patch 8 (NOCB wake) is reworked.  v1 woke the wrong waitqueue
> > > > >    (rdp_gp->nocb_gp_wq via wake_nocb_gp() rather than the leaf
> > > > >    rnp->nocb_gp_wq[] that an rcuog kthread waiting for a GP sleeps on),
> > > > >    and the wait condition only checked the normal ->gp_seq.  The rcuog
> > > > >    grace-period wait now tracks a struct rcu_gp_seq and is released via
> > > > >    poll_state_synchronize_rcu_full(); rcu_exp_wait_wake() wakes the leaf
> > > > >    node through the new rcu_nocb_exp_cleanup() (Frederic).
> > > > >  - rcu_pending() uses a new memory-ordering-free
> > > > >    poll_state_synchronize_rcu_full_unordered() to avoid memory barriers
> > > > >    on every tick, leaving the ordering duty to rcu_core() (Frederic).
> > > > >
> > > > > Still open: Frederic asked whether the first smp_mb() in
> > > > > poll_state_synchronize_rcu_full() is needed on the callback-advance path
> > > > > (patch 6).  That path still uses the fully ordered helper; only
> > > > > rcu_pending() was switched to the unordered variant.  Happy to revisit.
> > > > >
> > > > > Puranjay Mohan (11):
> > > > >   rcu: Rename struct rcu_gp_oldstate to rcu_gp_seq
> > > > >   rcu/segcblist: Add SRCU and Tasks RCU wrapper functions
> > > > >   rcu/segcblist: Factor out rcu_segcblist_advance_compact() helper
> > > > >   rcu/segcblist: Track segment grace periods with struct rcu_gp_seq
> > > > >   rcu: Add RCU_GET_STATE_NOT_TRACKED for subsystems without expedited
> > > > >     GPs
> > > > >   rcu: Enable RCU callbacks to benefit from expedited grace periods
> > > > >   rcu: Update comments for gp_seq and expedited GP tracking
> > > > >   rcu: Wake NOCB rcuog kthreads on expedited grace period completion
> > > > >   rcu: Detect expedited grace period completion in rcu_pending()
> > > > >   rcu: Advance callbacks for expedited GP completion in rcu_core()
> > > > >   rcuscale: Add concurrent expedited GP threads for callback scaling
> > > > >     tests
> > > >
> > > > I do see the occasional failure when running TREE01, as in a failure or
> > > > four every couple hundred hours of testing.  This is the only rcutorture
> > > > scenario that exercises callback (de)offloading, which might (or might
> > > > not) be a clue.
> > > >
> > > > The console log of a failing run is attached, or you can find it here:
> > > >
> > > > https://drive.google.com/file/d/1EH2P4SDy2a-SJndkN3GbCTGAXxI2xevm/view?usp=sharing
> > > >
> > > > This is a "??? Writer stall state" failure, where rcutorture never learns
> > > > of grace-period completion.  Possibly because no grace period completed.
> > > > But there are no RCU CPU stall warnings.  In addition, these two lines
> > > >
> > > > [  631.032657] ??? Writer stall state RTWS_EXP_SYNC(4) g152813 f0x0 ->state D cpu 1
> > > > . . .
> > > > [ 3610.863114] ??? Writer stall state RTWS_EXP_SYNC(4) g736160 f0x0 ->state D cpu 1
> > > >
> > > > show that the grace-period sequence number advanced from 152813 to
> > > > 736160 in about 50 minutes.  So grace periods really are completing,
> > > > lots of them.
> > > >
> > > > This rcutorture scenario has rcu_nocbs CPUs, and CPU 5 is in trouble:
> > > >
> > > > [  631.066109] rcu:    CB 5^4->4 KblSW F21044 L21044 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > > . . .
> > > > [ 3610.869914] rcu:    CB 5^4->4 KblSW F3000848 L3000848 C116 .W120(148380/168068).N295B101 q516 S CPU 16
> > > >
> > > > There are 120 callbacks on the WAIT segment, which are waiting on
> > > > grace-period sequence number 148380, which was long gone even back at
> > > > 631.032657 seconds in (g152813).  There are 295 more callbacks in the NEXT
> > > > segment, which long ago should have been assigned a grace period and moved
> > > > into the currently empty NEXT_READY segment (which is the "." right before
> > > > the "N295").  There are yet another 101 callbacks on the bypass list.
> > > >
> > > > This suggests that the rcuog kthread isn't being awakened when it
> > > > should be.  And that something is failing to advance callbacks out of
> > > > NEXT in a timely fashion.
> > > >
> > > > Thoughts?
> > >
> > > If a grace period has elapsed while in the middle of nocb_gp_wait(),
> > > after the call to rcu_advance_cbs() and before the following polling which
> > > determines if we a grace period must be waited upon, then the kthread
> > > will wait for new callbacks only (and will ignore waiting callbacks that
> > > can now be advanced).
> > >
> > > Does the following help?
> > >
> > > diff --git a/kernel/rcu/tree_nocb.h b/kernel/rcu/tree_nocb.h
> > > index 6da1b8f52476..96d76eb6259d 100644
> > > --- a/kernel/rcu/tree_nocb.h
> > > +++ b/kernel/rcu/tree_nocb.h
> > > @@ -792,22 +792,12 @@ static noinline_for_stack void nocb_gp_wait(struct rcu_data *my_rdp)
> > >                                                            RCU_NEXT_READY_TAIL);
> > >                         raw_spin_unlock_rcu_node(rnp); /* irqs disabled. */
> > >                 }
> > > -               // Need to wait on some grace period?
> > > +
> > >                 WARN_ON_ONCE(wasempty &&
> > >                              !rcu_segcblist_restempty(&rdp->cblist,
> > >                                                       RCU_NEXT_READY_TAIL));
> > > -               /*
> > > -                * Only request a GP wait if the next pending callback's
> > > -                * GP has not already completed (normal or expedited).
> > > -                * If poll_state_synchronize_rcu_full() says it completed,
> > > -                * then rcu_advance_cbs() above already moved those
> > > -                * callbacks to RCU_DONE_TAIL, so there is no GP to wait
> > > -                * for.  Any remaining callbacks got new (future) GP
> > > -                * numbers from rcu_accelerate_cbs() inside
> > > -                * rcu_advance_cbs() and will be handled on the next pass.
> > > -                */
> > > -               if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq) &&
> > > -                   !poll_state_synchronize_rcu_full(&cur_gp_seq)) {
> > > +               // Need to wait on some grace period?
> > > +               if (rcu_segcblist_nextgp(&rdp->cblist, &cur_gp_seq)) {
> > >                         /*
> > >                          * Track the earliest pending normal and expedited GP
> > >                          * across the group so the wait below can be released by
> >
> >
> > This looks correct to me, this optimization with
> > !poll_state_synchronize_rcu_full(&cur_gp_seq) opened up this race as
> > you correctly explained.
> 
> My local reproducer says that the problem is gone, but let's see what
> Paul's 100s of hours of testing says.

I will fire it up, and thank you both!!!

And if it passes the 100s of hours (about 600), it will get 1000s of
hours over the weekend.  ;-)

							Thanx, Paul