[PATCH 3/4] rcu: Perform early sequence fetch for polling locklessly

From: Frederic Weisbecker
Date: Wed Mar 16 2022 - 10:43:15 EST


The workqueue ordering guarantees that the work sees all the accesses
of the task prior to its call to the corresponding queue_work().

Therefore the sequence to poll can be retrieved locklessly.

The only downside is that it is then possible to miss the 0x1 flag set
by a prior work. But this could already happen concurrently anyway after
the exp_poll_lock is unlocked. In the worst case the slow path involving
synchronize_rcu_expedited() takes care of the situation.

Signed-off-by: Frederic Weisbecker <frederic@xxxxxxxxxx>
Cc: Neeraj Upadhyay <quic_neeraju@xxxxxxxxxxx>
Cc: Boqun Feng <boqun.feng@xxxxxxxxx>
Cc: Uladzislau Rezki <uladzislau.rezki@xxxxxxxx>
Cc: Joel Fernandes <joel@xxxxxxxxxxxxxxxxx>
---
kernel/rcu/tree_exp.h | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
index 763ec35546ed..c4a19c6a83cf 100644
--- a/kernel/rcu/tree_exp.h
+++ b/kernel/rcu/tree_exp.h
@@ -909,9 +909,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
struct rcu_node *rnp = container_of(wp, struct rcu_node, exp_poll_wq);
unsigned long s;

- raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
- s = rnp->exp_seq_poll_rq;
- raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
+ s = READ_ONCE(rnp->exp_seq_poll_rq);
if (s & 0x1)
return;
while (!sync_exp_work_done(s))
@@ -919,7 +917,7 @@ static void sync_rcu_do_polled_gp(struct work_struct *wp)
raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
s = rnp->exp_seq_poll_rq;
if (!(s & 0x1) && sync_exp_work_done(s))
- rnp->exp_seq_poll_rq |= 0x1;
+ WRITE_ONCE(rnp->exp_seq_poll_rq, s | 0x1);
raw_spin_unlock_irqrestore(&rnp->exp_poll_lock, flags);
}

@@ -949,7 +947,7 @@ unsigned long start_poll_synchronize_rcu_expedited(void)
if (rcu_init_invoked())
raw_spin_lock_irqsave(&rnp->exp_poll_lock, flags);
if ((rnp->exp_seq_poll_rq & 0x1) || ULONG_CMP_LT(rnp->exp_seq_poll_rq, s)) {
- rnp->exp_seq_poll_rq = s;
+ WRITE_ONCE(rnp->exp_seq_poll_rq, s);
if (rcu_init_invoked())
queue_work(rcu_gp_wq, &rnp->exp_poll_wq);
}
--
2.25.1