Re: [PATCH] rwsem: fix missed wakeup due to reordering of load

[Peter Zijlstra]

On Thu, Aug 24, 2017 at 01:29:27PM +0200, Peter Zijlstra wrote:
> 
> WTH did you not Cc the people that commented on your patch last time?
> 
> On Wed, Aug 23, 2017 at 04:58:55PM +0530, Prateek Sood wrote:
> > If a spinner is present, there is a chance that the load of
> > rwsem_has_spinner() in rwsem_wake() can be reordered with
> > respect to decrement of rwsem count in __up_write() leading
> > to wakeup being missed.
> 
> >  spinning writer                  up_write caller
> >  ---------------                  -----------------------
> >  [S] osq_unlock()                 [L] osq
> >   spin_lock(wait_lock)
> >   sem->count=0xFFFFFFFF00000001
> >             +0xFFFFFFFF00000000
> >   count=sem->count
> >   MB
> >                                    sem->count=0xFFFFFFFE00000001
> >                                              -0xFFFFFFFF00000001
> >                                    RMB
> 
> This doesn't make sense, it appears to order a STORE against something
> else.
> 
> >                                    spin_trylock(wait_lock)
> >                                    return
> >  rwsem_try_write_lock(count)
> >  spin_unlock(wait_lock)
> >  schedule()

Is this what you wanted to write?

---
 kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 02f660666ab8..813b5d3654ce 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -613,6 +613,33 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 	DEFINE_WAKE_Q(wake_q);
 
 	/*
+	 * __rwsem_down_write_failed_common(sem)
+	 *   rwsem_optimistic_spin(sem)
+	 *     osq_unlock(sem->osq)
+	 *   ...
+	 *   atomic_long_add_return(&sem->count)
+	 *
+	 *		- VS -
+	 *
+	 *			__up_write()
+	 *			  if (atomic_long_sub_return_release(&sem->count) < 0)
+	 *			    rwsem_wake(sem)
+	 *			      osq_is_locked(&sem->osq)
+	 *
+	 * And __up_write() must observe !osq_is_locked() when it observes the
+	 * atomic_long_add_return() in order to not miss a wakeup.
+	 *
+	 * This boils down to:
+	 *
+	 * [S.rel] X = 1		[RmW] r0 = (Y += 0)
+	 *	   MB			      RMB
+	 * [RmW]   Y += 1		[L]   r1 = X
+	 *
+	 * exists (r0=1 /\ r1=0)
+	 */
+	smp_rmb();
+
+	/*
 	 * If a spinner is present, it is not necessary to do the wakeup.
 	 * Try to do wakeup only if the trylock succeeds to minimize
 	 * spinlock contention which may introduce too much delay in the