[PATCH-tip v2 1/3] locking/rwsem: Check wait_list without lock if spinner present

[Waiman Long]

In rwsem_wake(), we can safely check the wait_list to see if waiters
are present without lock when there are spinners to fall back on in
case we miss a waiter.  The advantage is that we can save a pair of
spin_lock/unlock calls when the wait_list is empty. This translates
to a reduction in latency and hence slightly better performance.

The raw_spin_trylock_irqsave() call is also being replaced by a
raw_spin_is_locked() to reduce the overhead of irqsave/irqrestore
especially when the trylock fails.

On a 2-socket 36-core x86-64 E5-2699 v3 system, a rwsem microbenchmark
was run with 36 locking threads doing 1 million writer lock/unlock
operations each, the resulting locking rates (avg of 3 runs) on a
4.11-rc1 based kernel were 4,923 Mop/s and 5,136 Mop/s without and
with the patch respectively. That was an increase of about 4%.

On the same system, a 36-thread fio direct IO random write test to
the same 2GB file on a XFS formatted ramdisk was run. The aggregated
bandwidth was 1564.4 MB/s and 1593.0 MB/s before and after the
patch. That was an increase of about 2%.

Signed-off-by: Waiman Long <longman@xxxxxxxxxx>
---
 kernel/locking/rwsem-xadd.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..f31dd61b 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -587,7 +587,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 
 	/*
 	 * If a spinner is present, it is not necessary to do the wakeup.
-	 * Try to do wakeup only if the trylock succeeds to minimize
+	 * Try to do wakeup only if the wait_lock is free to minimize
 	 * spinlock contention which may introduce too much delay in the
 	 * unlock operation.
 	 *
@@ -595,7 +595,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 	 *    ---------------		-----------------------
 	 * [S]   osq_unlock()		[L]   osq
 	 *	 MB			      RMB
-	 * [RmW] rwsem_try_write_lock() [RmW] spin_trylock(wait_lock)
+	 * [RmW] rwsem_try_write_lock() [RmW] spin_is_locked(wait_lock)
 	 *
 	 * Here, it is important to make sure that there won't be a missed
 	 * wakeup while the rwsem is free and the only spinning writer goes
@@ -611,12 +611,18 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
 		 * state is consulted before reading the wait_lock.
 		 */
 		smp_rmb();
-		if (!raw_spin_trylock_irqsave(&sem->wait_lock, flags))
+
+		/*
+		 * Normally checking wait_list without wait_lock isn't safe
+		 * as we may miss an incoming waiter. With spinners present,
+		 * however, we have someone to fall back on in case that
+		 * happens.
+		 */
+		if (list_empty(&sem->wait_list) ||
+		    raw_spin_is_locked(&sem->wait_lock))
 			return sem;
-		goto locked;
 	}
 	raw_spin_lock_irqsave(&sem->wait_lock, flags);
-locked:
 
 	if (!list_empty(&sem->wait_list))
 		__rwsem_mark_wake(sem, RWSEM_WAKE_ANY, &wake_q);
-- 
1.8.3.1