Re: [PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer

[Waiman Long]

On 04/18/2019 05:00 AM, Peter Zijlstra wrote:
> On Wed, Apr 17, 2019 at 01:45:10PM -0400, Waiman Long wrote:
>> On 04/17/2019 09:58 AM, Peter Zijlstra wrote:
>>> On Sat, Apr 13, 2019 at 01:22:54PM -0400, Waiman Long wrote:
>>>> +/*
>>>> + * Try to acquire read lock before the reader is put on wait queue.
>>>> + * Lock acquisition isn't allowed if the rwsem is locked or a writer handoff
>>>> + * is ongoing.
>>>> + */
>>>> +static inline bool rwsem_try_read_lock_unqueued(struct rw_semaphore *sem)
>>>> +{
>>>> +	long count = atomic_long_read(&sem->count);
>>>> +
>>>> +	if (RWSEM_COUNT_WLOCKED_OR_HANDOFF(count))
>>>> +		return false;
>>>> +
>>>> +	count = atomic_long_fetch_add_acquire(RWSEM_READER_BIAS, &sem->count);
>>>> +	if (!RWSEM_COUNT_WLOCKED_OR_HANDOFF(count)) {
>>>> +		rwsem_set_reader_owned(sem);
>>>> +		lockevent_inc(rwsem_opt_rlock);
>>>> +		return true;
>>>> +	}
>>>> +
>>>> +	/* Back out the change */
>>>> +	atomic_long_add(-RWSEM_READER_BIAS, &sem->count);
>>>> +	return false;
>>>> +}
>>> Doesn't a cmpxchg 'loop' make more sense here?
>> Not really. A cmpxchg loop will have one more correctible failure mode -
>> a new reader acquire the lock or a reader owner does an unlock. Failures
>> caused by the setting of the handoff bit or writer acquiring the lock
>> are the same for both cases. I don't see any advantage in using cmpxchg
>> loop.
> It depends on how many failures vs successes you have. I was expecting
> failure to be the most common case, and then you go from 2 atomics to 1.

Well, it really depends on the workloads. Note that an atomic trylock
will only be issued when it sees the lock is ready to be acquired. I
don't see handoff as a likely scenario. So it is either a writer has
just acquired the lock which is not possible if there is existing owning
readers or the reader count changes. For ll//sc architectures, the
failure path above will have one acquire-atomic and one relaxed-atomic.
A cmpxchg loop will have 2 acquire-atomics. In case a large number of
readers is trying to acquire the lock, we may need multiple iterations
of the cmpxchg loop to really acquire the lock. So it can have a far
more worse worst-case situation.

Cheers,
Longman