Re: [PATCH v2] locking/rwsem: Add reader-owned state to the owner field

[Peter Zijlstra]

On Fri, May 13, 2016 at 10:58:05AM -0700, Peter Hurley wrote:
> > Note that barrier() and READ_ONCE() have overlapping but not identical
> > results and the combined use actually makes sense here.
> > 
> > Yes, a barrier() anywhere in the loop will force a reload of the
> > variable, _however_ it doesn't force that reload to not suffer from
> > load tearing.
> > 
> > Using volatile also forces a reload, but also ensures the load cannot
> > be torn IFF it is of machine word side and naturally aligned.
> > 
> > So while the READ_ONCE() here is pointless for forcing the reload;
> > that's already ensured, we still need to make sure the load isn't torn.
> 
> If load tearing a naturally aligned pointer is a real code generation
> possibility then the rcu list code is broken too (which loads ->next
> directly; cf. list_for_each_entry_rcu() & list_for_each_entry_lockless()).
> 
> For 4.4, Paul added READ_ONCE() checks for list_empty() et al, but iirc
> that had to do with control dependencies and not load tearing.

Well, Paul is the one who started the whole load/store tearing thing, so
I suppose he knows what he's doing.

That said; its a fairly recent as things go so lots of code hasn't been
updated yet, and its also a very unlikely thing for a compiler to do;
since it mostly doesn't make sense to emit multiple instructions where
one will do, so its not a very high priority thing either.

But from what I understand, the compiler is free to emit all kinds of
nonsense for !volatile loads/stores.

> OTOH, this patch might actually produce store-tearing:
> 
> +static inline void rwsem_set_reader_owned(struct rw_semaphore *sem)
> +{
> +	/*
> +	 * We check the owner value first to make sure that we will only
> +	 * do a write to the rwsem cacheline when it is really necessary
> +	 * to minimize cacheline contention.
> +	 */
> +	if (sem->owner != RWSEM_READER_OWNED)
> +		sem->owner = RWSEM_READER_OWNED;
> +}

Correct; which is why we should always use {READ,WRITE}_ONCE() for
anything that is used locklessly.