Re: [PATCH-tip v3 02/14] locking/rwsem: Make owner available even if !CONFIG_RWSEM_SPIN_ON_OWNER

[Waiman Long]

On 04/12/2019 10:24 PM, Waiman Long wrote:
> On 04/12/2019 02:05 PM, Waiman Long wrote:
>
>>>  [locking/rwsem] adc32e8877: will-it-scale.per_thread_ops -21.0% regression
>> Will look into that also.
> I can reproduce the regression on the same skylake system.
>
> The results of the page_fault1 will-it-scale test are as follows:
>
> ÂThreadsÂÂ K2ÂÂÂÂÂ K3ÂÂÂÂÂÂ K4ÂÂÂÂÂÂ K5
> Â-------ÂÂ --ÂÂÂÂÂ --ÂÂÂÂÂÂ --ÂÂÂÂÂÂ --
> ÂÂÂ 20Â 5549772Â 5550332Â 5463961Â 5400064
> ÂÂÂ 40Â 9540445 10286071Â 9705062Â 7706082
> ÂÂÂ 60Â 8187245Â 8212307Â 7777247Â 6647705
> ÂÂÂ 89Â 8390758Â 9619271Â 9019454Â 7124407
>
> So the wake-all-reader patch is good for this benchmark. The performance
> was reduced a bit with the reader-spin-on-writer patch. It got even worse
> with the writer-spin-on-reader patch.
>
> I looked at the perf output, rwsem contention accounted for less than
> 1% of the total cpu cycles. So I believe the regression was caused by
> the behavior change introduced by the two reader optimistic spinning
> patches. These patch will make writer less preferred than before. I
> think the performance of this microbenchmark may be more dependent on
> writer performance.
>
> Looking at the lock event counts for K5:
>
> Ârwsem_opt_fail=253647
> Ârwsem_opt_nospin=8776
> Ârwsem_opt_rlock=259941
> Ârwsem_opt_wlock=2543
> Ârwsem_rlock=237747
> Ârwsem_rlock_fail=0
> Ârwsem_rlock_fast=0
> Ârwsem_rlock_handoff=0
> Ârwsem_sleep_reader=237747
> Ârwsem_sleep_writer=23098
> Ârwsem_wake_reader=6033
> Ârwsem_wake_writer=47032
> Ârwsem_wlock=15890
> Ârwsem_wlock_fail=10
> Ârwsem_wlock_handoff=3991
>
> For K4, it was
>
> Ârwsem_opt_fail=479626
> Ârwsem_opt_rlock=8877
> Ârwsem_opt_wlock=114
> Ârwsem_rlock=453874
> Ârwsem_rlock_fail=0
> Ârwsem_rlock_fast=1234
> Ârwsem_rlock_handoff=0
> Ârwsem_sleep_reader=453058
> Ârwsem_sleep_writer=25836
> Ârwsem_wake_reader=11054
> Ârwsem_wake_writer=71568
> Ârwsem_wlock=24515
> Ârwsem_wlock_fail=3
> Ârwsem_wlock_handoff=5245
>
> It can be seen that a lot more readers got the lock via optimistic
> spinning. One possibility is that reader optimistic spinning causes
> readers to spread out into more lock acquisition groups than without. The
> K3 results show that grouping more readers into one lock acquisition
> group help to improve performance for this microbenchmark. I will need
> to run more tests to find out the root cause of this regression. It is
> not an easy problem to solve.

Just an update on my will-it-scale regression investigation. I have
tried various ways to tune the rwsem code to get more performance out
from this benchmark. I got some minor improvements but nothing major. So
it looks like that there are some workloads that have performance hurted
by reader optimistic spinning and this benchmark is one of them. Now I
am testing an adaptive reader optimistic spinning disabling patch that
shows great promise as I was able to bring back a major portion of the
lost performance. I will try to make the patch more aggressive to see if
it can bring most of the lost performance back.

Cheers,
Longman