Re: [PATCH RFC] locking/mutexes: don't spin on owner when wait list is not NULL.

From: Waiman Long
Date: Fri Jan 22 2016 - 08:38:29 EST

On 01/22/2016 01:09 AM, Davidlohr Bueso wrote:
On Thu, 21 Jan 2016, Waiman Long wrote:

On 01/21/2016 04:29 AM, Ding Tianhong wrote:

I got the vmcore and found that the ifconfig is already in the wait_list of the
rtnl_lock for 120 second, but my process could get and release the rtnl_lock
normally several times in one second, so it means that my process jump the
queue and the ifconfig couldn't get the rtnl all the time, I check the mutex lock
slow path and found that the mutex may spin on owner ignore whether the wait list
is empty, it will cause the task in the wait list always be cut in line, so add
test for wait list in the mutex_can_spin_on_owner and avoid this problem.

So this has been somewhat always known, at least in theory, until now. It's the cost
of spinning without going through the wait-queue, unlike other locks.


From: Waiman Long <Waiman.Long@xxxxxxx>
Date: Thu, 21 Jan 2016 17:53:14 -0500
Subject: [PATCH] locking/mutex: Enable optimistic spinning of woken task in wait list

Ding Tianhong reported a live-lock situation where a constant stream
of incoming optimistic spinners blocked a task in the wait list from
getting the mutex.

This patch attempts to fix this live-lock condition by enabling the
a woken task in the wait list to enter optimistic spinning loop itself
with precedence over the ones in the OSQ. This should prevent the
condition from happening.

And one of the reasons why we never bothered 'fixing' things was the additional
branching out in the slowpath (and lack of real issue, although this one being so
damn pathological). I fear that your approach is one of those scenarios where the
code ends up being bloated, albeit most of it is actually duplicated and can be
refactored *sigh*. So now we'd spin, then sleep, then try spinning then sleep again...
phew. Not to mention the performance implications, ie loosing the benefits of osq
over waiter spinning in scenarios that would otherwise have more osq spinners as
opposed to waiter spinners, or in setups where it is actually best to block instead
of spinning.

The patch that I sent out is just a proof of concept to make sure that it can fix that particular case. I do plan to refactor it if I decide to go ahead with an official one. Unlike the OSQ, there can be no more than one waiter spinner as the wakeup function is directed to only the first task in the wait list and the spinning won't happen until the task is first woken up. In the worst case scenario, there are only 2 spinners spinning on the lock and the owner field, one from OSQ and one from the wait list. That shouldn't put too much cacheline contention traffic to the system.