[PATCH v2 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

The top waiter of an rt_mutex sets its sleep state. Then it spins in
rtmutex_spin_on_owner() as long as the owner is running. A wakeup sets
the waiter back to TASK_RUNNING. The spin loop does not check for this.
So a timeout or a signal is only handled when the owner blocks, or when
the waiter has to reschedule.

The rtmutex_spin_on_owner() is just an optimization with the idea that
if the owner is still running on a CPU, it will likely release it soon.
If the owner gets preempted or schedules out on its own, the spinning
will stop and the waiter will sleep. This came from PREEMPT_RT adaptive
spinning for rt_spin_locks that allowed them to behave more like normal
spin locks and speed up PREEMPT_RT performance.

The problem is that when this was implemented for futex and normal
mutexes it did not account for timeouts or other mechanisms where
the owner releasing the lock wasn't the only way for the spinning
task to exit the mutex acquire. The spinning currently does not
account for anything else but the lock being released or the owner
sleeping.

The owner of a PI futex is a user task. It can run for a long time.
With an owner spinning in userspace, a 1ms pthread_mutex_timedlock()
returned up to 1.4s late in testing. The timeout signal or any signal
to the waiter will not stop it from spinning, and the signal will be
delayed until it exits the spin.

With PREEMPT_RT, two contexts that take two ww_mutexes in opposite
order spin on each other. Only something else that needs one of the two
CPUs stops them. By default, the fair server does it after 950ms, to
run SCHED_OTHER tasks.

Luckily, when the spinning task needs to exit the mutex lock code, its
state is set to TASK_RUNNING. By having the spin check the spinning
task's state to see if it switches to running, it can exit the spin
appropriately.

Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
Cc: stable@xxxxxxxxxxxxxxx
Reviewed-by: Steven Rostedt <rostedt@xxxxxxxxxxx>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
---
 kernel/locking/rtmutex.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 5a9534c715b8..323973be2074 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1516,9 +1516,13 @@ static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock,
 		 *  - current is not longer the top waiter
 		 *  - current is requested to reschedule (redundant
 		 *    for CONFIG_PREEMPT_RCU=y)
+		 *  - current is woken, by its timeout, a signal or
+		 *    whatever else would end the sleep the spin
+		 *    stands in for
 		 *  - the VCPU on which owner runs is preempted
 		 */
 		if (!owner_on_cpu(owner) || need_resched() ||
+		    task_is_running(current) ||
 		    !data_race(rt_mutex_waiter_is_top_waiter(lock, waiter))) {
 			res = false;
 			break;
-- 
2.47.3