[PATCH 1/5] locking/rtmutex: Stop spinning on the owner when the waiter is woken

[Mathieu Desnoyers]

The top waiter of an rt_mutex spins while the owner runs,
rtmutex_spin_on_owner(), and does not stop when it is woken: its timeout
and a signal find it running, set it TASK_RUNNING, and the spin goes on
until the owner leaves its CPU or the waiter's own CPU has to
reschedule. A PI futex owner is a user task that may run as long as it
likes. On v7.4-rc1, in 8-CPU VMs, behind an owner spinning in userspace,
a 1ms pthread_mutex_timedlock() returns ETIMEDOUT 126ms to 631ms late on
average, 1.4s at most, a SIGUSR1 is handled up to 1.9s late, and a
waiter killed with SIGKILL is reaped 0.5s late on average. With
PREEMPT_RT the wound of a ww_mutex is missed the same way: two contexts
in an ABBA on two CPUs at SCHED_FIFO spin on each other, and the older
one gets its second lock after 96ms on average, 0.95s at most.

Stop spinning once the waiter is woken, where the sleep the spin stands
in for would end: rt_mutex_slowlock_block() then sees the timeout, the
signal or the wound, as after schedule(). An rtlock waiter is woken out
of TASK_RTLOCK_WAIT by the lock handover alone, and spins as before.

With this patch the 1ms waits end after 1.01ms on average, a signal is
handled within 0.3ms (1.4ms with PREEMPT_RT), and the ww_mutex ABBA
resolves in 12us to 33us; a contended PI mutex keeps its throughput.
The futex_lock_pi_inherit and test-ww_mutex cases added later in this
series catch both.

Fixes: 992caf7f1724 ("locking/rtmutex: Add adaptive spinwait mechanism")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@xxxxxxxxxxxx>
---
 kernel/locking/rtmutex.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 5a9534c715b8..323973be2074 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1516,9 +1516,13 @@ static bool rtmutex_spin_on_owner(struct rt_mutex_base *lock,
 		 *  - current is not longer the top waiter
 		 *  - current is requested to reschedule (redundant
 		 *    for CONFIG_PREEMPT_RCU=y)
+		 *  - current is woken, by its timeout, a signal or
+		 *    whatever else would end the sleep the spin
+		 *    stands in for
 		 *  - the VCPU on which owner runs is preempted
 		 */
 		if (!owner_on_cpu(owner) || need_resched() ||
+		    task_is_running(current) ||
 		    !data_race(rt_mutex_waiter_is_top_waiter(lock, waiter))) {
 			res = false;
 			break;
-- 
2.47.3