[PATCH] futex: Fix missed wakeup during private hash resize

[Yao Kai]

A task performing a custom private hash resize can remain blocked in
uninterruptible sleep indefinitely.  The hung-task detector reports:

  INFO: task futex-resizer:314 blocked for more than 10 seconds.
  task:futex-resizer state:D stack:14824 pid:314 tgid:312 ppid:311

  Call Trace:
   __schedule+0x521/0xf30
   schedule+0x22/0xa0
   futex_hash_allocate+0x3db/0x490
   __do_sys_prctl+0x6f5/0xbd0
   do_syscall_64+0xf9/0x530
   entry_SYSCALL_64_after_hwframe+0x77/0x7f

  Kernel panic - not syncing: hung_task: blocked tasks

futex_pivot_pending() allows the resize request to continue when
either no replacement hash is pending (hash_new == NULL) or the current
hash reference count has reached zero.

After the final-reference wake, another futex task can complete the
pivot between the two observations:

  T1                                  T2

  futex_hash_allocate()
    wait_var_event(mm, ...)
      futex_pivot_pending(mm)
        hash_new != NULL
                                      futex_hash()
                                        futex_ref_get(old) -> false
                                        futex_pivot_hash(mm)
                                          hash_new = NULL
                                          __futex_pivot_hash(mm, new)
                                            rcu_assign_pointer(hash, new)
        fph = rcu_dereference(hash) /* new */
        futex_ref_is_dead(fph) -> false
      schedule()

The pivot changes the state from hash_new != NULL with a dead current
hash to hash_new == NULL with a live current hash.  The resize task can
observe hash_new in the pre-pivot state and hash in the post-pivot state,
causing futex_pivot_pending() to return false even though the pivot has
completed.  Since a successful pivot does not notify waiters, the task
can go to sleep after the only preceding wakeup has already been
consumed.

Wake waiters after every successful pivot.  A full memory barrier before
wake_up_var() pairs with set_current_state() in wait_var_event() and
orders the completed pivot before the lockless waitqueue_active() check
in wake_up_var().  The waiter therefore either observes hash_new == NULL
before sleeping or is made runnable.

Fixes: bd54df5ea7ca ("futex: Allow to resize the private local hash")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Yao Kai <yaokai34@xxxxxxxxxx>
---
 kernel/futex/core.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 90fa9d886f752..3e030e117783a 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -220,6 +220,12 @@ static bool __futex_pivot_hash(struct mm_struct *mm, struct futex_private_hash *
 		rcu_assign_pointer(mmph->hash, new);
 	}
 	kvfree_rcu(fph, rcu);
+	/*
+	 * Pair with set_current_state() in wait_var_event(), as required by
+	 * the lockless waitqueue_active() check in wake_up_var().
+	 */
+	smp_mb();
+	wake_up_var(mm);
 	return true;
 }
 
-- 
2.34.1