Re: futex: private hash can stay on the shared atomic refcount indefinitely after auto-scaling
From: Sebastian Andrzej Siewior
Date: Wed Sep 02 2026 - 04:09:37 EST
On 2026-09-01 23:05:59 [+0000], Nikita Taranov wrote:
> Following up with three things: a better reproducer, results from two more
> vendors, and a prototype of the approach you suggested.
>
> ## A new repro
>
> The code could be found at https://gist.github.com/nickitat/a7a46b61d6769ceec493881f82e7b344 .
>
> This one is an ordinary producer/consumer: threads park in plain FUTEX_WAIT
> like the idle workers of a thread pool, others issue plain FUTEX_WAKE on the
> same futexes, so waiters are woken, loop and re-park. Only plain FUTEX_WAIT and
> FUTEX_WAKE. The kernel's own auto-scaling during thread creation is the only
> thing that ever asks for a pivot.
>
> The control is one prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n) before any
> thread exists, which marks the hash custom and disables auto-scaling, so no
> pivot is ever requested and the stall cannot occur. Basically, the same story
> as in the first repro.
Depends. That PR_FUTEX_HASH_SET_SLOTS will block the requesting thread
until the change is active. The auto-scaling does not block the
requesting thread (the one creating the thread).
If you have already 20 threads which are busy on FUTEX_WAIT/WAKE, the
PR_FUTEX_HASH_SET_SLOTS request can also take some time if the active
counter never drops to zero (because multiple threads always own a
reference).
> v7.2, Xeon 6975P-C 96c/192t SNC=3, 8 runs of 4 s per point, clock pinned:
>
> threads auto pre-sized stalled
> 16 25.8 42.3 0/8
> 32 34.8 41.1 0/8
> 48 28.7 31.8 0/8
> 64 29.9 34.2 0/8
> 96 3.7 46.3 8/8
> 128 3.6 53.6 8/8
> 192 4.0 38.7 8/8
>
> Throughput is far lower than the numbers in my earlier mails because the work
> is real; the comparison is the point. The control (pre-sized) is almost flat
> across the whole range, so the work does not get harder with thread count
> -- only the arm that requests a pivot collapses, and only above a threshold.
How bad is this in general? Do you have a workload where you fire a
process, create 128 threads and then seconds later kill the whole thing?
I would imagine that in general it is long living process and once the
"final" privat hash gets installed, it remains…
> ## It is not only this Intel box
>
> > You might have hit the sweet spot with your CPU. I have here a 144 CPU
> > box and
>
> I wrote previously that this Intel host was the only one where the repro worked,
> based on a 96-thread sweep with the old reproducer. At 192 threads, with the
> reproducer above, Graviton4 does reach the same state -- less often,
> but permanently when it does.
>
> 192 threads, 15 s runs, per-second traces:
>
> latched permanently transient never
> Xeon 6975P-C 8 / 8 - 0
> Graviton4 Neoverse-V2 1 / 15 2 / 15 12 / 15
> AMD EPYC 9R45 0 / 14 14 / 14 -
>
> Graviton4 will stall at a higher rate if the wakers are created gradually
> rather than all at once -- the `ramp_ms` parameter spreads their creation
> over that many milliseconds in total, which is what a thread pool growing
> while already under load does. I saw 5/16 stalls on one run.
> AMD still never latched in ~200 runs.
>
> So the state is reachable on more than one vendor's hardware and usually permanent
> once entered. What differs is how often a process falls in.
> Arguably, this flakiness is worse than a deterministic stall for a real production.
>
> ## Prototype of your suggestion
>
> > Not sure what we could do here. One idea might be to block further
> > futex syscalls so they don't acquire a new reference on the existing
> > hash and allow a transition to the new hash more quickly.
>
> The only analogy available to me due to lack of any knowledge is rw-locks. I asked
> Claude to adapt the percpu-rwsem code to futex, and after a few iterations we
> produced a hopefully viable patch. I suppose it implements plus-minus your suggestion.
>
> The results (same box and all the rest of the environment):
>
> - the reproducer: 0/8 stalled at every thread count, against 8/8 on stock at
> 96 and above; 192 threads run at 54 Mops/s instead of 4.
> - futex kselftests: 14 passed, 0 failed, with the mechanism enabled and
> disabled; futex_numa and futex_numa_mpol pass.
> - requeue-PI selftests looped under a continuous pivot storm, to exercise the
> drop_fph handoff against the phase: all rounds passed.
> - stress-ng: 64 futex stressors, and a 40-way futex/clone/mmap mix, 0 failures.
> - PROVE_LOCKING and KASAN builds: clean.
> - measured overhead ~0.6% on a pthread_mutex/pthread_cond benchmark with pinned clock.
>
> Makes sense to note that the patch was applied to v7.2 and all testing was done on that build.
>
> The code could be found at https://gist.github.com/nickitat/0b1cc68deff5fe2b27209b150da6fb00 .
> If you find it useful, I can prepare a proper patch for the kernel tree.
So you create a new state and try to not deadlock. I'm too sure about
the PI bits.
But what about
diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index 51ba5e1257c04..b370b4cf168eb 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -1975,6 +1975,39 @@ static int futex_hash_allocate(unsigned int hash_slots, unsigned int flags)
return 0;
}
+void futex_hash_wait_new(void)
+{
+ struct wait_bit_queue_entry __wbq_entry;
+ struct mm_struct *mm = current->mm;
+ struct wait_queue_head *__wq_head;
+
+ if (!mm->futex.phash.hash_new)
+ return;
+
+ __wq_head = __var_waitqueue(mm);
+ init_wait_var_entry(&__wbq_entry, mm, 0);
+ __wbq_entry.wq_entry.func = woken_wake_bit_function;
+ add_wait_queue(__wq_head, &__wbq_entry.wq_entry);
+
+ /*
+ * add_wait_queue() futex_ref_put()
+ * MB (this) MB (implied)
+ * futex_pivot_pending() wake_up_var()
+ * waitqueue_active()
+ *
+ * Notably, it must not be possible to see
+ * !futex_pivot_pending() && !waitqueue_active().
+ */
+ smp_mb();
+
+ while (!futex_pivot_pending(mm) &&
+ wait_woken(&__wbq_entry.wq_entry, TASK_UNINTERRUPTIBLE,
+ MAX_SCHEDULE_TIMEOUT))
+ /* empty */;
+
+ remove_wait_queue(__wq_head, &__wbq_entry.wq_entry);
+}
+
int futex_hash_allocate_default(void)
{
unsigned int threads, buckets, current_buckets = 0;
diff --git a/kernel/futex/futex.h b/kernel/futex/futex.h
index f00f0863ed441..d538e8e15c336 100644
--- a/kernel/futex/futex.h
+++ b/kernel/futex/futex.h
@@ -246,10 +246,11 @@ struct futex_bucket_ref {
#ifdef CONFIG_FUTEX_PRIVATE_HASH
extern struct futex_private_hash *futex_private_hash(struct mm_struct *mm);
extern void futex_private_hash_put(struct futex_private_hash *fph);
-
+extern void futex_hash_wait_new(void);
#else /* !CONFIG_FUTEX_PRIVATE_HASH */
static inline struct futex_private_hash *futex_private_hash(struct mm_struct *mm) { return NULL; }
static inline void futex_private_hash_put(struct futex_private_hash *fph) { }
+static inline void futex_hash_wait_new(void) { }
#endif
extern struct futex_bucket_ref futex_hash(union futex_key *key);
diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c
index d4483d15d30a5..7ad8a060371c6 100644
--- a/kernel/futex/waitwake.c
+++ b/kernel/futex/waitwake.c
@@ -195,6 +195,8 @@ int futex_wake(u32 __user *uaddr, unsigned int flags, void __user *pop, int nr_w
if ((flags & FLAGS_STRICT) && !nr_wake)
return 0;
+ futex_hash_wait_new();
+
CLASS(hbr, hbr)(&key);
auto hb = hbr.hb;
@@ -283,6 +285,7 @@ int futex_wake_op(u32 __user *uaddr1, unsigned int flags, u32 __user *uaddr2,
int ret, op_ret;
DEFINE_WAKE_Q(wake_q);
+ futex_hash_wait_new();
retry:
ret = get_futex_key(uaddr1, flags, &key1, FUTEX_READ);
if (unlikely(ret != 0))
@@ -744,6 +747,8 @@ int futex_wait(u32 __user *uaddr, unsigned int flags, u32 val, ktime_t *abs_time
struct restart_block *restart;
int ret;
+ futex_hash_wait_new();
+
to = futex_setup_timer(abs_time, &timeout, flags,
current->timer_slack_ns);
It depends on v7.3-rc1 and should keep new waiters blocked until the new
hash is installed. I also added wake path assuming they will continue to
matter what. It might make sense to do this on the wake side only.
Does this do anything?
> Regards.
Sebastian