Re: [PATCH -rcu/dev 0/4] hazptrtorture: Fix shutdown hang and catch slot owner race

[KunWu Chan]

I ran into both issues independently while developing new shared-scan
hazptr series and its torture-test scenarios.

For patch 1, I reproduced a shutdown hang on 16-CPU ARM64 KVM with
`CONFIG_HAZPTR_ACQUIRE_FORCE_SLOWPATH=y`, `nreaders=16`,
and `defer_modulus=4000`. The readers remained in D state, with
deferred-context stall reports reaching 147 seconds.

My prototype uses the same all-CPU drain operation, but keeps the writer
ahead of the readers in the cleanup order. With that change, the same test
unloaded in seconds. The upstream configuration was harder to trigger; I did
not reproduce the hang in a 3-minute run on the unmodified `origin/dev`
baseline with `defer_modulus=100`.

Paul's concern about `torture_must_stop()` is valid: `fullstop` can make the
drainer exit before the readers have stopped, so its exit-time flush is not
necessarily final. The reader-exit approach looks worth testing against both
the forced-slowpath case and the PREEMPT scenario Andrea reported.

I also saw the patch-3 warning during ARM64 `PROVE_LOCKING` testing,
although I have not reproduced it consistently. The warning count dropped
from 3 to 0 when the relevant fixes were applied together, so I cannot attribute
 that change to patch 3 alone.

These tests independently reproduced both issues in my setup. I'll check the
final cleanup approach against the forced-slowpath scenario as the
series settles.

Thanks,
Kunwu

On Wed, Oct 28, 2026 at 12:38 AM Andrea Parri <parri.andrea@xxxxxxxxx> wrote:
>
> Patch 1 fixes a shutdown hang in hazptrtorture with deferral enabled, as
> in the PREEMPT scenario: hazptr_torture_cleanup() stops the kthread that
> releases deferred hazard pointers before the writer, which can then
> wait in ->sync() forever.  It now stops the readers before that
> kthread, which releases what is left once the readers have stopped, and
> that kthread before the writer.
>
> Patches 2 and 4 make hazptrtorture catch the race fixed by "hazptr:
> Record per-CPU slot owner after claiming the slot" (now on -rcu/dev)
> should it come back, as Paul asked [1] and agreed to [2].  Patch 2
> adds two CONFIG_HAZPTR_DEBUG checks of the recorded slot owner, in
> hazptr_release_debug() and hazptr_note_context_switch().  Patch 4 adds
> the nest_acquire module parameter, which makes readers take a second,
> nested hazard pointer that goes through __hazptr_acquire() while the
> first is in slot 0.
>
> Patch 3 fixes a false positive of an existing CONFIG_HAZPTR_DEBUG
> check, which the nesting of patch 4 exposes with CPU hotplug: a hazard
> pointer that __hazptr_acquire() puts directly in the backup slot warns
> if the task migrates before releasing it.  It comes before patch 4,
> which would otherwise make runs with CPU hotplug warn.
>
> The series applies on -rcu/dev.  30-minute hazptrtorture runs of the
> PREEMPT and NOPREEMPT scenarios pass on x86_64 with 8 CPUs, as do a
> 30-minute NOPREEMPT run with torture_type=hazptr and three 10-minute
> PREEMPT runs: one with irq_acquire=100 and nest_acquire=1, one with
> CONFIG_HAZPTR_DEBUG=n, and one with torture_type=hazptr-stack.  The
> 30-minute PREEMPT run ends with as many hazard pointers released from
> the pending lists as were deferred (reldefer: 650385 relundefer
> 650385).  With kthread_do_pending_ms=10000, two 3-minute PREEMPT runs
> shut down in 100 and 130ms, well within one period.
>
> Without patch 4, hazptrtorture passes even without the slot-owner fix:
> its readers hold one hazard pointer each, so they reach the slow path
> only when the ptr_eq() check fails or the pointer is NULL.  With
> patches 2 and 4 but without the fix, PREEMPT and NOPREEMPT report the
> race 6.3 and 6.0 seconds after boot, and with nest_acquire=0 both run
> 10 minutes without a report.
>
> The nesting costs about a third of the outer reader iterations at the
> default nest_acquire=2.  If that is too much for every run, it could
> default to zero and be enabled from the scenario files instead.
>
> Patches 2 and 4 stand in for the KUnit reproducer mentioned in the RFC
> notes of the fix, which needs a test-only hook in
> __hazptr_try_acquire().  The checks can miss one case that the KUnit
> test covers: an interrupt handler that acquires and releases the slot
> instead of detaching it, then a context switch before the task's
> release, when no other hazard pointer claims the slot in between.
> hazptrtorture does not produce that case either, as
> hazptr_torture_acquire() always detaches.
>
> Patch 2's release check also warns about a second hazptr_release() of
> a ctx whose per-CPU slot another hazard pointer has claimed since.  The
> hazptr_release() kernel-doc permits that second release, but it clears
> the other hazard pointer, and a second release of a backup slot oopses
> even without this series.  Nothing in-tree releases a hazard pointer
> twice.  I will follow up separately on the kernel-doc.
>
> An LLM found the hang fixed by patch 1, the crash shown in patch 2 and
> the false positive fixed by patch 3.  It also helped write all four
> patches and their changelogs, and ran the tests above.
>
> ---
> Changes since RFC:
> - Patch 1: stop the readers, then do_pending, then the writer, and
>   release all deferred hazard pointers after torture_kthread_stopping(),
>   instead of keeping do_pending in its loop until it is stopped (Paul).
>   Paul suggested changing the "while (!torture_must_stop())" condition
>   to "while (!kthread_should_stop())"; this keeps it, for the reason in
>   [3].  The changelog also says when kthread_stop() can still delay the
>   exit of do_pending by a full kthread_do_pending_ms sleep (Paul).
>   Bradley's Acked-by is not carried over, as the patch changed.
> - Post patch 1 in a series (Bradley), now with patches 2 and 4, so
>   that hazptrtorture catches the race fixed by the slot-owner RFC patch
>   (Paul).
> - Patch 3: new, for the false positive that patch 4 exposes.
> - Do not repost the slot-owner RFC patch, which Paul has queued.
> - Drop the Assisted-by tags, following mainline commit 1e035297ffda
>   ("docs: remove Assisted-by references"), and describe the LLM use
>   in the changelogs and above instead.
>
> [1] https://lore.kernel.org/r/fa16d2b1-594e-494b-8cfc-9b8c13f2c223@paulmck-laptop/
> [2] https://lore.kernel.org/r/e0145c14-9d6a-4c91-816c-57dabd98879b@paulmck-laptop/
> [3] https://lore.kernel.org/r/20261023120755.2463384-1-parri.andrea@xxxxxxxxx/
>
> RFC patch 1: https://lore.kernel.org/r/20261019162005.12888-1-parri.andrea@xxxxxxxxx/
> RFC slot-owner fix: https://lore.kernel.org/r/20261019161852.12783-1-parri.andrea@xxxxxxxxx/
>
> Andrea Parri (4):
>   hazptrtorture: Release all deferred hazard pointers before stopping
>   hazptr: Check per-CPU slot owner with CONFIG_HAZPTR_DEBUG
>   hazptr: Do not warn when a backup slot is released on another CPU
>   hazptrtorture: Nest a second hazard pointer in readers
>
>  .../admin-guide/kernel-parameters.txt         |  6 +++
>  include/linux/hazptr.h                        | 13 ++++++
>  kernel/hazptr.c                               |  4 ++
>  kernel/rcu/hazptrtorture.c                    | 40 ++++++++++++++++++-
>  4 files changed, 61 insertions(+), 2 deletions(-)
>
>
> base-commit: 633f91e3408cc65a5237752a3265b031c389e43e
> --
> 2.53.0
>