Re: [RFC PATCH v5 1/2] hazptr: Implement Hazard Pointers

[Paul E. McKenney]

On Tue, Jul 07, 2026 at 02:42:39PM -0700, Paul E. McKenney wrote:
> On Tue, Jul 07, 2026 at 03:40:41PM -0400, Mathieu Desnoyers wrote:
> > On 2026-06-27 13:36, Paul E. McKenney wrote:
> > > On Sat, Jun 27, 2026 at 11:21:28AM -0400, Mathieu Desnoyers wrote:
> > > > On 2026-06-27 08:56, Mathieu Desnoyers wrote:
> > > > > On 2026-06-27 00:14, Paul E. McKenney wrote:
> > > > > > On Fri, Jun 26, 2026 at 11:56:13PM -0400, Mathieu Desnoyers wrote:
> > > > > > > On 2026-06-26 19:06, Paul E. McKenney wrote:
> > > > > [...]
> > > > > > > > Initial runs suggest that hazard pointers cannot be acquired by one
> > > > > > > > task and released by another.  Is that expected behavior, or is my test
> > > > > > > > improperly passing the hazard pointers?  If the latter, we should of
> > > > > > > > course document the proper hazard-pointer-passing mechanism.
> > > > > > > 
> > > > > > > This is unexpected. I've looked at your test code quickly and could
> > > > > > > not figure out what's wrong though. But it's late. How do you observe
> > > > > > > that it fails ?
> > > > > > 
> > > > > > Not an emergency, so your schedule is my schedule.  Especially given
> > > > > > that I am mostly taking this coming week off.  And that it took me such
> > > > > > a long time to get this torture test done.  ;-)
> > > > > > 
> > > > > > I run either scenario (PREEMPT or NOPREEMPT) with:
> > > > > > 
> > > > > > --bootarg "hazptrtorture.onoff_interval=3333
> > > > > > hazptrtorture.stat_interval=15"
> > > > > > 
> > > > > > I get the following on the console, and things go downhill from there.
> > > > > > I get the same thing when I turn off CPU hotplug, for whatever that is
> > > > > > worth.
> > > > > That OOPS is very instructive. I think I found the root cause.
> > > > > 
> > > > > I need to add an API to hazptr so users wishing to explicitly transfer
> > > > > ownership of the hazptr ctx to another thread can do so.
> > > > > 
> > > > > Currently the scheduler hook is responsible for moving the hazptr
> > > > > ctx from per-cpu to the backup slots, which makes it entirely movable
> > > > > afterwards.
> > > > > 
> > > > > But this assumes the ctx is thread-local. Once you explicitly move
> > > > > ownership of that ctx to another thread, you race with the scheduler of
> > > > > the original thread.
> > > > > 
> > > > > Let me see what I can do to fix that.
> > > > > 
> > > > > Thanks a lot for the test-case !
> > > 
> > > Glad it helped?  ;-)
> > > 
> > > > Can you try with the attached POC diff ?
> > > 
> > > This increases the kernel's life significantly, but still fails as
> > > shown below.  This is the NOPREEMPT scenario as follows:
> > > 
> > > tools/testing/selftests/rcutorture/bin/kvm.sh --torture hazptr --allcpus --duration 5m --bootargs "hazptrtorture.onoff_interval=3333 hazptrtorture.nwriters=1 hazptrtorture.stat_interval=15" --trust-make
> > > 
> > > The PREEMPT failure gets similar failures, and the above command runs
> > > both of these scenarios.
> > > 
> > 
> > Re-reviewing kernel/rcu/hazptrtorture.c, I found the following issues:
> > 
> > In my patch provided earlier in this thread, I was missing a "detach
> > from task" call within hazptr_torture_reader_tail, because it calls
> > an IPI to do the hazptr release on a remote CPU. e.g.:
> > 
> > diff --git a/kernel/rcu/hazptrtorture.c b/kernel/rcu/hazptrtorture.c
> > index 311b387d6ae7..ae9ccd89ff87 100644
> > --- a/kernel/rcu/hazptrtorture.c
> > +++ b/kernel/rcu/hazptrtorture.c
> > @@ -411,6 +411,7 @@ hazptr_torture_reader_tail(struct hazptr_pending *hppp, struct torture_random_st
> >         preempt_enable();
> >         if (irq_release && !(torture_random(trsp) % irq_release)) {
> >                 guard(preempt)();
> > +               hazptr_detach_from_task(&hppp->hpp_hc);
> >                 cpu = cpumask_next_wrap(smp_processor_id(), cpu_online_mask);
> >                 smp_call_function_single(cpu, hazptr_torture_release, hppp, 1);
> >         } else {
> > @@ -428,6 +429,7 @@ static void hazptr_torture_defer(struct hazptr_pending *hppp, struct torture_ran
> >         struct llist_head *llhp;
> >         guard(preempt)();
> > +       hazptr_detach_from_task(&hppp->hpp_hc);
> >         cpu = cpumask_next_wrap(cpu, cpu_online_mask);
> >         llhp = per_cpu_ptr(&hazptr_pending, cpu);
> >         llist_add(&hppp->hpp_node, llhp);
> > 
> > Moreover, I think the whole ownership model of hazptr_torture_do_pending
> > is broken wrt hazptr requirements. We would need to "detach from task" each
> > of the hazptr before handing them off to such a remote drain, otherwise
> > the remote drain iteration on all cpus would basically have to hold the
> > runqueue locks for each of those cpus to prevent the scheduler from
> > racing against either hazptr_detach_from_task or hazptr release.
> 
> I applied your earlier patch and this one on top of my -rcu "dev" branch.
> Please check in case I messed something up.
> 
> This provided an additional lease on kernel life, but still resulted in
> issues as shown below.  I agree that it would be good to avoid needing
> to explicitly mark hazard pointers that are to be released elsewhere,
> if that is what you are getting at.  At the same time, it would also be
> severely sub-optimal to acquire multiple CPUs' runqueue locks!

OK, I did check, and all of the hazptr_torture_do_pending() function's
hazard pointers have been passed to hazptr_detach_from_task().

But sometimes a given hazard pointer might be passed to
hazptr_detach_from_task() twice, once just before that llist_add(),
and again if hazptr_torture_reader_tail() decides to release that hazard
pointer via smp_call_function_single().

Do I instead need to be careful to avoid detaching a given hazard pointer
more than once?

							Thanx, Paul

> ------------------------------------------------------------------------
> 
> [   21.169870] Oops: general protection fault, probably for non-canonical address 0xdead000000000110: 0000 [#1] SMP PTI
> [   21.171423] CPU: 6 UID: 0 PID: 130 Comm: hazptr_torture_ Not tainted 7.1.0-rc4-00076-g548241132e40-dirty #1500 PREEMPTLAZY 
> [   21.172859] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
> [   21.173906] RIP: 0010:hazptr_synchronize_overflow_list+0x3e/0x90
> [   21.174670] Code: 08 48 89 c6 48 85 db 74 54 4d 8b 65 10 eb 1b e8 38 2e e0 00 f3 90 4c 89 ef e8 3e 30 e0 00 48 89 c6 49 8b 45 10 4c 39 e0 75 41 <48> 8b 43 10 4c 89 ef 48 39 e8 74 d9 48 83 f8 01 74 d3 e8 0b 2e e0
> [   21.176968] RSP: 0018:ffffa33ac04cfe60 EFLAGS: 00010086
> [   21.177621] RAX: 0000000000001ed7 RBX: dead000000000100 RCX: 0000000000000000
> [   21.178538] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffffa230df398078
> [   21.179462] RBP: ffffffff977a83a0 R08: 0000000000105ea8 R09: 0000000000000006
> [   21.180374] R10: 0000000000000001 R11: ffffa230c1bc2c00 R12: 0000000000001ed7
> [   21.181289] R13: ffffa230df398078 R14: ffffffff977a83a0 R15: ffffa230df398040
> [   21.182208] FS:  0000000000000000(0000) GS:ffffa23147c61000(0000) knlGS:0000000000000000
> [   21.183195] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   21.183881] CR2: 0000000000000000 CR3: 0000000006230000 CR4: 00000000000006f0
> [   21.184763] Call Trace:
> [   21.185113]  <TASK>
> [   21.185395]  hazptr_synchronize+0xd1/0x130
> [   21.185917]  hazptr_torture_writer+0x180/0x3c0
> [   21.186491]  ? __pfx_hazptr_torture_writer+0x10/0x10
> [   21.187121]  kthread+0xd7/0x110
> [   21.187529]  ? __pfx_kthread+0x10/0x10
> [   21.188014]  ret_from_fork+0x1bd/0x220
> [   21.188499]  ? __pfx_kthread+0x10/0x10
> [   21.189137]  ret_from_fork_asm+0x1a/0x30
> [   21.189627]  </TASK>
> [   21.189920] Modules linked in:
> [   21.190340] ---[ end trace 0000000000000000 ]---