Re: [RFC PATCH v5 1/2] hazptr: Implement Hazard Pointers
From: Paul E. McKenney
Date: Sat Jun 27 2026 - 13:37:13 EST
On Sat, Jun 27, 2026 at 11:21:28AM -0400, Mathieu Desnoyers wrote:
> On 2026-06-27 08:56, Mathieu Desnoyers wrote:
> > On 2026-06-27 00:14, Paul E. McKenney wrote:
> > > On Fri, Jun 26, 2026 at 11:56:13PM -0400, Mathieu Desnoyers wrote:
> > > > On 2026-06-26 19:06, Paul E. McKenney wrote:
> > [...]
> > > > > Initial runs suggest that hazard pointers cannot be acquired by one
> > > > > task and released by another. Is that expected behavior, or is my test
> > > > > improperly passing the hazard pointers? If the latter, we should of
> > > > > course document the proper hazard-pointer-passing mechanism.
> > > >
> > > > This is unexpected. I've looked at your test code quickly and could
> > > > not figure out what's wrong though. But it's late. How do you observe
> > > > that it fails ?
> > >
> > > Not an emergency, so your schedule is my schedule. Especially given
> > > that I am mostly taking this coming week off. And that it took me such
> > > a long time to get this torture test done. ;-)
> > >
> > > I run either scenario (PREEMPT or NOPREEMPT) with:
> > >
> > > --bootarg "hazptrtorture.onoff_interval=3333
> > > hazptrtorture.stat_interval=15"
> > >
> > > I get the following on the console, and things go downhill from there.
> > > I get the same thing when I turn off CPU hotplug, for whatever that is
> > > worth.
> > That OOPS is very instructive. I think I found the root cause.
> >
> > I need to add an API to hazptr so users wishing to explicitly transfer
> > ownership of the hazptr ctx to another thread can do so.
> >
> > Currently the scheduler hook is responsible for moving the hazptr
> > ctx from per-cpu to the backup slots, which makes it entirely movable
> > afterwards.
> >
> > But this assumes the ctx is thread-local. Once you explicitly move
> > ownership of that ctx to another thread, you race with the scheduler of
> > the original thread.
> >
> > Let me see what I can do to fix that.
> >
> > Thanks a lot for the test-case !
Glad it helped? ;-)
> Can you try with the attached POC diff ?
This increases the kernel's life significantly, but still fails as
shown below. This is the NOPREEMPT scenario as follows:
tools/testing/selftests/rcutorture/bin/kvm.sh --torture hazptr --allcpus --duration 5m --bootargs "hazptrtorture.onoff_interval=3333 hazptrtorture.nwriters=1 hazptrtorture.stat_interval=15" --trust-make
The PREEMPT failure gets similar failures, and the above command runs
both of these scenarios.
Thanx, Paul
> Thanks,
>
> Mathieu
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
> diff --git a/include/linux/hazptr.h b/include/linux/hazptr.h
> index 461f481a480b..d96414575739 100644
> --- a/include/linux/hazptr.h
> +++ b/include/linux/hazptr.h
> @@ -98,6 +98,41 @@ bool hazptr_slot_is_backup(struct hazptr_ctx *ctx, struct hazptr_slot *slot)
> return slot == &ctx->backup_slot.slot;
> }
>
> +/* Internal helper. */
> +static inline
> +void hazptr_promote_to_backup_slot(struct hazptr_ctx *ctx, struct hazptr_slot *slot)
> +{
> + struct hazptr_slot *backup_slot;
> +
> + backup_slot = hazptr_chain_backup_slot(ctx);
> + /*
> + * Move hazard pointer from the per-CPU slot to the
> + * backup slot. This requires hazard pointer
> + * synchronize to iterate on per-CPU slots with
> + * load-acquire before iterating on the overflow list.
> + */
> + WRITE_ONCE(backup_slot->addr, slot->addr);
> + /*
> + * store-release orders store to backup slot addr before
> + * store to per-CPU slot addr.
> + */
> + smp_store_release(&slot->addr, NULL);
> + /* Use the backup slot for context. */
> + ctx->slot = backup_slot;
> +}
> +
> +static inline
> +void hazptr_detach_from_task(struct hazptr_ctx *ctx)
> +{
> + struct hazptr_slot *slot;
> +
> + guard(preempt)();
> + slot = ctx->slot;
> + if (unlikely(hazptr_slot_is_backup(ctx, slot)))
> + return;
> + hazptr_promote_to_backup_slot(ctx, slot);
> +}
> +
> static inline
> void hazptr_note_context_switch(void)
> {
> @@ -106,27 +141,11 @@ void hazptr_note_context_switch(void)
>
> for (idx = 0; idx < NR_HAZPTR_PERCPU_SLOTS; idx++) {
> struct hazptr_slot_item *item = &percpu_slots->items[idx];
> - struct hazptr_slot *slot = &item->slot, *backup_slot;
> - struct hazptr_ctx *ctx;
> + struct hazptr_slot *slot = &item->slot;
>
> if (!slot->addr)
> continue;
> - ctx = item->ctx.ctx;
> - backup_slot = hazptr_chain_backup_slot(ctx);
> - /*
> - * Move hazard pointer from the per-CPU slot to the
> - * backup slot. This requires hazard pointer
> - * synchronize to iterate on per-CPU slots with
> - * load-acquire before iterating on the overflow list.
> - */
> - WRITE_ONCE(backup_slot->addr, slot->addr);
> - /*
> - * store-release orders store to backup slot addr before
> - * store to per-CPU slot addr.
> - */
> - smp_store_release(&slot->addr, NULL);
> - /* Use the backup slot for context. */
> - ctx->slot = backup_slot;
> + hazptr_promote_to_backup_slot(item->ctx.ctx, slot);
> }
> }
>
> diff --git a/kernel/rcu/hazptrtorture.c b/kernel/rcu/hazptrtorture.c
> index 311b387d6ae7..93b383887fc5 100644
> --- a/kernel/rcu/hazptrtorture.c
> +++ b/kernel/rcu/hazptrtorture.c
> @@ -427,6 +427,7 @@ static void hazptr_torture_defer(struct hazptr_pending *hppp, struct torture_ran
> int cpu = torture_random(trsp) % nr_cpu_ids;
> struct llist_head *llhp;
>
> + hazptr_detach_from_task(&hppp->hpp_hc);
> guard(preempt)();
> cpu = cpumask_next_wrap(cpu, cpu_online_mask);
> llhp = per_cpu_ptr(&hazptr_pending, cpu);
------------------------------------------------------------------------
[ 16.477672] hazptr-stack-torture: rtc: 00000000927f2a0a ver: 273 HTWS_SYNC tfle: 0 rta: 274 rtaf: 0 rtf: 264 onoff: 1/1:4/4 7,7:10,14 7:44 (HZ=1000) acq: 268127149 rel: 268125688 acqirq: 15502 relirq: 16773 reldefer: 0 relundefer 0
[ 16.480858] hazptr-stack-torture: Reader Pipe: 268140258 1798 0 0 0 0 0 0 0 0 0
[ 16.482093] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 16.483149] hazptr-stack-torture: Free-Block Circulation: 273 272 271 270 269 268 267 266 265 264 0
[ 18.133721] clocksource: Watchdog remote CPU 8 read timed out
[ 18.652293] Oops: general protection fault, probably for non-canonical address 0xdead000000000110: 0000 [#1] SMP PTI
[ 18.653967] CPU: 13 UID: 0 PID: 132 Comm: hazptr_torture_ Not tainted 7.1.0-rc4-00071-g136a43a847a7-dirty #1494 PREEMPTLAZY
[ 18.655726] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
[ 18.656999] RIP: 0010:hazptr_synchronize_overflow_list+0x3e/0x90
[ 18.657968] Code: 08 48 89 c6 48 85 db 74 54 4d 8b 65 10 eb 1b e8 78 2d e0 00 f3 90 4c 89 ef e8 7e 2f e0 00 48 89 c6 49 8b 45 10 4c 39 e0 75 41 <48> 8b 43 10 4c 89 ef 48 39 e8 74 d9 48 83 f8 01 74 d3 e8 4b 2d e0
[ 18.660803] RSP: 0000:ffffb255004dfe60 EFLAGS: 00010086
[ 18.661592] RAX: 00000000000013b2 RBX: dead000000000100 RCX: 0000000000000003
[ 18.662706] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff9cdc5f3d8060
[ 18.663788] RBP: ffffffffa21a7f00 R08: 000000000006d924 R09: 000000000000000d
[ 18.664864] R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000013b2
[ 18.665975] R13: ffff9cdc5f3d8060 R14: ffffffffa21a7f00 R15: ffff9cdc5f3d8040
[ 18.667095] FS: 0000000000000000(0000) GS:ffff9cdcbd421000(0000) knlGS:0000000000000000
[ 18.668332] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 18.669228] CR2: 0000000000000000 CR3: 0000000018030000 CR4: 00000000000006f0
[ 18.670328] Call Trace:
[ 18.670735] <TASK>
[ 18.671077] hazptr_synchronize+0xd1/0x130
[ 18.671744] hazptr_torture_writer+0x180/0x3c0
[ 18.672449] ? __pfx_hazptr_torture_writer+0x10/0x10
[ 18.673228] kthread+0xd7/0x110
[ 18.673719] ? __pfx_kthread+0x10/0x10
[ 18.674281] ret_from_fork+0x1bd/0x220
[ 18.674867] ? __pfx_kthread+0x10/0x10
[ 18.675462] ret_from_fork_asm+0x1a/0x30
[ 18.676058] </TASK>
[ 18.676424] Modules linked in:
[ 18.676900] ---[ end trace 0000000000000000 ]---
[ 18.677596] RIP: 0010:hazptr_synchronize_overflow_list+0x3e/0x90
[ 18.678504] Code: 08 48 89 c6 48 85 db 74 54 4d 8b 65 10 eb 1b e8 78 2d e0 00 f3 90 4c 89 ef e8 7e 2f e0 00 48 89 c6 49 8b 45 10 4c 39 e0 75 41 <48> 8b 43 10 4c 89 ef 48 39 e8 74 d9 48 83 f8 01 74 d3 e8 4b 2d e0
[ 18.681215] RSP: 0000:ffffb255004dfe60 EFLAGS: 00010086
[ 18.681816] RAX: 00000000000013b2 RBX: dead000000000100 RCX: 0000000000000003
[ 18.682950] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff9cdc5f3d8060
[ 18.683736] RBP: ffffffffa21a7f00 R08: 000000000006d924 R09: 000000000000000d
[ 18.684812] R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000013b2
[ 18.685886] R13: ffff9cdc5f3d8060 R14: ffffffffa21a7f00 R15: ffff9cdc5f3d8040
[ 18.686652] FS: 0000000000000000(0000) GS:ffff9cdcbd421000(0000) knlGS:0000000000000000
[ 18.687509] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 18.688134] CR2: 0000000000000000 CR3: 0000000018030000 CR4: 00000000000006f0
[ 18.689205] note: hazptr_torture_[132] exited with irqs disabled
[ 18.689861] note: hazptr_torture_[132] exited with preempt_count 1
[ 18.690519] hazptr_torture_ (132) used greatest stack depth: 13704 bytes left
[ 18.695774] smpboot: CPU 2 is now offline
[ 22.039232] smpboot: CPU 5 is now offline
[ 25.376816] smpboot: Booting Node 0 Processor 2 APIC 0x2
[ 31.837655] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 2/2:6/6 7,7:10,50 14:104 (HZ=1000) acq: 727372071 rel: 727368302 acqirq: 42089 relirq: 45682 reldefer: 0 relundefer 0
[ 31.840069] hazptr-stack-torture: Reader Pipe: 727411641 2343 0 0 0 0 0 0 0 0 0
[ 31.840922] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 31.841674] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 32.057111] smpboot: CPU 7 is now offline
[ 35.401073] smpboot: CPU 15 is now offline
[ 38.741634] smpboot: Booting Node 0 Processor 1 APIC 0x1
[ 42.089276] smpboot: CPU 3 is now offline
[ 45.426763] smpboot: Booting Node 0 Processor 15 APIC 0xf
[ 47.199600] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 4/4:9/9 7,10:10,50 31:135 (HZ=1000) acq: 977618948 rel: 977613883 acqirq: 56384 relirq: 61259 reldefer: 0 relundefer 0
[ 47.203087] hazptr-stack-torture: Reader Pipe: 977672190 2343 0 0 0 0 0 0 0 0 0
[ 47.204315] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 47.205430] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 47.206996] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 48.781598] smpboot: CPU 11 is now offline
[ 52.125175] smpboot: CPU 1 is now offline
[ 55.468194] smpboot: CPU 8 is now offline
[ 58.816598] smpboot: CPU 9 is now offline
[ 62.160238] smpboot: CPU 10 is now offline
[ 62.208650] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 4/4:14/14 7,10:10,50 31:197 (HZ=1000) acq: 1336717896 rel: 1336710172 acqirq: 76179 relirq: 83727 reldefer: 0 relundefer 0
[ 62.212039] hazptr-stack-torture: Reader Pipe: 1336791556 2343 0 0 0 0 0 0 0 0 0
[ 62.213106] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 62.214028] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 62.215326] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 65.504487] smpboot: CPU 4 is now offline
[ 72.176750] smpboot: Booting Node 0 Processor 5 APIC 0x5
[ 75.517839] smpboot: Booting Node 0 Processor 11 APIC 0xb
[ 77.406600] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 6/6:15/15 7,10:10,50 47:209 (HZ=1000) acq: 1513030623 rel: 1513021030 acqirq: 85225 relirq: 94635 reldefer: 0 relundefer 0
[ 77.409704] hazptr-stack-torture: Reader Pipe: 1513113120 2343 0 0 0 0 0 0 0 0 0
[ 77.410838] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 77.411832] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 77.413108] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 78.860630] smpboot: Booting Node 0 Processor 10 APIC 0xa
[ 82.209063] smpboot: CPU 10 is now offline
[ 85.546766] smpboot: Booting Node 0 Processor 3 APIC 0x3
[ 88.899717] smpboot: CPU 13 is now offline
[ 92.237759] smpboot: Booting Node 0 Processor 1 APIC 0x1
[ 92.765706] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 9/9:17/17 7,10:10,50 70:235 (HZ=1000) acq: 1815314543 rel: 1815302193 acqirq: 101451 relirq: 113625 reldefer: 0 relundefer 0
[ 92.769757] hazptr-stack-torture: Reader Pipe: 1815413475 2343 0 0 0 0 0 0 0 0 0
[ 92.771206] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 92.772480] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 92.774229] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 95.577713] smpboot: Booting Node 0 Processor 6 APIC 0x6
[ 98.920645] smpboot: Booting Node 0 Processor 4 APIC 0x4
[ 102.273055] smpboot: CPU 3 is now offline
[ 105.616678] smpboot: CPU 11 is now offline
[ 108.126606] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 11/11:19/19 7,15:10,50 93:256 (HZ=1000) acq: 2100515525 rel: 2100501173 acqirq: 117324 relirq: 131491 reldefer: 0 relundefer 0
[ 108.129407] hazptr-stack-torture: Reader Pipe: 2100629725 2343 0 0 0 0 0 0 0 0 0
[ 108.130408] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 108.131460] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 108.132682] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 108.964599] smpboot: CPU 4 is now offline
[ 112.308116] smpboot: CPU 6 is now offline
[ 115.645725] smpboot: Booting Node 0 Processor 10 APIC 0xa
[ 118.987152] smpboot: Booting Node 0 Processor 7 APIC 0x7
[ 122.334115] smpboot: CPU 5 is now offline
[ 123.485674] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 13/13:22/22 7,15:10,50 108:289 (HZ=1000) acq: 2385152588 rel: 2385135762 acqirq: 132665 relirq: 149315 reldefer: 0 relundefer 0
[ 123.488311] hazptr-stack-torture: Reader Pipe: 2385282734 2343 0 0 0 0 0 0 0 0 0
[ 123.489243] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 123.490188] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 123.491340] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 125.677361] smpboot: CPU 12 is now offline
[ 129.017162] smpboot: Booting Node 0 Processor 4 APIC 0x4
[ 132.358758] smpboot: Booting Node 0 Processor 5 APIC 0x5
[ 135.705206] smpboot: CPU 7 is now offline
[ 138.846403] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 15/15:24/24 7,15:10,50 125:309 (HZ=1000) acq: 2601900628 rel: 2601881685 acqirq: 144045 relirq: 162805 reldefer: 0 relundefer 0
[ 138.849806] hazptr-stack-torture: Reader Pipe: 2602041602 2343 0 0 0 0 0 0 0 0 0
[ 138.850887] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 138.851839] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 138.853198] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 139.044658] smpboot: Booting Node 0 Processor 3 APIC 0x3
[ 145.721755] smpboot: Booting Node 0 Processor 6 APIC 0x6
[ 149.062641] smpboot: Booting Node 0 Processor 12 APIC 0xc
[ 152.404795] smpboot: Booting Node 0 Processor 13 APIC 0xd
[ 154.205637] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 19/19:24/24 7,15:10,50 158:309 (HZ=1000) acq: 2891489001 rel: 2891468129 acqirq: 160165 relirq: 180861 reldefer: 0 relundefer 0
[ 154.208700] hazptr-stack-torture: Reader Pipe: 2891646647 2343 0 0 0 0 0 0 0 0 0
[ 154.209779] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 154.210707] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 154.211966] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 155.745783] smpboot: Booting Node 0 Processor 14 APIC 0xe
[ 159.090655] smpboot: Booting Node 0 Processor 8 APIC 0x8
[ 162.439216] smpboot: CPU 4 is now offline
[ 165.782180] smpboot: CPU 8 is now offline
[ 169.133489] smpboot: CPU 14 is now offline
[ 169.567402] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 21/21:27/27 7,15:10,50 178:346 (HZ=1000) acq: 3279646888 rel: 3279623874 acqirq: 182381 relirq: 205207 reldefer: 0 relundefer 0
[ 169.571174] hazptr-stack-torture: Reader Pipe: 3279826075 2343 0 0 0 0 0 0 0 0 0
[ 169.572554] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 169.573771] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 169.575421] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 172.477225] smpboot: CPU 13 is now offline
[ 175.815697] smpboot: Booting Node 0 Processor 13 APIC 0xd
[ 179.158957] smpboot: Booting Node 0 Processor 7 APIC 0x7
[ 182.508261] smpboot: CPU 1 is now offline
[ 184.605659] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 23/23:29/29 7,15:10,50 197:366 (HZ=1000) acq: 3558237081 rel: 3558212141 acqirq: 197989 relirq: 222753 reldefer: 0 relundefer 0
[ 184.608653] hazptr-stack-torture: Reader Pipe: 3558432551 2343 0 0 0 0 0 0 0 0 0
[ 184.609616] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 184.610722] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 184.612241] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 185.852919] smpboot: CPU 5 is now offline
[ 189.195273] smpboot: Booting Node 0 Processor 9 APIC 0x9
[ 192.544150] smpboot: CPU 7 is now offline
[ 195.887234] smpboot: CPU 6 is now offline
[ 199.226674] smpboot: Booting Node 0 Processor 14 APIC 0xe
[ 199.774416] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 25/25:32/32 7,15:10,50 221:398 (HZ=1000) acq: 3878923397 rel: 3878895983 acqirq: 215530 relirq: 242758 reldefer: 0 relundefer 0
[ 199.778265] hazptr-stack-torture: Reader Pipe: 3879136232 2343 0 0 0 0 0 0 0 0 0
[ 199.779338] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 199.780277] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 199.781617] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 202.570760] smpboot: Booting Node 0 Processor 7 APIC 0x7
[ 205.917183] smpboot: CPU 10 is now offline
[ 209.268602] smpboot: CPU 13 is now offline
[ 212.606780] smpboot: Booting Node 0 Processor 10 APIC 0xa
[ 215.133662] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 27/27:34/34 7,15:10,50 235:425 (HZ=1000) acq: 4111862347 rel: 4111833331 acqirq: 228399 relirq: 257239 reldefer: 0 relundefer 0
[ 215.136064] hazptr-stack-torture: Reader Pipe: 4112088227 2343 0 0 0 0 0 0 0 0 0
[ 215.136819] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 215.137467] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 215.138386] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 215.947665] smpboot: Booting Node 0 Processor 8 APIC 0x8
[ 219.301118] smpboot: CPU 14 is now offline
[ 222.645272] smpboot: CPU 7 is now offline
[ 225.988823] smpboot: CPU 8 is now offline
[ 229.337095] smpboot: CPU 10 is now offline
[ 230.494608] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 28/28:38/38 7,15:10,50 242:477 (HZ=1000) acq: 4451936851 rel: 4451905359 acqirq: 247133 relirq: 278442 reldefer: 0 relundefer 0
[ 230.499098] hazptr-stack-torture: Reader Pipe: 4452181088 2343 0 0 0 0 0 0 0 0 0
[ 230.500667] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 230.502040] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 230.504007] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 232.675700] smpboot: Booting Node 0 Processor 6 APIC 0x6
[ 236.021288] smpboot: CPU 2 is now offline
[ 239.358045] smpboot: Booting Node 0 Processor 8 APIC 0x8
[ 242.704108] smpboot: CPU 15 is now offline
[ 245.853640] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 30/30:40/40 7,15:10,50 257:497 (HZ=1000) acq: 4631094358 rel: 4631061240 acqirq: 256471 relirq: 289413 reldefer: 0 relundefer 0
[ 245.857674] hazptr-stack-torture: Reader Pipe: 4631348310 2343 0 0 0 0 0 0 0 0 0
[ 245.859076] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 245.860302] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 245.862004] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 246.042747] smpboot: Booting Node 0 Processor 15 APIC 0xf
[ 252.717782] smpboot: Booting Node 0 Processor 10 APIC 0xa
[ 256.065089] smpboot: CPU 6 is now offline
[ 259.405180] smpboot: Booting Node 0 Processor 11 APIC 0xb
[ 261.213631] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 33/33:41/41 7,15:10,50 282:507 (HZ=1000) acq: 4971990752 rel: 4971954460 acqirq: 274559 relirq: 310673 reldefer: 0 relundefer 0
[ 261.216128] hazptr-stack-torture: Reader Pipe: 4972262790 2343 0 0 0 0 0 0 0 0 0
[ 261.217093] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 261.217860] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 261.218909] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 266.085175] smpboot: CPU 15 is now offline
[ 269.424203] smpboot: Booting Node 0 Processor 15 APIC 0xf
[ 272.775068] smpboot: CPU 8 is now offline
[ 276.120771] smpboot: CPU 3 is now offline
[ 276.223722] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 34/34:44/44 7,15:10,50 290:542 (HZ=1000) acq: 5139645571 rel: 5139607698 acqirq: 283686 relirq: 321376 reldefer: 0 relundefer 0
[ 276.226721] hazptr-stack-torture: Reader Pipe: 5139926527 2343 0 0 0 0 0 0 0 0 0
[ 276.227780] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 276.228708] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 276.229992] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 279.461630] smpboot: Booting Node 0 Processor 6 APIC 0x6
[ 282.809228] smpboot: CPU 11 is now offline
[ 286.153098] smpboot: CPU 9 is now offline
[ 289.491678] smpboot: Booting Node 0 Processor 7 APIC 0x7
[ 291.421660] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 36/36:46/46 7,15:10,50 311:563 (HZ=1000) acq: 5422489753 rel: 5422448772 acqirq: 298192 relirq: 338997 reldefer: 0 relundefer 0
[ 291.424169] hazptr-stack-torture: Reader Pipe: 5422785426 2343 0 0 0 0 0 0 0 0 0
[ 291.424922] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 291.425589] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 291.426514] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 292.840168] smpboot: CPU 15 is now offline
[ 296.179186] smpboot: Booting Node 0 Processor 9 APIC 0x9
[ 299.521672] smpboot: Booting Node 0 Processor 11 APIC 0xb
[ 301.097957] hazptr-stack-torture: torture_shutdown task shutting down system
[ 301.098815] hazptr-stack-torture: Stopping torture_shuffle task
[ 301.099526] hazptr-stack-torture: torture_shuffle is stopping
[ 301.100361] hazptr-stack-torture: Stopping torture_stutter task
[ 301.101078] hazptr-stack-torture: torture_stutter is stopping
[ 301.101861] hazptr-stack-torture: Stopping torture_onoff task
[ 301.102557] hazptr-stack-torture: torture_onoff is stopping
[ 301.108685] smpboot: Booting Node 0 Processor 1 APIC 0x1
[ 301.112089] smpboot: Booting Node 0 Processor 2 APIC 0x2
[ 301.115250] smpboot: Booting Node 0 Processor 3 APIC 0x3
[ 301.118468] smpboot: Booting Node 0 Processor 4 APIC 0x4
[ 301.121784] smpboot: Booting Node 0 Processor 5 APIC 0x5
[ 301.125039] smpboot: Booting Node 0 Processor 8 APIC 0x8
[ 301.128491] smpboot: Booting Node 0 Processor 13 APIC 0xd
[ 301.132060] smpboot: Booting Node 0 Processor 14 APIC 0xe
[ 301.135836] smpboot: Booting Node 0 Processor 15 APIC 0xf
[ 301.139564] hazptr-stack-torture: Stopping hazptr_torture_writer task
[ 301.140338] ------------[ cut here ]------------
[ 301.140885] refcount_t: addition on 0; use-after-free.
[ 301.141484] WARNING: lib/refcount.c:25 at refcount_warn_saturate+0x48/0x90, CPU#9: torture_shutdow/137
[ 301.142559] Modules linked in:
[ 301.142934] CPU: 9 UID: 0 PID: 137 Comm: torture_shutdow Tainted: G D 7.1.0-rc4-00071-g136a43a847a7-dirty #1494 PREEMPTLAZY
[ 301.144376] Tainted: [D]=DIE
[ 301.144727] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
[ 301.145651] RIP: 0010:refcount_warn_saturate+0x48/0x90
[ 301.146257] Code: ce 02 4d 01 67 48 0f b9 3a c3 cc cc cc cc 85 f6 74 44 48 8d 3d c9 02 4d 01 67 48 0f b9 3a e9 4f aa 9d 00 48 8d 3d c8 02 4d 01 <67> 48 0f b9 3a e9 3e aa 9d 00 48 8d 3d c7 02 4d 01 67 48 0f b9 3a
[ 301.148379] RSP: 0018:ffffb25500507e90 EFLAGS: 00010246
[ 301.148986] RAX: 0000000000000000 RBX: ffff9cdc42221d80 RCX: 3fffffffffffdfff
[ 301.149811] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffffffffa19a9a20
[ 301.150640] RBP: 431bde82d7b634db R08: ffffffffa1933da8 R09: 3fffffffffffdfff
[ 301.151452] R10: ffffffffa1853dc0 R11: 00000000000007ed R12: ffff9cdc42221da8
[ 301.152273] R13: ffffb25500013d70 R14: 0000000000000000 R15: ffff9cdc420cc9c0
[ 301.153090] FS: 0000000000000000(0000) GS:ffff9cdcbd321000(0000) knlGS:0000000000000000
[ 301.154008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 301.154669] CR2: 0000000000000000 CR3: 0000000018030000 CR4: 00000000000006f0
[ 301.155482] Call Trace:
[ 301.155789] <TASK>
[ 301.156052] kthread_stop+0xeb/0xf0
[ 301.156467] _torture_stop_kthread+0x24/0x90
[ 301.156972] hazptr_torture_cleanup+0x59/0x130
[ 301.157496] torture_shutdown+0xdf/0x2a0
[ 301.157957] ? __pfx_torture_shutdown+0x10/0x10
[ 301.158487] kthread+0xd7/0x110
[ 301.158878] ? __pfx_kthread+0x10/0x10
[ 301.159323] ret_from_fork+0x1bd/0x220
[ 301.159765] ? __pfx_kthread+0x10/0x10
[ 301.160208] ret_from_fork_asm+0x1a/0x30
[ 301.160670] </TASK>
[ 301.160934] ---[ end trace 0000000000000000 ]---
[ 301.161485] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 301.162275] #PF: supervisor write access in kernel mode
[ 301.162868] #PF: error_code(0x0002) - not-present page
[ 301.163457] PGD 0 P4D 0
[ 301.163761] Oops: Oops: 0002 [#2] SMP PTI
[ 301.164229] CPU: 9 UID: 0 PID: 137 Comm: torture_shutdow Tainted: G D W 7.1.0-rc4-00071-g136a43a847a7-dirty #1494 PREEMPTLAZY
[ 301.165589] Tainted: [D]=DIE, [W]=WARN
[ 301.165966] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.15.0-1 04/01/2014
[ 301.166754] RIP: 0010:kthread_stop+0x3b/0xf0
[ 301.167192] Code: 55 53 48 89 fb f0 0f c1 47 28 85 c0 0f 84 bb 00 00 00 8d 50 01 09 c2 78 79 f6 43 2e 20 0f 84 86 00 00 00 48 8b ab 60 06 00 00 <f0> 80 4d 00 02 48 89 df e8 28 ff ff ff f0 80 0b 04 48 89 df e8 3c
[ 301.168991] RSP: 0018:ffffb25500507e98 EFLAGS: 00010202
[ 301.169512] RAX: 0000000000000000 RBX: ffff9cdc42221d80 RCX: 3fffffffffffdfff
[ 301.170235] RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffffffffa19a9a20
[ 301.170927] RBP: 0000000000000000 R08: ffffffffa1933da8 R09: 3fffffffffffdfff
[ 301.171629] R10: ffffffffa1853dc0 R11: 00000000000007ed R12: ffff9cdc42221da8
[ 301.172331] R13: ffffb25500013d70 R14: 0000000000000000 R15: ffff9cdc420cc9c0
[ 301.173037] FS: 0000000000000000(0000) GS:ffff9cdcbd321000(0000) knlGS:0000000000000000
[ 301.173819] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 301.174389] CR2: 0000000000000000 CR3: 0000000018030000 CR4: 00000000000006f0
[ 301.175090] Call Trace:
[ 301.175343] <TASK>
[ 301.175564] _torture_stop_kthread+0x24/0x90
[ 301.175998] hazptr_torture_cleanup+0x59/0x130
[ 301.176445] torture_shutdown+0xdf/0x2a0
[ 301.176840] ? __pfx_torture_shutdown+0x10/0x10
[ 301.177299] kthread+0xd7/0x110
[ 301.177621] ? __pfx_kthread+0x10/0x10
[ 301.178003] ret_from_fork+0x1bd/0x220
[ 301.178382] ? __pfx_kthread+0x10/0x10
[ 301.178759] ret_from_fork_asm+0x1a/0x30
[ 301.179157] </TASK>
[ 301.179405] Modules linked in:
[ 301.179718] CR2: 0000000000000000
[ 301.180060] ---[ end trace 0000000000000000 ]---
[ 301.180517] RIP: 0010:hazptr_synchronize_overflow_list+0x3e/0x90
[ 301.181114] Code: 08 48 89 c6 48 85 db 74 54 4d 8b 65 10 eb 1b e8 78 2d e0 00 f3 90 4c 89 ef e8 7e 2f e0 00 48 89 c6 49 8b 45 10 4c 39 e0 75 41 <48> 8b 43 10 4c 89 ef 48 39 e8 74 d9 48 83 f8 01 74 d3 e8 4b 2d e0
[ 301.182909] RSP: 0000:ffffb255004dfe60 EFLAGS: 00010086
[ 301.183430] RAX: 00000000000013b2 RBX: dead000000000100 RCX: 0000000000000003
[ 301.184136] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff9cdc5f3d8060
[ 301.184831] RBP: ffffffffa21a7f00 R08: 000000000006d924 R09: 000000000000000d
[ 301.185533] R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000013b2
[ 301.186249] R13: ffff9cdc5f3d8060 R14: ffffffffa21a7f00 R15: ffff9cdc5f3d8040
[ 301.186943] FS: 0000000000000000(0000) GS:ffff9cdcbd321000(0000) knlGS:0000000000000000
[ 301.187734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 301.188304] CR2: 0000000000000000 CR3: 0000000018030000 CR4: 00000000000006f0
[ 301.189003] note: torture_shutdow[137] exited with irqs disabled
[ 301.189621] torture_shutdow (137) used greatest stack depth: 13680 bytes left
[ 304.469828] kworker/u65:2 (153) used greatest stack depth: 12344 bytes left
[ 306.084634] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084635] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084648] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084650] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084652] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084652] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084653] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084656] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084666] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084667] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084667] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084667] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084667] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084671] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084686] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.084688] hazptr-stack-torture: hazptr_torture_reader is stopping
[ 306.781635] hazptr-stack-torture: rtc: 0000000073942ea9 ver: 359 HTWS_SYNC tfle: 0 rta: 360 rtaf: 0 rtf: 350 onoff: 38/38:47/47 7,15:10,50 329:573 (HZ=1000) acq: 5572447965 rel: 5572405682 acqirq: 306205 relirq: 348312 reldefer: 0 relundefer 0
[ 306.784165] hazptr-stack-torture: Reader Pipe: 5572751651 2343 0 0 0 0 0 0 0 0 0
[ 306.785055] hazptr-stack-torture: Reader Batch: 0 0 0 0 0 0 0 0 0 0 0
[ 306.785817] hazptr-stack-torture: Free-Block Circulation: 359 358 357 356 355 354 353 352 351 350 0
[ 306.786875] ??? Writer stall state HTWS_SYNC(3) g0 f0x0 ->state X cpu 13
[ 306.787662] hazptr-stack-torture: hazptr_torture_stats is stopping