Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills

From: David Laight

Date: Fri Nov 20 2026 - 07:43:09 EST


On Fri, 20 Nov 2026 11:38:25 +0000
David Laight <david.laight.linux@xxxxxxxxx> wrote:

> On Thu, 19 Nov 2026 14:28:04 -0800
> Nick Desaulniers <ndesaulniers@xxxxxxxxxx> wrote:
>
> > On Thu, Nov 19, 2026 at 1:32 PM Usama Arif <usama.arif@xxxxxxxxx> wrote:
> > >
> > > With KSTACK_ERASE, every return to user space poisons the part of the
> > > task stack that was used since the previous return. The generic
> > > __stackleak_poison() does that one word per loop iteration, so a syscall
> > > that went 1-2 KiB deep pays for 128-256 separate stores. On Meta hosts
> > > running many small VMs, stackleak_erase() accounts for about 1.5% of
> > > non-idle host kernel cycles.
> > >
> > > Override __stackleak_poison() on x86-64, as s390 already does, through
> > > the hook added by commit 491a78663e03 ("stackleak: allow to specify arch
> > > specific stackleak poison function"). Fill spans larger than 512 bytes
> > > with REP STOSQ and keep the word loop for shorter ones, where the fixed
> > > startup cost of REP STOSQ does not pay off. REP STOSQ starts to win at
> > > about 300-400 bytes on Zen 5c and Cooper Lake, so on both no span gets
> > > slower.
> > >
> > > The fill is inline asm, so the noinstr function still makes no calls.
> > > memset64() would be the same instruction, but it is only static inline,
> > > and with KMSAN it is an out-of-line function.
> > >
> > > In a Zen 5c KVM guest, stackleak_erase() got about 35% cheaper per return
> > > to user space (71 ns to 46 ns, scan included) over a futex, epoll, procfs
> > > and Unix socket syscall mix. Same-vCPU futex wake/wait and Unix socket
> > > sendmsg()/recvmsg() ping-pong round trips got 5-8% faster.
> > >
> > > Signed-off-by: Usama Arif <usama.arif@xxxxxxxxx>
> > > ---
> > > arch/x86/include/asm/processor.h | 25 +++++++++++++++++++++++++
> > > 1 file changed, 25 insertions(+)
> > >
> > > diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
> > > index 77488ff5eb8c7c..049231d484ad0e 100644
> > > --- a/arch/x86/include/asm/processor.h
> > > +++ b/arch/x86/include/asm/processor.h
> > > @@ -580,6 +580,31 @@ static __always_inline bool on_thread_stack(void)
> > > current_stack_pointer) < THREAD_SIZE;
> > > }
> > >
> > > +#ifdef CONFIG_X86_64
> > > +#define __stackleak_poison __stackleak_poison
> > > +static __always_inline void __stackleak_poison(unsigned long erase_low,
> > > + unsigned long erase_high,
> > > + unsigned long poison)
> > > +{
> > > + unsigned long count;
> > > +
> > > + /* Short spans do not amortize the startup cost of REP STOSQ. */
> > > + if (erase_high <= erase_low + 512) {
> > > + while (erase_low < erase_high) {
> > > + *(unsigned long *)erase_low = poison;
> > > + erase_low += sizeof(unsigned long);
> > > + }
> > > + return;
>
> gcc 13 onwards makes a meal of that - https://godbolt.org/z/KPP4h8xz6
> It does manage to unroll the loop once - which is good because the intel
> cpu (I've got) won't execute a 'one clock loop' (the zen5 will); but the
> setup code is horrid.
> But I also think some of the more recent cpu can execute more than
> one write per clock.
> If you unroll the loop 4 times it may run a lot faster.
> Perhaps:
> long count = (erase_high - erase_low) / sizeof(unsigned long);
> if (count <= 64) {
> while ((count -= 4) >= 0) {
> *(unsigned long *)erase_low = poison;
> *(unsigned long *)(erase_low + 8) = poison;

The add (probably) needs to be in the middle...
erase_low += 32;
> *(unsigned long *)(erase_low - 32 + 16) = poison;
> *(unsigned long *)(erase_low - 32 + 24) = poison;
> }
> if (count & 2) {
> *(unsigned long *)erase_low = poison;
> *(unsigned long *)(erase_low + 8) = poison;
> erase_low += 16;
> }
> if (count & 1)
> *(unsigned long *)erase_low = poison;
> } else {
> ...
> }
>
> Except that gcc makes a mess of it again.
> I don't think the 'erase_low += 32' is going to execute at the
> same time as the last (two probably) writes so costs a clock.
> So it may need to be asm.

I threw in enough OPTIMIZER_HIDE_VAR() to stop the optimiser breaking
the code. https://godbolt.org/z/nx477T4M1
gcc then generates:
movq %rdx, (%rdi)
movq %rdx, 8(%rdi)
addq $32, %rdi
movq %rdx, -16(%rdi)
movq %rdx, -8(%rdi)
.L32:
subq $4, %rcx
jns .L22
for the code loop (clang is a bit worse - but should be ok).
I think that will be a 'two clock loop' on cpu that can do
two writes/clock; zen3 and ice lake).
For 'stack splatting' rounding up to a multiple of 32 bytes
will save all the 'remainder' code.

David

>
> > > + }
> > > +
> > > + count = (erase_high - erase_low) / sizeof(unsigned long);
> > > + asm volatile("rep stosq"
> > > + : "+D" (erase_low), "+c" (count)
> > > + : "a" (poison)
> > > + : "memory");
> >
> > Surely this isn't the only place using `rep stosq` in the tree, right?
> > It's not, see memset64 in arch/x86/include/asm/string_64.h.
>
> Which seems to have very few users.
> memset_p() and memset_l() have almost no users.
>
> David
>
> >
> > > +}
> > > +#endif
> > > +
> > > #ifdef CONFIG_PARAVIRT_XXL
> > > #include <asm/paravirt.h>
> > > #else
> > > --
> > > 2.53.1-Meta
> > >
> >
> >
>