Re: [PATCH 2/2] x86/kstack_erase: Use REP STOSQ for large stack fills
From: Usama Arif
Date: Sat Nov 21 2026 - 09:38:54 EST
On 20/11/2026 14:22, David Laight wrote:
> On Fri, 20 Nov 2026 18:43:38 +0000
> Usama Arif <usama.arif@xxxxxxxxx> wrote:
>
>> On 20/11/2026 04:26, Peter Zijlstra wrote:
>>> On Fri, Nov 20, 2026 at 11:45:34AM +0100, Peter Zijlstra wrote:
>>>> On Thu, Nov 19, 2026 at 01:31:08PM -0800, Usama Arif wrote:
>>>>> +#ifdef CONFIG_X86_64
>>>>> +#define __stackleak_poison __stackleak_poison
>>>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>>>>> + unsigned long erase_high,
>>>>> + unsigned long poison)
>>>>> +{
>>>>> + unsigned long count;
>>>>> +
>>>>> + /* Short spans do not amortize the startup cost of REP STOSQ. */
>>>>> + if (erase_high <= erase_low + 512) {
>>>>> + while (erase_low < erase_high) {
>>>>> + *(unsigned long *)erase_low = poison;
>>>>> + erase_low += sizeof(unsigned long);
>>>>> + }
>>>>> + return;
>>>>> + }
>>>>> +
>>>>> + count = (erase_high - erase_low) / sizeof(unsigned long);
>>>>> + asm volatile("rep stosq"
>>>>> + : "+D" (erase_low), "+c" (count)
>>>>> + : "a" (poison)
>>>>> + : "memory");
>>>>> +}
>>>>> +#endif
>>>>
>>>> Gemini seems to suggest 16-32 words would be sufficient to amortize the
>>>> REP STOS setup. It was also adamant I try and unroll that earlier loop.
>>>>
>>>> How about something like so, does that make sense? Also, I suppose we
>>>> should make memset64() __always_inline if we use it here. I'm not at all
>>>> sold on the unroll here, the asm is pretty horrid for what it does.
>>>
>>> This one is much improved:
>>>
>>> --- a/arch/x86/include/asm/processor.h
>>> +++ b/arch/x86/include/asm/processor.h
>>> @@ -28,6 +28,7 @@ struct vm86;
>>> #include <asm/vmxfeatures.h>
>>> #include <asm/vdso/processor.h>
>>> #include <asm/shstk.h>
>>> +#include <asm/string.h>
>>>
>>> #include <linux/personality.h>
>>> #include <linux/cache.h>
>>> @@ -580,6 +581,35 @@ static __always_inline bool on_thread_st
>>> current_stack_pointer) < THREAD_SIZE;
>>> }
>>>
>>> +#ifdef CONFIG_X86_64
>>> +#define __stackleak_poison __stackleak_poison
>>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>>> + unsigned long erase_high,
>>> + unsigned long poison)
>>> +{
>>> + unsigned long count = (erase_high - erase_low) / sizeof(unsigned long);
>>> +
>>> + /* Amortize the startup cost of REP STOSQ. */
>>> + if (count < 32) {
>>> + unsigned long *ptr = (unsigned long *)erase_low;
>>> +
>>> + while (count) {
>>> + unsigned long mod = (count-1) % 4;
>>> + count -= mod+1;
>>> + switch (mod) {
>>> + case 3: ptr[count+3] = poison; fallthrough;
>>> + case 2: ptr[count+2] = poison; fallthrough;
>>> + case 1: ptr[count+1] = poison; fallthrough;
>>> + case 0: ptr[count+0] = poison; break;
>>> + }
>>> + }
>>> + return;
>>> + }
>>> +
>>
>> Hello Nick, Peter and David,
>>
>> Thanks for the reviews! I should have used memset64 from the get go :)
>>
>> I have tested 2 possible revisions now as well as this one from Peter,
>> have added both of them at the end of the reply.
>>
>> - v2.1 where you loop upto 512B and then memset
>> - v2.2: 4 stores per loop up to 512 B, then 64-byte aligned memset64()
>>
>> I timed stackleak_erase() in a Zen 5c guest over a futex/epoll/procfs/unix
>> syscall:
>>
>> No patch 68.3 ns per erase
>> v1 (loop up to 512 B, then REP) 46.6 ns
>> v2.1 (loop up to 512 B, memset64()) 46.6 ns
>> this version 46.4 ns
>> v2.2 (4 stores/iteration, aligned REP) 43.7 ns
>
> What are the lengths of those erases?
Its a mixture of lengths, coming from getppid(), read(), write(),
futex(), unix socket send(), receive(). I asked LLM to come up
with a benchmark with a mixture of syscalls that would target
a real world usecase and it came up with this which I found
to be quite good.
> (and where does the 46ns come from??)
It's the whole stackleak_erase(), the search for the poison plus the
fill, timed with rdtsc_ordered() in the kernel on every return to user,
minus an empty timed region, and averaged over all of the above.
> I'd definitely time 'just the code loop - various loops' and
> 'just rep stos'.
>
>> The switch doesn't seem to help.
>
> To many conditions in the loop.
> You could do it this way:
> switch (count & 3) {
> case 0: do { *ptr++ = poison;
> case 3: *ptr++ = poison;
> case 2: *ptr++ = poison;
> case 1: *ptr++ = poison;
> } while ((count -= 4) > 0);
> }
> (give or take some off-by-one here and there).
> But I suspect that using Duff's device is frowned up.
>
>> Either way I kept 512 bytes: REP STOSQ only wins from ~430 bytes on Zen
>> 5c against the plain loop (~850 against the unrolled one),
>
> For zen5 you should only need to unroll the loop once (two writes per iteration).
> In my experiments it can execute a loop in one clock.
> Unless the compiler unrolled it for you (and got it right!) the original
> loop would be half the speed
>
>> and from ~320 on Cooper Lake.
>
> I'd also try some older Intel cpu.
> The setup costs are likely to be higher with FRMS and friends.
>
It will be difficult for me to get access to CPUs older than Cooper Lake
which is 6 years old. I think if we try and profile and fit for every CPU,
this patch wont go anywhere. As seen in [1], the differences between the
the different optimizations are small, IMHO, we should just go for the most
maintainable approach, which I think is you loop upto 512B and then memset.
[1] https://lore.kernel.org/all/bad389b1-f4a6-43d4-9531-54b75e067c7a@xxxxxxxxx/
> David
>
>>
>>
>> My personal preference is v2.1, as its much simpler code and the difference
>> in performance between v2.1 and v2.2 isn't actually that much. But if there is
>> a stronger preference for v2.2, I can send that as it is faster.
>>
>>
>> v2.1:
>>
>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>> index 77488ff5eb8c7c..3d205ae8566114 100644
>> --- a/arch/x86/include/asm/processor.h
>> +++ b/arch/x86/include/asm/processor.h
>> @@ -28,6 +28,7 @@ struct vm86;
>> #include <asm/vmxfeatures.h>
>> #include <asm/vdso/processor.h>
>> #include <asm/shstk.h>
>> +#include <asm/string.h>
>>
>> #include <linux/personality.h>
>> #include <linux/cache.h>
>> @@ -580,6 +581,26 @@ static __always_inline bool on_thread_stack(void)
>> current_stack_pointer) < THREAD_SIZE;
>> }
>>
>> +#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
>> +#define __stackleak_poison __stackleak_poison
>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>> + unsigned long erase_high,
>> + unsigned long poison)
>> +{
>> + /* Short spans do not amortize the startup cost of REP STOSQ. */
>> + if (erase_high <= erase_low + 512) {
>> + while (erase_low < erase_high) {
>> + *(unsigned long *)erase_low = poison;
>> + erase_low += sizeof(unsigned long);
>> + }
>> + return;
>> + }
>> +
>> + memset64((u64 *)erase_low, poison,
>> + (erase_high - erase_low) / sizeof(u64));
>> +}
>> +#endif
>> +
>> #ifdef CONFIG_PARAVIRT_XXL
>> #include <asm/paravirt.h>
>> #else
>> diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
>> index 2dae70c2b01465..73cc91254b1381 100644
>> --- a/arch/x86/include/asm/string_64.h
>> +++ b/arch/x86/include/asm/string_64.h
>> @@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
>> }
>>
>> #define __HAVE_ARCH_MEMSET64
>> -static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>> +static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>> {
>> const auto s0 = s;
>> asm volatile (
>>
>>
>>
>>
>>
>> v2.2:
>>
>> diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
>> index 77488ff5eb8c7c..5e76e1140908d7 100644
>> --- a/arch/x86/include/asm/processor.h
>> +++ b/arch/x86/include/asm/processor.h
>> @@ -28,7 +28,9 @@ struct vm86;
>> #include <asm/vmxfeatures.h>
>> #include <asm/vdso/processor.h>
>> #include <asm/shstk.h>
>> +#include <asm/string.h>
>>
>> +#include <linux/align.h>
>> #include <linux/personality.h>
>> #include <linux/cache.h>
>> #include <linux/threads.h>
>> @@ -580,6 +582,48 @@ static __always_inline bool on_thread_stack(void)
>> current_stack_pointer) < THREAD_SIZE;
>> }
>>
>> +#if defined(CONFIG_X86_64) && !defined(CONFIG_KMSAN)
>> +#define __stackleak_poison __stackleak_poison
>> +static __always_inline void __stackleak_poison(unsigned long erase_low,
>> + unsigned long erase_high,
>> + unsigned long poison)
>> +{
>> + unsigned long *p = (unsigned long *)erase_low;
>> + unsigned long *end = (unsigned long *)erase_high;
>> + unsigned long i;
>> +
>> + /* Fewer than four words, or nothing to do. */
>> + if (erase_high < erase_low + 4 * sizeof(*p)) {
>> + while (p < end)
>> + *p++ = poison;
>> + return;
>> + }
>> +
>> + /* Four stores per iteration; the last four may overlap. */
>> + if (erase_high - erase_low <= 512) {
>> + end -= 4;
>> + while (p < end) {
>> + p[0] = poison;
>> + p[1] = poison;
>> + p[2] = poison;
>> + p[3] = poison;
>> + p += 4;
>> + }
>> + end[0] = poison;
>> + end[1] = poison;
>> + end[2] = poison;
>> + end[3] = poison;
>> + return;
>> + }
>> +
>> + /* REP STOSQ is faster from a 64-byte aligned destination. */
>> + for (i = 0; i < 8; i++)
>> + p[i] = poison;
>> + p = (unsigned long *)ALIGN(erase_low + 1, 64);
>> + memset64((u64 *)p, poison, end - p);
>> +}
>> +#endif
>> +
>> #ifdef CONFIG_PARAVIRT_XXL
>> #include <asm/paravirt.h>
>> #else
>> diff --git a/arch/x86/include/asm/string_64.h b/arch/x86/include/asm/string_64.h
>> index 2dae70c2b01465..73cc91254b1381 100644
>> --- a/arch/x86/include/asm/string_64.h
>> +++ b/arch/x86/include/asm/string_64.h
>> @@ -54,7 +54,7 @@ static inline void *memset32(uint32_t *s, uint32_t v, size_t n)
>> }
>>
>> #define __HAVE_ARCH_MEMSET64
>> -static inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>> +static __always_inline void *memset64(uint64_t *s, uint64_t v, size_t n)
>> {
>> const auto s0 = s;
>> asm volatile (
>>
>>> + memset64((void *)erase_low, poison, count);
>>> +}
>>> +#endif
>>> +
>>> #ifdef CONFIG_PARAVIRT_XXL
>>> #include <asm/paravirt.h>
>>> #else
>>
>>
>