Re: [PATCH 00/13] Virtually mapped stacks with guard pages (x86, core)

[Heiko Carstens]

On Fri, Jun 17, 2016 at 10:38:24AM -0700, Andy Lutomirski wrote:
> > A disassembly looks like this (r15 is the stackpointer):
> >
> > 0000000000000670 <setup_arch>:
> >      670:       eb 6f f0 48 00 24       stmg    %r6,%r15,72(%r15)
> >      676:       c0 d0 00 00 00 00       larl    %r13,676 <setup_arch+0x6>
> >      67c:       a7 f1 3f 80             tmll    %r15,16256  <--- test if enough space left
> >      680:       b9 04 00 ef             lgr     %r14,%r15
> >      684:       a7 84 00 01             je      686 <setup_arch+0x16> <--- branch to illegal op
> >      688:       e3 f0 ff 90 ff 71       lay     %r15,-112(%r15)
> >
> > The branch jumps actually into the branch instruction itself since the 0001
> > part of the "je" instruction is an illegal instruction.
> >
> > This catches at least wild stack overflows because of two many functions
> > being called.
> >
> > Of course it doesn't catch wild accesses outside the stack because e.g. the
> > index into an array on the stack is wrong.
> >
> > The runtime overhead is within noise ratio, therefore we have this always
> > enabled.
> >
> 
> Neat!  What exactly does tmll do?  I assume this works by checking the
> low bits of the stack pointer.
> 
> x86_64 would have to do:
> 
> movl %esp, %r11d
> shll %r11d, $18
> cmpl %r11d, <threshold>
> jg error
> 
> Or similar.  I think the cmpl could be eliminated if the threshold
> were a power of two by simply testing the low bits of the stack
> pointer.

The tmll instruction tests if any of the higher bits within the 16k
stackframe address are set. In this specific case that would be bits 7-15
(mask 0x3f80). If no bit would be set we know that only up to 128 bytes
would be left on the stack, and thus trigger an exception.

This check does of course only work if a 16k stack is also 16k aligned,
which is always the case.