Re: [NEEDS-REVIEW] Re: [PATCH v11 25/25] x86/cet/shstk: Add arch_prctl functions for shadow stack

[Edgecombe, Rick P]

On Mon, 2020-09-14 at 11:31 -0700, Andy Lutomirski wrote:
> > On Sep 14, 2020, at 7:50 AM, Dave Hansen <dave.hansen@xxxxxxxxx>
> > wrote:
> > 
> > On 9/11/20 3:59 PM, Yu-cheng Yu wrote:
> > ...
> > > Here are the changes if we take the mprotect(PROT_SHSTK)
> > > approach.
> > > Any comments/suggestions?
> > 
> > I still don't like it. :)
> > 
> > I'll also be much happier when there's a proper changelog to
> > accompany
> > this which also spells out the alternatives any why they suck so
> > much.
> > 
> 
> Let’s take a step back here. Ignoring the precise API, what exactly
> is
> a shadow stack from the perspective of a Linux user program?
> 
> The simplest answer is that it’s just memory that happens to have
> certain protections.  This enables all kinds of shenanigans.  A
> program could map a memfd twice, once as shadow stack and once as
> non-shadow-stack, and change its control flow.  Similarly, a program
> could mprotect its shadow stack, modify it, and mprotect it back.  In
> some threat models, though could be seen as a WRSS bypass.  (Although
> if an attacker can coerce a process to call mprotect(), the game is
> likely mostly over anyway.)
> 
> But we could be more restrictive, or perhaps we could allow user code
> to opt into more restrictions.  For example, we could have shadow
> stacks be special memory that cannot be written from usermode by any
> means other than ptrace() and friends, WRSS, and actual shadow stack
> usage.
> 
> What is the goal?
> 
> No matter what we do, the effects of calling vfork() are going to be
> a
> bit odd with SHSTK enabled.  I suppose we could disallow this, but
> that seems likely to cause its own issues.

Hi,

Resurrecting this old thread to highlight a consequence of the design
change that came out of it. I am going to be taking over this series
from Yu-cheng, and wanted to check if people would be interested in re-
visiting this interface.

The consequence I wanted to highlight, is that making userspace be
responsible for mapping memory as shadow stack, also requires moving
the writing of the restore token to userspace for glibc ucontext
operations. Since these operations involve creating/pivoting to new
stacks in userspace, ucontext cet support involves also creating a new
shadow stack. For normal thread stacks, the kernel has always done the
shadow stack allocation and so it is never writable (in the normal
sense) from userspace. But after this change makecontext() now first
has to mmap() writable memory, then write the restore token, then
mprotect() it as shadow stack. See the glibc changes to support
PROT_SHADOW_STACK here[0].

The writable window leaves an opening for an attacker to create an
arbitrary shadow stack that could be pivoted to later by tweaking the
ucontext_t structure. To try to see how much this matters, we have done
a small test that uses this window to ROP from writes in another
thread during the makecontext()/setcontext() window. (offensive work
credit to Joao on CC). This would require a real app to already to be
using ucontext in the course of normal runtime.

The original prctl solution prevents this case since the kernel did the
allocation and restore token setup, but of course it had other issues.
The other ideas discussed previously were a new syscall, or some sort
of new madvise() operation that could be involved in setting up shadow
stack, such that it is never writable in userspace. Or, simpler but
uglier, tweaking the existing PROT_SHADOW_STACK based solution by
making core mmap code write a restore token.

Those are still probably not enough to stop all ROP given extreme
threat models, as Andy alluded. But to me, this one seemed maybe worth
trying to improve. Especially thinking to avoid future ABI changes if
it becomes a problem. Would anyone like to see attempts to tighten this
up by revisiting this shadow stack allocation interface?

Thanks,

Rick

[0] 
https://gitlab.com/x86-glibc/glibc/-/commit/98eba0b829a1884c7d7bf76b0b39725919a9b6af#92c86763df6845c5b29664ac2ff40277678fd2c5_0_1