SGX notes from KS/LPC

[Andy Lutomirski]

Hi-

Here are my notes on SGX issues from KS/LPC.  It seems that I never
emailed it out to a public list -- oops.  It may contain any number of
typos or outright errors.

+++ cut here +++

=== Background and terminology ===

An enclave is an SGX enclave.  Once launched, unless the enclave is explicitly
debuggable, the enclave's data cannot be seen by the OS.

An "ordinary enclave" is an enclave without the PROVISIONKEY, EINITTOKENKEY,
or any reserved attributes set.  I expect the vast majority of enclaves
developed by ISVs to be ordinary enclaves.

A "special enclave" is any enclave that isn't an ordinary enclave.  A
"launch enclave" is a special enclave that has EINITTOKENKEY set.  Launch
enclaves are unusual in that a launch enclave can *only* be run if it is
signed by the private key associated with SGXLEPUBKEYHASH -- a launch
enclave cannot chain to another launch enclave by creating an EINITTOKEN
authorizing it.  (I'm not sure why this is the case, but that's what
the SDM says.)

"Flexible launch control" is the Intel feature that adds SGXLEPUBKEYHASH.

=== Goals for upstream Linux SGX support ===

I consider the following to be fairly strong requirements:

 * Linux SGX support should be useful to normal people.  This means that
   it should be convenient for developers to create, run, and distribute
   enclaves and to recompile and use other developers' enclaves.  Doing
   so should not require paying a fee or signing a contract in the general
   case.  There must not be any conditions incompatible with the use of
   open source enclaves.

 * System owners, admins, distributions, and designers of embedded systems
   should be able to apply reasonable policies that restrict what enclaves
   may be run on their systems.

 * Software should be portable across systems to a reasonable extent.
   This means that software should run on any machine with the
   appropriate hardware and kernel as long as that machine's policy
   permits the software to work.  In the context of SGX with flexible
   launch control, it should be at least inconvenient and maybe even
   impossible for an ISV to ship software signed by a particular
   launch enclave such that an end user cannot run the software
   without trusting the launch enclave.  See below.

=== Recommendations ===

Linux should not support SGX on non-flexible-launch-control systems except as
outlined below.  As it stands, it is impossible to write a Linux SGX driver
that can be used in any meaningful way by open source developers.  I doubt
that any upstream maintainers are willing to participate in the existing
licensing process.  I am certainly not willing to participate on my own
behalf.

On flexible launch control systems, SGX should be supported, but I think that
the existing proposed API workflow for loading an enclave should change.

I'll call the current proposal "user-provided EINITTOKEN".  It works like
this: A user program bundles a launch enclave and asks the kernel to launch
it.  Then the user program asks the launch enclave to generate an EINITTOKEN
for the normal enclave it wants to use.  If the launch enclave approves the
request, it gives the user program an EINITTOKEN.  Then the user program asks
the kernel to launch the normal enclave and supplies the EINITTOKEN to the
kernel.  The kernel launches the enclave.

I suggest an alternative that I'll call "kernel-provided EINITTOKEN".  In this
model, a user program supplies a normal enclave to the kernel and does *not*
supply EINITTOKEN.  The kernel applies its own policy, if any, to decide whether
to accept the enclave.

Here are some reasons that I think kernel-provided EINITTOKEN will work much
better for Linux:

 * Portability: Suppose an ISV ships a program (e.g. Chrome) that bundles a
   normal enclave.  That ISV will sign the enclave.  If they want the enclave
   to run on a system using, say, Red Hat's launch policy, they'll arrange for
   Red Hat to accept their signing key.  Under the user-provided EINITTOKEN
   model, they will ship a Red Hat-provided launch enclave that accepts their
   key.  They'll use this launch enclave at run tiem to generate their
   EINITTOKEN.  This has major problems:

    * It may be incompatible with fully open-source software.  If the ISV
      doesn't have the source to Red Hat's launch enclave, they are forced to
      bundle a binary blob and to give that blob access to their address
      space.  (In the Red Hat example, I imagine that source would be
      available, but this nonetheless imposes an unpleasant auditing
      requirement on the ISV.)

    * It is non-portable.  If a Debian user wants to run Chrome, they'll need
      to ensure that their system policy accepts Chrome's enclave.  This is
      fine, but even if they do so, Chrome still won't work unless Debian or
      the user also accepts the Red Hat launch enclave that Chrome bundles.
      The user may not want to do this.  I expect that distributions like
      Debian are unwilling to grant blanket trust to other vendors' launch
      enclaves.

    In the kernel-provided EINITTOKEN model, this is a non-issue: the launch
    enclave will be provided by the system and non-Red Hat users will simply
    provide a different launch enclave.

 * Performance: Imagine that a program like Chrome bundles an enclave.  The
   first time any user runs Chrome, an EINITTOKEN will need to be generated.
   Once this happens, in the user-provided EINITTOKEN model, if a different
   user runs Chrome, they'll have to generate their own EINITTOKEN.  In
   the kernel-provided EINITTOKEN model, the kernel can maintain a simple
   cache of EINITTOKENs and the second user's launch of Chrome can be
   faster.

 * Simplicity of SGXLEPUBKEYHASH handling: with user-provided EINITTOKEN,
   the kernel has to check that it considers the launch enclave's signer
   to be acceptable and update SGXLEPUBKEYHASH accordingly.  Then, when
   the normal enclave is launched, the kernel needs to make sure that
   the correct SGXLEPUBKEYHASH is set so that the CPU will accept the
   EINITTOKEN.

 * Security: there is no hardware mechanism that guarantees EINITTOKEN
   freshness.  This means that, if the kernel policy is changed by the
   system owner, old EINITTOKENs are still accepted by the hardware.
   As a result, for kernel policy to work properly, the kernel cannot trust
   a user-provided EINITTOKEN as an indication that an enclave is acceptable.

 * Flexibility: the kernel may want to enforce policies that are not directly
   enforceable by launch enclaves.  For example, different users may be
   granted permission to use different sets of enclaves.  Alternatively, the
   kernel may want to revoke permission to use certain enclaves.  As a
   practical matter, this means that the kernel may need to enforce its own
   policy when the user asks it to do EINIT, which means that the kernel
   needs to do most of the work anyway.

I'm inclined to suggest that normal users not be permitted to run launch
enclaves directly, mainly because, in the kernel-provided EINITTOKEN
model, I don't see the point.  If users can do this, they might do it
by rote even if they don't need to, this breaking portability.

I think that, in the first version at least, only root should be able
to run provisioning enclaves.  This is because I don't fully
understand the provisioning workflow and because there are potential
privacy implications to allowing unprivileged users to obtain a
provisioning key.

I think that the kernel should reject enclaves that have reserved attributes
set.  I don't see how the driver could handle such enclaves correctly without
knowing their semantics.  (For example, an enclave with the EINITTOKEN
attribute set is launched differently from other enclaves.)

Since it's currently quite awkward for the kernel to run CPL3 code, which it
would need to do to easily invoke a launch enclave, I hereby volunteer
to make it possible for the kernel to do this.

=== Virtualization considerations ===

At some point, someone will need to decide what restrictions if any
KVM should impose on its guests' use of SGX.

For example, should KVM limit the set of SGXLEPUBKEYHASH values that
its guests can use?  Should KVM intercept EINIT and filter enclaves
directly?

=== Support for non-flexible-launch-control CPUs (e.g. Skylake) ===

If there's a way to make Skylake work similarly to flexible launch control
systems under a kernel-provided EINITTOKEN model, I think that such a mechanism
could be acceptable upstream.  For example, if Intel provided a signed enclave
that could be distributed in linux-firmware.git that would allow the kernel to
enforce its own launch policy and generate the requisite EINITTOKEN
values in a comparably flexible and unrestrictive manner, I can see
this being acceptable upstream.

I'm not sure that upstream would want to support this on CPUs that
are capable of flexible launch control but have it locked by firmware.
I think we want to very strongly discourage firmware from locking
the SGXLEPUBKEYHASH registers.

=== Launch policies that the kernel should provide ===

The kernel's launch policy should be configurable by some reasonable
means.  One of the options should be "launch any normal enclave".  Another
should probably be "accept launch enclaves provided by root (e.g. in
/lib/firmware) that match one of the following SGXLEPUBKEYHASH
values).

I can imagine use cases for another policy like "launch the following
enclaves, listed by hash".

=== A security flaw that Intel should seriously consider fixing ===

SGX, as presently designed, has what I consider to be a significant design flaw
that is relevant in multi-tenant installations.  Specifically, suppose a
single cloud CPU hosts VMs from two unrelated tenants, A and B.  A uses
SGX to protect its data.  If B is able to obtain a dump of A's storage, B
may obtain an enclave that belongs to A along with data sealed by that
enclave.  While B cannot directly unseal A's data, there is no existing
mechanism that can usefully prevent B from running A's enclave and using
that enclave's API to manipulate A's data.

This could be mitigated in future SGX revisions by adding an additional
personalization parameter to EINIT.  This parameter would be mixed in
to the SGX key derivation process such that enclaves initialized using
different personalization keys would obtain different outputs from EGETKEY.
A hypervisor would trap EINIT, replace the personalization key with
KDF(guest's requested personalization key, tenant-specific personalization
secret).  The result would be that one tenant on a machine would be unable
to use another tenant's enclaves unless the hypervisor were compromised.