[PATCH 00/30] [v2] KAISER: unmap most of the kernel from userspace page tables

[Dave Hansen]

Thanks, everyone for all the review of v1.  I hope I managed to
address all the feedback given so far, except for the TODOs of
course.

Changes from v1:
 * Updated to be on top of Andy L's new entry code
 * Allow global pages again, and use them for pages mapped into
   userspace page tables.
 * Use trampoline stack instead of process stack at entry so no
   longer need to map process stack (big win in fork() speed)
 * Made the page table walking less generic by restricting it
   to kernel addresses and !_PAGE_USER pages.
 * Added a debugfs file to enable/disable CR3 switching at
   runtime.  This does not remove all the KAISER overhead, but
   it removes the largest source.
 * Use runtime disable with Xen to permit Xen-PV guests with
   KAISER=y.
 * Moved assembly code from "core" to "prepare assembly" patch
 * Pass full register name to asm macros
 * Remove double stack switch in entry_SYSENTER_compat
 * Disable vsyscall native case when KAISER=y
 * Separate PER_CPU_USER_MAPPED generic definitions from use
   by arch/x86/.

TODO:
 * Allow dumping the shadow page tables with the ptdump code
 * Put LDT at top of userspace
 * Create separate tlb flushing functions for user and kernel
 * Chase down the source of the new !CR4.PGE warning that 0day
   found with i386

---

tl;dr:

KAISER makes it harder to defeat KASLR, but makes syscalls and
interrupts slower.  These patches are based on work from a team at
Graz University of Technology posted here[1].  The major addition is
support for Intel PCIDs which builds on top of Andy Lutomorski's PCID
work merged for 4.14.  PCIDs make KAISER's overhead very reasonable
for a wide variety of use cases.

Full Description:

KAISER is a countermeasure against attacks on kernel address
information.  There are at least three existing, published,
approaches using the shared user/kernel mapping and hardware features
to defeat KASLR.  One approach referenced in the paper locates the
kernel by observing differences in page fault timing between
present-but-inaccessable kernel pages and non-present pages.

KAISER addresses this by unmapping (most of) the kernel when
userspace runs.  It leaves the existing page tables largely alone and
refers to them as "kernel page tables".  For running userspace, a new
"shadow" copy of the page tables is allocated for each process.  The
shadow page tables map all the same user memory as the "kernel" copy,
but only maps a minimal set of kernel memory.

When we enter the kernel via syscalls, interrupts or exceptions,
page tables are switched to the full "kernel" copy.  When the system
switches back to user mode, the "shadow" copy is used.  Process
Context IDentifiers (PCIDs) are used to to ensure that the TLB is not
flushed when switching between page tables, which makes syscalls
roughly 2x faster than without it.  PCIDs are usable on Haswell and
newer CPUs (the ones with "v4", or called fourth-generation Core).

The minimal kernel page tables try to map only what is needed to
enter/exit the kernel such as the entry/exit functions, interrupt
descriptors (IDT) and the kernel trampoline stacks.  This minimal set
of data can still reveal the kernel's ASLR base address.  But, this
minimal kernel data is all trusted, which makes it harder to exploit
than data in the kernel direct map which contains loads of
user-controlled data.

KAISER will affect performance for anything that does system calls or
interrupts: everything.  Just the new instructions (CR3 manipulation)
add a few hundred cycles to a syscall or interrupt.  Most workloads
that we have run show single-digit regressions.  5% is a good round
number for what is typical.  The worst we have seen is a roughly 30%
regression on a loopback networking test that did a ton of syscalls
and context switches.  More details about possible performance
impacts are in the new Documentation/ file.

This code is based on a version I downloaded from
(https://github.com/IAIK/KAISER).  It has been heavily modified.

The approach is described in detail in a paper[2].  However, there is
some incorrect and information in the paper, both on how Linux and
the hardware works.  For instance, I do not share the opinion that
KAISER has "runtime overhead of only 0.28%".  Please rely on this
patch series as the canonical source of information about this
submission.

Here is one example of how the kernel image grow with CONFIG_KAISER
on and off.  Most of the size increase is presumably from additional
alignment requirements for mapping entry/exit code and structures.

    text    data     bss      dec filename
11786064 7356724 2928640 22071428 vmlinux-nokaiser
11798203 7371704 2928640 22098547 vmlinux-kaiser
  +12139  +14980       0   +27119

To give folks an idea what the performance impact is like, I took
the following test and ran it single-threaded:

	https://github.com/antonblanchard/will-it-scale/blob/master/tests/lseek1.c

It's a pretty quick syscall so this shows how much KAISER slows
down syscalls (and how much PCIDs help).  The units here are
lseeks/second:

        no kaiser: 5.2M
    kaiser+  pcid: 3.0M
    kaiser+nopcid: 2.2M

"nopcid" is literally with the "nopcid" command-line option which
turns PCIDs off entirely.

Thanks to:
The original KAISER team at Graz University of Technology.
Andy Lutomirski for all the help with the entry code.
Kirill Shutemov for a helpful review of the code.

1. https://github.com/IAIK/KAISER
2. https://gruss.cc/files/kaiser.pdf

--

The code is available here:

	https://git.kernel.org/pub/scm/linux/kernel/git/daveh/x86-kaiser.git/

Dave Hansen (30):
      x86, mm: do not set _PAGE_USER for init_mm page tables
      x86, tlb: make CR4-based TLB flushes more robust
      x86, mm: document X86_CR4_PGE toggling behavior
      x86, kaiser: disable global pages by default with KAISER
      x86, kaiser: prepare assembly for entry/exit CR3 switching
      x86, kaiser: introduce user-mapped percpu areas
      x86, kaiser: mark percpu data structures required for entry/exit
      x86, kaiser: unmap kernel from userspace page tables (core patch)
      x86, kaiser: only populate shadow page tables for userspace
      x86, kaiser: allow NX to be set in p4d/pgd
      x86, kaiser: make sure static PGDs are 8k in size
      x86, kaiser: map GDT into user page tables
      x86, kaiser: map dynamically-allocated LDTs
      x86, kaiser: map espfix structures
      x86, kaiser: map entry stack variables
      x86, kaiser: map trace interrupt entry
      x86, kaiser: map debug IDT tables
      x86, kaiser: map virtually-addressed performance monitoring buffers
      x86, mm: Move CR3 construction functions
      x86, mm: remove hard-coded ASID limit checks
      x86, mm: put mmu-to-h/w ASID translation in one place
      x86, pcid, kaiser: allow flushing for future ASID switches
      x86, kaiser: use PCID feature to make user and kernel switches faster
      x86, kaiser: disable native VSYSCALL
      x86, kaiser: add debugfs file to turn KAISER on/off at runtime
      x86, kaiser: add a function to check for KAISER being enabled
      x86, kaiser: un-poison PGDs at runtime
      x86, kaiser: allow KAISER to be enabled/disabled at runtime
      x86, kaiser: add Kconfig
      x86, kaiser, xen: Dynamically disable KAISER when running under Xen PV

 Documentation/x86/kaiser.txt                | 160 +++++
 arch/x86/Kconfig                            |   8 +
 arch/x86/entry/calling.h                    |  89 +++
 arch/x86/entry/entry_64.S                   |  40 +-
 arch/x86/entry/entry_64_compat.S            |   8 +
 arch/x86/events/intel/ds.c                  |  57 +-
 arch/x86/include/asm/cpufeatures.h          |   1 +
 arch/x86/include/asm/desc.h                 |   2 +-
 arch/x86/include/asm/kaiser.h               |  62 ++
 arch/x86/include/asm/mmu_context.h          |  29 +-
 arch/x86/include/asm/pgalloc.h              |  33 +-
 arch/x86/include/asm/pgtable.h              |  20 +-
 arch/x86/include/asm/pgtable_64.h           | 135 +++++
 arch/x86/include/asm/pgtable_types.h        |  25 +-
 arch/x86/include/asm/processor.h            |   2 +-
 arch/x86/include/asm/tlbflush.h             | 230 +++++++-
 arch/x86/include/uapi/asm/processor-flags.h |   3 +-
 arch/x86/kernel/cpu/common.c                |  21 +-
 arch/x86/kernel/espfix_64.c                 |  27 +-
 arch/x86/kernel/head_64.S                   |  30 +-
 arch/x86/kernel/ldt.c                       |  25 +-
 arch/x86/kernel/process.c                   |   2 +-
 arch/x86/kernel/process_64.c                |   2 +-
 arch/x86/kernel/traps.c                     |  46 +-
 arch/x86/kvm/x86.c                          |   3 +-
 arch/x86/mm/Makefile                        |   1 +
 arch/x86/mm/init.c                          |  75 ++-
 arch/x86/mm/kaiser.c                        | 616 ++++++++++++++++++++
 arch/x86/mm/pageattr.c                      |  18 +-
 arch/x86/mm/pgtable.c                       |  16 +-
 arch/x86/mm/tlb.c                           | 105 +++-
 include/asm-generic/vmlinux.lds.h           |  17 +
 include/linux/kaiser.h                      |  34 ++
 include/linux/percpu-defs.h                 |  30 +
 init/main.c                                 |   3 +
 kernel/fork.c                               |   1 +
 security/Kconfig                            |  10 +
 37 files changed, 1838 insertions(+), 148 deletions(-)

Cc: Moritz Lipp <moritz.lipp@xxxxxxxxxxxxxx>
Cc: Daniel Gruss <daniel.gruss@xxxxxxxxxxxxxx>
Cc: Michael Schwarz <michael.schwarz@xxxxxxxxxxxxxx>
Cc: Richard Fellner <richard.fellner@xxxxxxxxxxxxxxxxx>
Cc: Andy Lutomirski <luto@xxxxxxxxxx>
Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
Cc: Kees Cook <keescook@xxxxxxxxxx>
Cc: Hugh Dickins <hughd@xxxxxxxxxx>
Cc: x86@xxxxxxxxxx
Cc: Juergen Gross <jgross@xxxxxxxx>