Re: [PATCH 0/5] KVM/x86: add a new hypercall to execute host system

[Andrei Vagin]

On Fri, Jul 22, 2022 at 4:41 PM Sean Christopherson <seanjc@xxxxxxxxxx> wrote:
>
> +x86 maintainers, patch 1 most definitely needs acceptance from folks beyond KVM.
>
> On Fri, Jul 22, 2022, Andrei Vagin wrote:
> > Another option is the KVM platform. In this case, the Sentry (gVisor
> > kernel) can run in a guest ring0 and create/manage multiple address
> > spaces. Its performance is much better than the ptrace one, but it is
> > still not great compared with the native performance. This change
> > optimizes the most critical part, which is the syscall overhead.
>
> What exactly is the source of the syscall overhead,

Here are perf traces for two cases: when "guest" syscalls are executed via
hypercalls and when syscalls are executed by the user-space VMM:
https://gist.github.com/avagin/f50a6d569440c9ae382281448c187f4e

And here are two tests that I use to collect these traces:
https://github.com/avagin/linux-task-diag/commit/4e19c7007bec6a15645025c337f2e85689b81f99

If we compare these traces, we can find that in the second case, we spend extra
time in vmx_prepare_switch_to_guest, fpu_swap_kvm_fpstate, vcpu_put,
syscall_exit_to_user_mode.

> and what alternatives have been explored?  Making arbitrary syscalls from
> within KVM is mildly terrifying.

"mildly terrifying" is a good sentence in this case:). If I were in your place,
I would think about it similarly.

I understand these concerns about calling syscalls from the KVM code, and this
is why I hide this feature under a separate capability that can be enabled
explicitly.

We can think about restricting the list of system calls that this hypercall can
execute. In the user-space changes for gVisor, we have a list of system calls
that are not executed via this hypercall. For example, sigprocmask is never
executed by this hypercall, because the kvm vcpu has its signal mask.  Another
example is the ioctl syscall, because it can be one of kvm ioctl-s.

As for alternatives, we explored different ways:

== Host Ring3/Guest ring0 mixed mode ==

This is how the gVisor KVM platform works right now. We don’t have a separate
hypervisor, and the Sentry does its functions. The Sentry creates a KVM virtual
machine instance, sets it up, and handles VMEXITs. As a result, the Sentry runs
in the host ring3 and the guest ring0 and can transparently switch between
these two contexts.

When the Sentry starts, it creates a new kernel VM instance and maps its memory
to the guest physical. Then it makes a set of page tables for the Sentry that
mirrors the host virtual address space. When host and guest address spaces are
identical, the Sentry can switch between these two contexts.

The bluepill function switches the Sentry into guest ring0. It calls a
privileged instruction (CLI) that is a no-op in the guest (by design, since we
ensure interrupts are disabled for guest ring 0 execution) and triggers a
signal on the host. The signal is handled by the bluepillHandler that takes a
virtual CPU and executes it with the current thread state grabbed from a signal
frame.

As for regular VMs, user processes have their own address spaces (page tables)
and run in guest ring3. So when the Sentry is going to execute a user process,
it needs to be sure that it is running inside a VM, and it is the exact point
when it calls bluepill(). Then it executes a user process with its page tables
before it triggers an exception or a system call. All such events are trapped
and handled in the Sentry.

The Sentry is a normal Linux process that can trigger a fault and execute
system calls. To handle these events, the Sentry returns to the host mode. If
ring0 sysenter or exception entry point detects an event from the Sentry, they
save the current thread state on a per-CPU structure and trigger VMEXIT. This
returns us into bluepillHandler, where we set the thread state on a signal
frame and exit from the signal handler, so the Sentry resumes from the point
where it has been in the VM.

In this scheme, the sentry syscall time is 3600ns. This is for the case when a
system call is called from gr0.

The benefit of this way is that only a first system call triggers vmexit and
all subsequent syscalls are executed on the host natively.

But it has downsides:
* Each sentry system call trigger the full exit to hr3.
* Each vmenter/vmexit requires to trigger a signal but it is expensive.
* It doesn't allow to support Confidential Computing (SEV-ES/SGX). The Sentry
  has to be fully enclosed in a VM to be able to support these technologies.

== Execute system calls from a user-space VMM ==

In this case, the Sentry is always running in VM, and a syscall handler in GR0
triggers vmexit to transfer control to VMM (user process that is running in
hr3), VMM executes a required system call, and transfers control back to the
Sentry. We can say that it implements the suggested hypercall in the
user-space.

The sentry syscall time is 2100ns in this case.

The new hypercall does the same but without switching to the host ring 3. It
reduces the sentry syscall time to 1000ns.


== A new BPF hook to handle vmexit-s  ==

https://github.com/avagin/linux-task-diag/commits/kvm-bpf

This way allows us to reach the same performance numbers, but it gives less
control over who and how use this functionality. Second, it requires adding a
few questionable BPF helpers like calling syscall from BPF hooks.

== Non-KVM platforms ==

We are experimenting with non-KVM platforms. We have the ptrace platform, but it
is almost for experiments due to the slowness of the ptrace interface.

Another idea was to add the process_vm_exec system call:
https://lwn.net/Articles/852662/

This system call can significantly increase performance compared with the
ptrace platform, but it is still slower than the KVM platform in its current
form (without the new hypercall). But this is true only if we run the KVM
platform on a bare-metal. In the case of nested-virtualization, the KVM
platform becomes much slower, which is expected.

We have another idea to use the seccomp notify to trap system calls, but it
requires some kernel change to reach a reasonable performance. I am working on
these changes and will present them soon.

I want to emphasize that non-KVM platforms don't allow us to implement the
confidential concept in gVisor, but this is one of our main goals concerning
the KVM platform.

All previous numbers have been getting from the same host (Xeon(R) Gold 6268CL,
5.19-rc5).

>
> > The idea of using vmcall to execute system calls isn’t new. Two large users
> > of gVisor (Google and AntFinacial) have out-of-tree code to implement such
> > hypercalls.
> >
> > In the Google kernel, we have a kvm-like subsystem designed especially
> > for gVisor. This change is the first step of integrating it into the KVM
> > code base and making it available to all Linux users.
>
> Can you please lay out the complete set of changes that you will be proposing?
> Doesn't have to be gory details, but at a minimum there needs to be a high level
> description that very clearly defines the scope of what changes you want to make
> and what the end result will look like.
>
> It's practically impossible to review this series without first understanding the
> bigger picture, e.g. if KVM_HC_HOST_SYSCALL is ultimately useless without the other
> bits you plan to upstream, then merging it without a high level of confidence that
> the other bits are acceptable is a bad idea since it commits KVM to supporting
> unused ABI.

I was not precise in my description. This is the only change that we
need right now.
The gVisor KVM platform is the real thing that exists today and works
on the upstream kernels:
https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/platform/kvm/

This hypercall improves its performance and makes it comparable with
the google-internal platform.

Thanks,
Andrei