Re: [PATCH v1 00/15] Add support for Nitro Enclaves

[Paraschiv, Andra-Irina]

On 23/04/2020 16:42, Paolo Bonzini wrote:
> On 23/04/20 15:19, Paraschiv, Andra-Irina wrote:
> > 2. The enclave itself - a VM running on the same host as the primary VM
> > that spawned it.
> >
> > The enclave VM has no persistent storage or network interface attached,
> > it uses its own memory and CPUs + its virtio-vsock emulated device for
> > communication with the primary VM.
> >
> > The memory and CPUs are carved out of the primary VM, they are dedicated
> > for the enclave. The Nitro hypervisor running on the host ensures memory
> > and CPU isolation between the primary VM and the enclave VM.
> >
> > These two components need to reflect the same state e.g. when the
> > enclave abstraction process (1) is terminated, the enclave VM (2) is
> > terminated as well.
> >
> > With regard to the communication channel, the primary VM has its own
> > emulated virtio-vsock PCI device. The enclave VM has its own emulated
> > virtio-vsock device as well. This channel is used, for example, to fetch
> > data in the enclave and then process it. An application that sets up the
> > vsock socket and connects or listens, depending on the use case, is then
> > developed to use this channel; this happens on both ends - primary VM
> > and enclave VM.
> >
> > Let me know if further clarifications are needed.
> Thanks, this is all useful.  However can you please clarify the
> low-level details here?
>
> > > - the initial CPU state: CPL0 vs. CPL3, initial program counter, etc.

The enclave VM has its own kernel and follows the well-known Linux boot 
protocol, in the end getting to the user application after init finishes 
its work, so that's CPL3.

> > > - the communication channel; does the enclave see the usual local APIC
> > > and IOAPIC interfaces in order to get interrupts from virtio-vsock, and
> > > where is the virtio-vsock device (virtio-mmio I suppose) placed in
> > > memory?
vsock is using eventfd for signalling; wrt enclave VM, it sees the usual 
interfaces to get interrupts from virtio dev.

It's placed below the typical 4GB; in general, it may depend based on arch.

> > > - what the enclave is allowed to do: can it change privilege levels,
> > > what happens if the enclave performs an access to nonexistent memory,
> > > etc.

If talking about the enclave abstraction process, it is running in the 
primary VM as a user space process, so it will get into primary VM guest 
kernel if privileged instructions need to be executed.

Same happens with the user space application running in the enclave VM. 
And the VM itself will get to the hypervisor running on the host for 
privileged instructions. The Nitro hypervisor is based on core KVM 
technology.

Access to nonexistent memory gets faults.

> > > - whether there are special hypercall interfaces for the enclave

The path towards creating / setting resources / terminating an enclave 
(here referring to enclave VM) is towards the ioctl interface, with the 
corresponding misc device, and the emulated PCI device. That's the 
interface used to manage enclaves. Once booted, the enclave resources 
setup is not modified anymore. And the way to communicate with the 
enclave after booting, with the application running in the enclave, is 
via the vsock comm channel.


Thanks,
Andra

> Thanks,
>
> Paolo




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.