Re: [PATCH v1 00/15] Add support for Nitro Enclaves

From: Paraschiv, Andra-Irina
Date: Mon May 11 2020 - 08:05:45 EST



On 10/05/2020 12:57, Li Qiang wrote:


Paraschiv, Andra-Irina <andraprs@xxxxxxxxxx <mailto:andraprs@xxxxxxxxxx>> ä2020å4æ24æåä äå10:03åéï



On 24/04/2020 12:59, Tian, Kevin wrote:
>
>> From: Paraschiv, Andra-Irina
>> Sent: Thursday, April 23, 2020 9:20 PM
>>
>> On 22/04/2020 00:46, Paolo Bonzini wrote:
>>> On 21/04/20 20:41, Andra Paraschiv wrote:
>>>> An enclave communicates with the primary VM via a local
communication
>> channel,
>>>> using virtio-vsock [2]. An enclave does not have a disk or a
network device
>>>> attached.
>>> Is it possible to have a sample of this in the samples/ directory?
>> I can add in v2 a sample file including the basic flow of how
to use the
>> ioctl interface to create / terminate an enclave.
>>
>> Then we can update / build on top it based on the ongoing
discussions on
>> the patch series and the received feedback.
>>
>>> I am interested especially in:
>>>
>>> - the initial CPU state: CPL0 vs. CPL3, initial program
counter, etc.
>>>
>>> - the communication channel; does the enclave see the usual
local APIC
>>> and IOAPIC interfaces in order to get interrupts from
virtio-vsock, and
>>> where is the virtio-vsock device (virtio-mmio I suppose)
placed in memory?
>>>
>>> - what the enclave is allowed to do: can it change privilege
levels,
>>> what happens if the enclave performs an access to nonexistent
memory,
>> etc.
>>> - whether there are special hypercall interfaces for the enclave
>> An enclave is a VM, running on the same host as the primary VM,
that
>> launched the enclave. They are siblings.
>>
>> Here we need to think of two components:
>>
>> 1. An enclave abstraction process - a process running in the
primary VM
>> guest, that uses the provided ioctl interface of the Nitro Enclaves
>> kernel driver to spawn an enclave VM (that's 2 below).
>>
>> How does all gets to an enclave VM running on the host?
>>
>> There is a Nitro Enclaves emulated PCI device exposed to the
primary VM.
>> The driver for this new PCI device is included in the current
patch series.
>>
>> The ioctl logic is mapped to PCI device commands e.g. the
>> NE_ENCLAVE_START ioctl maps to an enclave start PCI command or the
>> KVM_SET_USER_MEMORY_REGION maps to an add memory PCI command.
>> The PCI
>> device commands are then translated into actions taken on the
hypervisor
>> side; that's the Nitro hypervisor running on the host where the
primary
>> VM is running.
>>
>> 2. The enclave itself - a VM running on the same host as the
primary VM
>> that spawned it.
>>
>> The enclave VM has no persistent storage or network interface
attached,
>> it uses its own memory and CPUs + its virtio-vsock emulated
device for
>> communication with the primary VM.
> sounds like a firecracker VM?

It's a VM crafted for enclave needs.

>
>> The memory and CPUs are carved out of the primary VM, they are
dedicated
>> for the enclave. The Nitro hypervisor running on the host
ensures memory
>> and CPU isolation between the primary VM and the enclave VM.
> In last paragraph, you said that the enclave VM uses its own
memory and
> CPUs. Then here, you said the memory/CPUs are carved out and
dedicated
> from the primary VM. Can you elaborate which one is accurate? or
a mixed
> model?

Memory and CPUs are carved out of the primary VM and are dedicated
for
the enclave VM. I mentioned above as "its own" in the sense that the
primary VM doesn't use these carved out resources while the
enclave is
running, as they are dedicated to the enclave.

Hope that now it's more clear.

>
>>
>> These two components need to reflect the same state e.g. when the
>> enclave abstraction process (1) is terminated, the enclave VM
(2) is
>> terminated as well.
>>
>> With regard to the communication channel, the primary VM has
its own
>> emulated virtio-vsock PCI device. The enclave VM has its own
emulated
>> virtio-vsock device as well. This channel is used, for example,
to fetch
>> data in the enclave and then process it. An application that
sets up the
>> vsock socket and connects or listens, depending on the use
case, is then
>> developed to use this channel; this happens on both ends -
primary VM
>> and enclave VM.
> How does the application in the primary VM assign task to be
executed
> in the enclave VM? I didn't see such command in this series, so
suppose
> it is also communicated through virtio-vsock?

The application that runs in the enclave needs to be packaged in an
enclave image together with the OS ( e.g. kernel, ramdisk, init )
that
will run in the enclave VM.

Then the enclave image is loaded in memory. After booting is
finished,
the application starts. Now, depending on the app implementation
and use
case, one example can be that the app in the enclave waits for
data to
be fetched in via the vsock channel.


HiÂParaschiv,

So here the custom's application should be programmed to respect the enclave VM spec,
and can't be any binary, right? And also the application in enclave can't use any other IO
except the vsock?

Hi,

The application running in the enclave should be built so that it uses the available exposed functionality e.g. the vsock comm channel.

With regard to I/O, vsock is the means to interact with the primary / parent VM. The enclave VM doesn't have a network interface attached or persistent storage.

There is also an exposed device in the enclave, for the attestation flow e.g. to get the signed attestation document generated by the Nitro Hypervisor on the host where the primary VM and the enclave VM run.

From a previous mail thread on LKML, where I added a couple of clarifications on the attestation flow:

"

Hash values are computed for the entire enclave image (EIF), the kernel
and ramdisk(s). That's used, for example, to check that the enclave image
that is loaded in the enclave VM is the one that was intended to be run.

These crypto measurements are included in a signed attestation document
generated by the Nitro Hypervisor and further used to prove the identity
of the enclave. KMS is an example of service that NE is integrated with
and that checks the attestation doc.

"


Thanks,
Andra


>
>> Let me know if further clarifications are needed.
>>
>>>> The proposed solution is following the KVM model and uses the
KVM API
>> to be able
>>>> to create and set resources for enclaves. An additional ioctl
command,
>> besides
>>>> the ones provided by KVM, is used to start an enclave and
setup the
>> addressing
>>>> for the communication channel and an enclave unique id.
>>> Reusing some KVM ioctls is definitely a good idea, but I
wouldn't really
>>> say it's the KVM API since the VCPU file descriptor is
basically non
>>> functional (without KVM_RUN and mmap it's not really the KVM API).
>> It uses part of the KVM API or a set of KVM ioctls to model the
way a VM
>> is created / terminated. That's true, KVM_RUN and mmap-ing the
vcpu fd
>> are not included.
>>
>> Thanks for the feedback regarding the reuse of KVM ioctls.
>>
>> Andra
>>
> Thanks
> Kevin




Amazon Development Center (Romania) S.R.L. registered office: 27A
Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045,
Romania. Registered in Romania. Registration number J22/2621/2005.





Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.