Re: [RFC 00/19] KVM: s390/crypto/vfio: guest dedicated crypto adapters

[Tony Krowiak]

On 10/13/2017 01:38 PM, Tony Krowiak wrote:
Ping
> Overview:
> --------
> An adjunct processor (AP) facility is an IBM Z cryptographic facility. The
> AP facility is comprised of three AP instructions and from 1 to 256 AP
> adapter cards. The design takes advantage of the interpretive execution mode
> provided by the SIE architecture. With interpretive execution mode, the AP
> instructions executed on the guest are interpreted by the hardware. This
> allows guests direct access to AP adapter cards. The first goal of this
> patch series is to provide direct access by a KVM guest to an AP as a
> pass-through device. The second goal is to provide administrators with the
> means to configure KVM guests to grant direct access to AP facilities
> assigned to the LPAR in which the host linux system is running.
>
> To facilitate the comprehension of the design, let's present an overview of
> the AP architecture.
>
> AP Architectural Overview
> -------------------------
> Let's start with some definitions:
>
> * AP adapter
>
>    An AP adapter is an IBM Z adapter card that can perform cryptographic
>    functionality. There can be from 0 to 256 adapters assigned to an LPAR.
>    Each adapter is identified by a number from 0 to 255.   When
>    installed, an AP is accessed by AP instructions executed by any CPU.
>
> * AP domain
>
>    An adapter can be partitioned into domains. An adapter can hold up to 256
>    domains. Each domain is identified by a number from 0 to 255. Domains can
>    be further classified into two types:
>    
>      * Usage domains are domains that can be accessed directly to process AP
>        commands
>    
>      * Control domains are domains that are accessed indirectly by AP
>        commands sent to a usage domain to control or change the domain.
>
> * AP Queue
>
>    An AP queue is the means by which an AP command is sent to an
>    AP usage domain inside a specific AP. An AP queue is identified by a tuple
>    comprised of an AP adapter ID and a usage domain index corresponding
>    to a given usage domain within the adapter. This tuple forms an AP Queue
>    Number (APQN) uniquely identifying an AP queue. AP instructions include
>    a field containing the APQN to identify the AP queue to which the AP
>    command is targetted.
>
> * AP Instructions:
>
>    There are three AP instructions:
>
>    * NQAP: to enqueue an AP command-request message to a queue
>    * DQAP: to dequeue an AP command-reply message from a queue
>    * PQAP: to adminster the queues
>
> Let's now see how AP instructions are interpreted by the hardware.
>
> Start Interpretive Execution (SIE) Instruction
> ----------------------------------------------
> A KVM guest is started by executing the Start Interpretive Execution (SIE)
> instruction. The SIE state description is a control block that contains the
> state information for a KVM guest and is supplied as input to the SIE
> instruction. The SIE state description contains a field that references
> a Crypto Control Block (CRYCB). The CRYCB contains three bitmask fields
> identifying the adapters, usage domains and control domains assigned to the
> KVM guest:
>
> * The AP Mask (APM) field specifies the AP adapters assigned to the
>    KVM guest. The APM controls which adapters are valid for the KVM guest.
>    The bits in the mask, from left to right, correspond to APIDs
>    0 up to the number of adapters that can be assigned to the LPAR. If a bit
>    is set, the corresponding adapter is valid for use by the KVM guest.
>
> * The AP Queue Mask (AQM) field specifies the AP usage domains assigned
>    to the KVM guest. The bits in the mask, from left to right, correspond
>    to the usage domains, from 0 up to the number of domains that can be
>    assigned to the LPAR. If a bit is set, the corresponding usage domain is
>    valid for use by the KVM guest.
>
> * The AP Domain Mask field specifies the AP control domains assigned to the
>    KVM guest. The ADM bitmask controls which domains can be changed by an AP
>    command-request message sent to a usage domain from the guest. The bits in
>    the mask, from left to right, correspond to domain 0 up to the number of
>    domains that can be assigned to the LPAR. If a bit is set, the
>    corresponding domain can be modified by an AP command-request message
>    sent to a usage domain configured for the KVM guest.
>
> If you recall from the description of an AP Queue, AP instructions include
> an APQN to identify the AP adapter and the specific usage domain within
> the adapter to which an AP command-request message is to be sent (NQAP
> and PQAP instructions), or from which a command-reply message is to be
> received (DQAP instruction). The validity of an APQN is defined by the
> matrix calculated from the APM and AQM; it is the intersection of all
> assigned adapter numbers (APM) with all assigned usage domain numbers (AQM).
> For example, if adapters 1 and 2 and usage domains 5 and 6 are assigned to
> a guest, the APQNs (1,5), (1,6), (2,5) and (2,6) will be valid for the
> guest.
>
> The APQNs provide secure key functionality - i.e., the key is stored on the
> adapter card - so when the adapter card is not virtualized - i.e., the
> adapter is accessed directly by the guest - each APQN must be assigned to
> at most one guest.
>
>     Example 1: Valid configuration:
>     ------------------------------
>     Guest1: adapters 1,2  domains 5,6
>     Guest2: adapter  1,2  domain 7
>
>     This is valid because both guests have a unique set of APQNs: Guest1 has
>     APQNs (1,5), (1,6), (2,5) and (2,6); Guest2 has APQN (1,7) and (2,7).
>
>     Example 2: Invalid configuration:
>     --------------------------------
>     Guest1: adapters 1,2  domains 5,6
>     Guest2: adapter  1    domains 6,7
>
>     This is an invalid configuration because both guests have access to
>     APQNs (1,6).
>
> Interruption architecture:
>
> The AP interruption architecture may or may not generate interruptions to
> signal to the CPU the end of an AP transaction. The SIE interruption
> architecture, depending upon its configuration, may or may not redirect
> AP interrupts directly to a guest if the associated queue is valid for a
> guest, and may or may not report the interruption to the host.
>
> Effective masking for guest level I and II:
>
> A linux host running in the LPAR operates at guest-level 1 and has its own
> SIE state description. When operating at guest-level 1, the masks from the
> host's state description are used directly. A linux guest running in the
> host operates at guest-level 2. When operating at guest-level 2, the masks
> from the guest-level 1 (host) and guest-level 2 (guest) state descriptions
> are combined into a single description called an effective mask by
> performing a logical AND of the two state descriptions.
>
> The effective mask algorithm is used for the APM, AQM and ADM to create
> an EAPM, EAQM and EADM respectively. Use of the EAPM, EAQM and EADM
> precludes a guest-level 1 host program from passing to a guest-level 2
> program APQNs to which it does not have access.
>
> Linux cryptographic bus driver:
>
> Linux already has a cryptographic bus driver that provides one AP device per
> AP adapter and one device per AP queue. There is a device driver for each
> type of AP adapter device and each type of AP queue device. This design
> utilizes some of the interfaces and functionality provided by the AP bus
> driver.
>
> Design Origin:
> -------------
>
> The original design was based on modelling AP Queue devices. The design
> utilized the VFIO mediated device framework whereby a mediated AP queue
> device would be created for each AP Queue bound to the VFIO AP Queue device
> driver. This at first seemed like the most logical design choice for the
> following reasons:
>
> * Securing access to an AP Queue device by unbinding it from its default
>    device driver and binding it to the VFIO device driver would not preclude
>    the host from having access to the other usage domains contained within
>    the same adapter card connected to the AP queue.
>
> * An AP command is sent to a usage domain within a specific AP adapter via
>    an AP queue.
>
> It became readily apparent that modelling the design on an AP queue was very
> convoluted for a number of reasons:
>
>    * There is no convenient way to notify the VFIO device driver which guest
>      will have access to a given mediated AP queue device until the mediated
>      device's file descriptor is opened by the guest. Recall that the APQNs
>      configured for the guest are an intersection of all of the bits set in
>      both the APM and AQM, so the guest's APQNs can not be validated nor
>      its SIE state description configured until all of the guest's mediated
>      AP queue device file descriptors have been opened.
>
>      For example, suppose a guest opens file descriptors for mediated AP
>      queue devices representing APQNs 3,5 and 4,6. If bits 3 and 4 are set in
>      the guest's APM and bits 5 and 6 are set in the guest's AQM, then APQNs
>      (3,5), (3,6), (4,5) and (4,6) will be valid for the guest, but mediated
>      AP queue devices have been created only for APQNs (3,5) and (4,6). In
>      this case, APQNs still assigned to the host would also be available to
>      the guest which is a potential security breach.
>
>    * Control domains are not devices and are not logically modelled as
>      mediated devices. In our original design, they were modelled as
>      attributes of a mediated AP queue device, but this was a clumsy use of
>      the VFIO mediated device model.
>
>    * The SIE state description models the assignment of AP resources as a
>      matrix via the APM, AQM and ADM.
>       
> The design we ultimately settled upon was modelled on the AP matrix as
> defined by the SIE state description. Supplying the complete AP matrix
> to SIE using bitmasks when starting a guest simplifies the code, is far
> easier to secure, and more closely matches the model employed by SIE. This
> is the design model implemented via this patch set.
>
> The Design
> ----------
> This design introduces four new objects:
>
> 1. AP matrix bus
>
>     The sysfs location of the AP matrix bus is /sys/bus/ap_matrix. This
>     bus will create a single AP matrix device (see below).
>
> 2. AP matrix device
>
>     The AP matrix device is a singleton that hangs off of the AP matrix bus.
>     This device holds the AP Queues that have been reserved for use by
>     KVM guests. The sysfs location of the AP matrix device is
>     /sys/devices/ap_matrix/matrix. It is also linked from the AP matrix
>     bus at /sys/bus/ap_matrix/devices/matrix.
>
> 3. VFIO AP matrix driver
>
>     This driver is based on the VFIO mediated device framework. When the
>     driver is initialized, it will:
>
>     * Get the AP matrix device created by AP matrix bus from the bus
>
>     * Register with the AP bus to indicate that it can control AP Queue
>       devices. This allows AP Queue devices unbound from AP device drivers
>       to be bound to the VFIO AP matrix driver. The AP Queues bound to the
>       VFIO AP matrix driver will be stored by the driver in the AP matrix
>       device.
>
>     * Register the AP matrix device with the VFIO mediated device
>       framework (MDEV). Registration with MDEV will create the sysfs
>       structures needed to create mediated matrix devices. Each MDEV matrix
>       device is used to configure the AP matrix for a KVM guest. The MDEV
>       matrix device's file descriptor can be used by QEMU to communicate
>       with the VFIO AP matrix device driver.
>
>     The VFIO AP matrix driver:
>
>     * Provides the interfaces the administrator can use to secure AP Queues
>       for use by KVM guests. This is accomplished by unbinding the AP Queues
>       needed by each KVM guest from its AP device driver and binding it to
>       the VFIO AP queue driver. This prevents the host linux system from
>       using these Queues.
>
>     * Provides an ioctl that can be used by QEMU to configure the
>       CRYCB referenced by the KVM guest's SIE state description. The ioctl
>       will
>
>       * Create an EAPM, EAQM and EADM by performing a logical AND of the
>         APM, AQM and ADM configured via the MDEV matrix device's sysfs
>         attributes files (see below) with the APM, AQM and ADM of the host's
>         SIE state description respectively.
>
>       * Configure the SIE state description for the KVM guest using the
>         effective masks created in the previous step.
>
> 4. VFIO MDEV matrix passthrough device
>
>     An MDEV matrix passthrough device must be created for each KVM guest that
>     will need access to AP facilities. An MDEV matrix passthrough device is
>     used by QEMU to configure the APM, AQM and ADM fields of the CRYCB
>     referenced by the KVM guest's SIE state description. The file descriptor
>     for the MDEV matrix passthrough device provides the communication pathway
>     between QEMU and the VFIO AP matrix device driver.
>
>     The MDEV matrix passthrough device, like the CRYCB, contains three
>     bitmasks - an APM, AQM and ADM - for specifying the AP matrix for the
>     KVM guest. Three sets of attributes files will be provided to allow an
>     administrator to set the bits in the MDEV matrix device's APM, AQM and
>     ADM:
>
>     * A file to assign an AP adapter
>     * A file to unassign an AP adapter
>     * A file to display the adapters assigned
>
>     * A file to assign an AP domain
>     * A file to unassign an AP domain
>     * A file to display the domains assigned
>
>     * A file to assign an AP control domain
>     * A file to unassign an AP control domain
>     * A file to display the control domains assigned
>
> Example:
> -------
> Let's now provide an example to illustrate how KVM guests may be given
> access to AP facilities. For this example, we will show how to configure
> two guests such that executing the lszcrypt command on the guests would
> look like this:
>
> Guest1
> ------
> CARD.DOMAIN TYPE  MODE
> ------------------------------
> 05          CEX5C CCA-Coproc
> 05.0004     CEX5C CCA-Coproc
> 05.00ab     CEX5C CCA-Coproc
> 06          CEX5A Accelerator
> 06.0004     CEX5A Accelerator
> 06.00ab     CEX5C CCA-Coproc
>
> Guest2
> ------
> CARD.DOMAIN TYPE  MODE
> ------------------------------
> 05          CEX5A Accelerator
> 05.0047     CEX5A Accelerator
> 05.00ff     CEX5A Accelerator
>
> One thing to notice in this example is that each AP Queue set is identical.
> For example, the two AP Queue sets for Guest1 both contain APQI 0004 and
> 00ab. It would be an invalid condition if both queue sets did not contain
> the same set of queues. We could not, for example, configure Guest1 with
> access to AP queue 05.00ff because the AP queue set for adapter 06 does not
> contain AP queue 06.00ff. The point is, one must be careful to reserve
> a valid set of AP queues for a given guest.
> a valid configuration.
>
> These are the steps for configuring the Guest1 and Guest2:
>     
> 1. The first thing that needs to be done is to secure the AP queues to be
>     used by the two guests so that the host can not access them. This is done
>     by unbinding each AP Queue device from its respective AP driver. In our
>     example, these queues are bound to the cex4queue driver. This would be
>     the sysfs location of these devices:
>
>     /sys/bus/ap
>     --- [drivers]
>     ------ [cex4queue]
>     --------- [05.0004]
>     --------- [05.0047]
>     --------- [05.00ab]
>     --------- [05.00ff]
>     --------- [06.0004]
>     --------- [06.00ab]
>     --------- unbind
>
>     To unbind AP queue 05.0004 from the cex4queue device driver:
>
> 	echo 05.0004 > unbind
>
>     This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
>     and 06.00ab.
>
> 2. The next step is to reserve the queues for use by the two KVM guests.
>     This is accomplished by binding them to the VFIO AP matrix device driver.
>     This is the sysfs location of the VFIO AP matrix device driver:
>
>     /sys/bus/ap
>     ---[drivers]
>     ------ [vfio_ap_matrix]
>     ---------- bind
>
>     To bind queue 05.0004 to the vfio_ap_matrix driver:
>
> 	echo 05.0004 > bind
>
>     This must also be done for AP queues 05.00ab, 05.0047, 05.00ff, 06.0004,
>     and 06.00ab.
>
> 3. Create the mediated devices needed to configure the AP matrices for the
>     two guests and to provide an interface to the vfio_ap_matrix driver for
>     use by the guests:
>
>     /sys/devices/
>     --- [ap_matrix]
>     ------ [matrix] (this is the matrix device)
>     --------- [mdev_supported_types]
>     ------------ [ap_matrix-passthrough] (passthrough mediated device type)
>     --------------- create
>     --------------- [devices]
>
>     To create the mediated devices for the two guests:
>
> 	uuidgen > create
> 	uuidgen > create
>
>     This will create two mediated devices in the [devices] subdirectory named
>     with the UUID written to the create attribute file. We call them $uuid1
>     and $uuid2:
>
>     /sys/devices/
>     --- [ap_matrix]
>     ------ [matrix]
>     --------- [mdev_supported_types]
>     ------------ [ap_matrix-passthrough]
>     --------------- [devices]
>     ------------------ [$uuid1]
>     --------------------- adapters
>     --------------------- assign_adapter
>     --------------------- assign_control_domain
>     --------------------- assign_domain
>     --------------------- control_domains
>     --------------------- domains
>     --------------------- unassign_adapter
>     --------------------- unassign_control_domain
>     --------------------- unassign_domain
>     ------------------ [$uuid2]
>     --------------------- adapters
>     --------------------- assign_adapter
>     --------------------- assign_control_domain
>     --------------------- assign_domain
>     --------------------- control_domains
>     --------------------- domains
>     --------------------- unassign_adapter
>     --------------------- unassign_control_domain
>     --------------------- unassign_domain
>
> 4. The administrator now needs to configure the matrices for mediated
>     devices $uuid1 (for Guest1) and $uuid2 (for Guest2).
>
>     This is how the matrix is configured for Guest1:
>
>     echo 5 > assign_adapter
>     echo 6 > assign_adapter
>     echo 4 > assign_domain
>     echo ab > assign_domain
>
>     When the assign.xxx file is written, the corresponding bit in the
>     respective MDEV matrix device's bitmask will be set. For example, when
>     adapter 5 is assigned, bit 5 - numbered from left to right starting with
>     bit 0 - will be set in the MDEV matrix device's APM.
>
>     By architectural convention, all usage domains - i.e., domains assigned
>     via the assign_domain attribute file - will also be configured in the ADM
>     field of the KVM guest's CRYCB, so there is no need to assign control
>     domains here unless you want to assign control domains that are not
>     assigned as usage domains.
>
>     If a mistake is made configuring an adapter, domain or control domain,
>     you can use the unassign_xxx files to unassign the adapter, domain or
>     control domain.
>
>     To display the matrix configuration for Guest1:
>
>     cat adapters
>     cat domains
>     cat control_domains
>
>     This is how the matrix is configured for Guest2:
>
>     echo 5 > assign_adapter
>     echo 47 > assign_domain
>     echo ff > assign_domain
>
> When a KVM guest is started, QEMU will open the file descriptor for its
> MDEV matrix device. The VFIO AP matrix device driver will be notified
> and will store the reference to the KVM guest's SIE state description.
> QEMU will then call the VFIO AP matrix ioctl requesting that the
> KVM guest's matrix be configured. The matrix driver will set the bits in the
> APM, AQM and ADM fields of the CRYCB referenced by the guest's SIE state
> description from the EAPM, EAQM and EADM created by performing a logical AND
> of the AP masks configured in the MDEV matrix device and the masks
> configured in the host's SIE state description. When the guest comes up, it
> will have access to the APQNs identified in the AP matrix specified in the
> KVM guest's SIE state description. Programs running on the guest will then
> be able to use the cryptographic functions provided by the AP facilities
> configured for the guest.
>
> Tony Krowiak (19):
>    KVM: s390: SIE considerations for AP Queue virtualization
>    KVM: s390: refactor crypto initialization
>    s390/zcrypt: new AP matrix bus
>    s390/zcrypt: create an AP matrix device on the AP matrix bus
>    s390/zcrypt: base implementation of AP matrix device driver
>    s390/zcrypt: register matrix device with VFIO mediated device
>      framework
>    KVM: s390: introduce AP matrix configuration interface
>    s390/zcrypt: support for assigning adapters to matrix mdev
>    s390/zcrypt: validate adapter assignment
>    s390/zcrypt: sysfs interfaces supporting AP domain assignment
>    s390/zcrypt: validate domain assignment
>    s390/zcrypt: sysfs support for control domain assignment
>    s390/zcrypt: validate control domain assignment
>    KVM: s390: Connect the AP mediated matrix device to KVM
>    s390/zcrypt: introduce ioctl access to VFIO AP Matrix driver
>    KVM: s390: interface to configure KVM guest's AP matrix
>    KVM: s390: validate input to AP matrix config interface
>    KVM: s390: New ioctl to configure KVM guest's AP matrix
>    s390/facilities: enable AP facilities needed by guest
>
>   MAINTAINERS                                  |   13 +
>   arch/s390/Kconfig                            |   13 +
>   arch/s390/configs/default_defconfig          |    1 +
>   arch/s390/configs/gcov_defconfig             |    1 +
>   arch/s390/configs/performance_defconfig      |    1 +
>   arch/s390/defconfig                          |    1 +
>   arch/s390/include/asm/ap-config.h            |   32 +
>   arch/s390/include/asm/kvm_host.h             |   26 +-
>   arch/s390/kvm/Makefile                       |    2 +-
>   arch/s390/kvm/ap-config.c                    |  224 ++++++++
>   arch/s390/kvm/kvm-s390.c                     |   17 +-
>   arch/s390/tools/gen_facilities.c             |    2 +
>   drivers/s390/crypto/Makefile                 |    6 +-
>   drivers/s390/crypto/ap_matrix_bus.c          |  115 ++++
>   drivers/s390/crypto/ap_matrix_bus.h          |   25 +
>   drivers/s390/crypto/vfio_ap_matrix_drv.c     |  107 ++++
>   drivers/s390/crypto/vfio_ap_matrix_ops.c     |  790 ++++++++++++++++++++++++++
>   drivers/s390/crypto/vfio_ap_matrix_private.h |   50 ++
>   include/uapi/linux/vfio.h                    |   22 +
>   19 files changed, 1438 insertions(+), 10 deletions(-)
>   create mode 100644 arch/s390/include/asm/ap-config.h
>   create mode 100644 arch/s390/kvm/ap-config.c
>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.c
>   create mode 100644 drivers/s390/crypto/ap_matrix_bus.h
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_drv.c
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_ops.c
>   create mode 100644 drivers/s390/crypto/vfio_ap_matrix_private.h