[PATCH v2 7/7] s390: vfio-ap: update documentation

From: Tony Krowiak
Date: Fri May 03 2019 - 17:15:47 EST

This patch updates the vfio-ap documentation to include the information
below.

Changes made to the mdev matrix assignment interfaces:

* We now allow assignment of APQNs that are not bound to the vfio-ap
device driver

* We now use assignment interfaces to hot plug AP resources into a
a running guest using the mdev matrix device assignment interfaces

* We now use unassignment interfaces to hot unplug AP resources from
a running guest using the mdev matrix device unassignment interfaces

Changes made to the vfio_ap device driver probe and remove callbacks:

* We now plug the queue being bound to the vfio_ap into the running
guest if the APQN of the queue is assigned to the mdev matrix device
in use by the guest.

* If a guest is using the queue being unbound from the vfio_ap device
driver, we now unplug the adapter card with the APID of the queue being
unbound from the guest.

This patch also clarifies the section on configuring the AP bus's apmask
and aqmask.

Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
---
Documentation/s390/vfio-ap.txt | 191 +++++++++++++++++++++++++++--------------
1 file changed, 127 insertions(+), 64 deletions(-)

diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
index 65167cfe4485..ca6aa8eeff8a 100644
--- a/Documentation/s390/vfio-ap.txt
+++ b/Documentation/s390/vfio-ap.txt
@@ -81,10 +81,19 @@ definitions:
which the AP command is to be sent for processing.

The AP bus will create a sysfs device for each APQN that can be derived from
- the cross product of the AP adapter and usage domain numbers detected when the
- AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
- domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
- following sysfs entries:
+ the Cartesian product of the AP adapter and usage domain numbers detected when
+ the AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and
+ usage domains 6 and 71 (0x47) are assigned to the LPAR, the Cartesian product
+ would be defined by the following table:
+
+ 06 71
+ +-----------+-----------+
+ 04 | (04,06) | (04,47) |
+ +-----------|-----------+
+ 10 | (0a,06) | (0a,47) |
+ +-----------|-----------+
+
+ The AP bus will create the following sysfs entries:

/sys/devices/ap/card04/04.0006
/sys/devices/ap/card04/04.0047
@@ -146,10 +155,20 @@ If you recall from the description of an AP Queue, AP instructions include
an APQN to identify the AP queue to which an AP command-request message is to be
sent (NQAP and PQAP instructions), or from which a command-reply message is to
be received (DQAP instruction). The validity of an APQN is defined by the matrix
-calculated from the APM and AQM; it is the cross product of all assigned adapter
-numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1
-and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6),
-(2,5) and (2,6) will be valid for the guest.
+calculated from the APM and AQM; it is the Cartesian product of all assigned
+adapter numbers (APM) with all assigned queue indexes (AQM). For example, if
+adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest:
+
+
+ 05 06
+ +-----------+-----------+
+ 01 | (01,05) | (01,06) |
+ +-----------|-----------+
+ 02 | (02,05) | (02,06) |
+ +-----------|-----------+
+
+
+APQNs (01,05), (01,06), (02,05) and (02,06) will be valid for the guest.

The APQNs can provide secure key functionality - i.e., a private key is stored
on the adapter card for each of its domains - so each APQN must be assigned to
@@ -349,8 +368,9 @@ matrix device.
number of the the usage domain is echoed to the respective attribute
file.
* matrix:
- A read-only file for displaying the APQNs derived from the cross product
- of the adapter and domain numbers assigned to the mediated matrix device.
+ A read-only file for displaying the APQNs derived from the Caresian
+ product of the adapter and domain numbers assigned to the mediated matrix
+ device.
* assign_control_domain:
* unassign_control_domain:
Write-only attributes for assigning/unassigning an AP control domain
@@ -513,35 +533,44 @@ These are the steps:
/sys/bus/ap/aqmask

The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
- (APID). Each bit in the mask, from left to right (i.e., from most significant
- to least significant bit in big endian order), corresponds to an APID from
- 0-255. If a bit is set, the APID is marked as usable only by the default AP
- queue device drivers; otherwise, the APID is usable by the vfio_ap
- device driver.
+ (APID). Each bit in the mask, from left to right, corresponds to an APID from
+ 0-255.

The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
- (APQI). Each bit in the mask, from left to right (i.e., from most significant
- to least significant bit in big endian order), corresponds to an APQI from
- 0-255. If a bit is set, the APQI is marked as usable only by the default AP
- queue device drivers; otherwise, the APQI is usable by the vfio_ap device
- driver.
+ (APQI). Each bit in the mask, from left to right, corresponds to an APQI from
+ 0-255.

- Take, for example, the following mask:
+ The Cartesian product of the APIDs set in the apmask and the APQIs set in
+ the aqmask identify the APQNs of AP queue devices owned by the zcrypt
+ device drivers.

- 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
+ Take, for example, the following masks:

- It indicates:
+ apmask: 0x0700000000000000000000000000000000000000000000000000000000000000

- 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
- belong to the vfio_ap device driver's pool.
+ aqmask: 0x0180000000000000000000000000000000000000000000000000000000000000

- The APQN of each AP queue device assigned to the linux host is checked by the
- AP bus against the set of APQNs derived from the cross product of APIDs
- and APQIs marked as usable only by the default AP queue device drivers. If a
- match is detected, only the default AP queue device drivers will be probed;
- otherwise, the vfio_ap device driver will be probed.
+ The bits set in apmask are bits 1, 2 and 3. The bits set in aqmask are bits
+ 7 and 8. The Cartesian product of the bits set in the two masks is:

- By default, the two masks are set to reserve all APQNs for use by the default
+ 07 08
+ +-----------+-----------+
+ 01 | (01,07) | (01,08) |
+ +-----------|-----------+
+ 02 | (02,07) | (02,08) |
+ +-----------|-----------+
+ 03 | (03,07) | (03,08) |
+ +-----------|-----------+
+
+ The masks indicate that the queues with APQNs (01,07), (01,08), (02,07),
+ (02,08), (03,07) and (03,08) are owned by the zcrypt drivers. When the AP bus
+ detects an AP queue device, its APQN is checked against the set of APQNs
+ derived from the apmask and aqmask. If a match is detected, the zcrypt
+ device driver registered for the device type of the queue will be probed. If
+ a match is not detected and the device type of the queue is CEX4 or newer,
+ the vfio_ap device driver will be probed.
+
+ By default, the two masks are set to reserve all APQNs for use by the zcrypt
AP queue device drivers. There are two ways the default masks can be changed:

1. The sysfs mask files can be edited by echoing a string into the
@@ -554,8 +583,7 @@ These are the steps:

0x4100000000000000000000000000000000000000000000000000000000000000

- Keep in mind that the mask reads from left to right (i.e., most
- significant to least significant bit in big endian order), so the mask
+ Keep in mind that the mask reads from left to right, so the mask
above identifies device numbers 1 and 7 (01000001).

If the string is longer than the mask, the operation is terminated with
@@ -563,7 +591,7 @@ These are the steps:

* Individual bits in the mask can be switched on and off by specifying
each bit number to be switched in a comma separated list. Each bit
- number string must be prepended with a ('+') or minus ('-') to indicate
+ number string must be prefixed with a ('+') or minus ('-') to indicate
the corresponding bit is to be switched on ('+') or off ('-'). Some
valid values are:

@@ -594,11 +622,6 @@ These are the steps:
aqmask:
0x4000000000000000000000000000000000000000000000000000000000000000

- Resulting in these two pools:
-
- default drivers pool: adapter 0-15, domain 1
- alternate drivers pool: adapter 16-255, domains 0, 2-255
-
Securing the APQNs for our example:
----------------------------------
To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
@@ -747,14 +770,13 @@ These are the steps:
higher than the maximum is specified, the operation will terminate with
an error (ENODEV).

- * All APQNs that can be derived from the adapter ID and the IDs of
- the previously assigned domains must be bound to the vfio_ap device
- driver. If no domains have yet been assigned, then there must be at least
- one APQN with the specified APID bound to the vfio_ap driver. If no such
- APQNs are bound to the driver, the operation will terminate with an
- error (EADDRNOTAVAIL).
+ * Each APQN that can be derived from the adapter ID and the IDs of
+ the previously assigned domains must not be reserved for use by the
+ zcrypt device drivers as specified by the /sys/bus/ap/apmask and
+ /sys/bus/ap/aqmask syfs interfaces. If any APQN is reserved, the operation
+ will terminate with an error (EADDRNOTAVAIL).

- No APQN that can be derived from the adapter ID and the IDs of the
+ * No APQN that can be derived from the adapter ID and the IDs of the
previously assigned domains can be assigned to another mediated matrix
device. If an APQN is assigned to another mediated matrix device, the
operation will terminate with an error (EADDRINUSE).
@@ -766,14 +788,13 @@ These are the steps:
higher than the maximum is specified, the operation will terminate with
an error (ENODEV).

- * All APQNs that can be derived from the domain ID and the IDs of
- the previously assigned adapters must be bound to the vfio_ap device
- driver. If no domains have yet been assigned, then there must be at least
- one APQN with the specified APQI bound to the vfio_ap driver. If no such
- APQNs are bound to the driver, the operation will terminate with an
- error (EADDRNOTAVAIL).
+ * Each APQN that can be derived from the domain ID and the IDs of
+ the previously assigned adapters must not be reserved for use by the
+ zcrypt device drivers as specified by the /sys/bus/ap/apmask and
+ /sys/bus/ap/aqmask syfs interfaces. If any APQN is reserved, the operation
+ will terminate with an error (EADDRNOTAVAIL).

- No APQN that can be derived from the domain ID and the IDs of the
+ * No APQN that can be derived from the domain ID and the IDs of the
previously assigned adapters can be assigned to another mediated matrix
device. If an APQN is assigned to another mediated matrix device, the
operation will terminate with an error (EADDRINUSE).
@@ -822,16 +843,58 @@ Using our example again, to remove the mediated matrix device $uuid1:
host. If the mdev matrix device is removed, one may want to also reconfigure
the pool of adapters and queues reserved for use by the default drivers.

-Limitations
-===========
-* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
- to the default drivers pool of a queue that is still assigned to a mediated
- device in use by a guest. It is incumbent upon the administrator to
- ensure there is no mediated device in use by a guest to which the APQN is
- assigned lest the host be given access to the private data of the AP queue
- device such as a private key configured specifically for the guest.
+Hot plug/unplug via mdev matrix device sysfs interfaces:
+=======================================================
+If an mdev matrix device is in use by a running guest, AP resources can be
+plugged into or unplugged from the guest via the mdev device's sysfs
+assignment interfaces. Below are some examples.
+
+ To plug adapter 10 into a running guest:
+
+ echo 0xa > assign_adapter
+
+ To unplug domain 5 from a running guest:
+
+ echo 5 > unassign_domain
+
+To display the matrix of a guest using the mdev matrix device:
+
+ cat guest_matrix
+
+If you attempt to display the guest matrix when a guest is not using the
+mdev matrix device, an error will be displayed (ENODEV).
+
+Considerations for binding and unbinding AP queue devices:
+=========================================================
+There are considerations for binding AP queue devices to and unbinding AP queue
+devices from the vfio_ap device driver. Keep in mind that binding/unbinding of
+AP queue devices is controlled by the two AP bus sysfs interfaces:
+
+ /sys/bus/ap/apmask
+ /sys/bus/ap/aqmask

-* Dynamically modifying the AP matrix for a running guest (which would amount to
- hot(un)plug of AP devices for the guest) is currently not supported
+Binding considerations:
+----------------------
+When an AP queue device is bound to the vfio_ap device driver and its APQN is
+assigned to an mdev matrix device in use by a running guest, the driver will
+hot plug the queue into the guest as long as each APQN that can be derived from
+the APID of the queue device and the APQIs of the queues already in use by the
+guest are bound to the vfio_ap device driver.
+
+Unbinding considerations:
+------------------------
+When an AP queue device being used by a running guest is unbound from the
+vfio_ap device driver, the adapter card to which the queue is connected will
+be unplugged from the guest.
+
+Note: The AP architecture does not provide a way to unplug an individual queue.
+
+Live migration:
+==============
+Live guest migration is not supported for guests using AP devices. All AP
+devices in use by the guest must be unplugged prior to initiating live
+migration (see "Hot plug/unplug via mdev matrix device sysfs interfaces" section
+above). If you are using QEMU to run your guest and it supports hot plug/unplug
+of the vfio-ap device, this would be another option (consult the QEMU
+documentation for details).

-* Live guest migration is not supported for guests using AP devices.
--
2.7.4