[PATCH v3 15/15] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
From: Anthony Krowiak
Date: Tue Jun 30 2026 - 06:56:14 EST
Add a 'Live Guest Migration' chapter to the
Documentation/arch/s390/vfio-ap.rst document to describe the details for
initiating live guest migration for a guest to which AP adapters, domains
and control domains have been passed through.
Signed-off-by: Anthony Krowiak <akrowiak@xxxxxxxxxxxxx>
---
Documentation/arch/s390/vfio-ap.rst | 514 +++++++++++++++++++++++-----
1 file changed, 433 insertions(+), 81 deletions(-)
diff --git a/Documentation/arch/s390/vfio-ap.rst b/Documentation/arch/s390/vfio-ap.rst
index ac0c07f76ddd..fc0707c0dffd 100644
--- a/Documentation/arch/s390/vfio-ap.rst
+++ b/Documentation/arch/s390/vfio-ap.rst
@@ -1012,7 +1012,7 @@ guest_matrix dyn ap_config
the following features are advertised:
----------------+---------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
| Flag | Description |
+==============+===============================================================+
| guest_matrix | guest_matrix attribute exists. It reports the matrix of |
@@ -1021,105 +1021,457 @@ the following features are advertised:
+--------------+---------------------------------------------------------------+
| dyn | Indicates hot plug/unplug of AP adapters, domains and control |
| | domains for a guest to which the mdev is attached. |
-+------------+-----------------------------------------------------------------+
++--------------+---------------------------------------------------------------+
| ap_config | ap_config interface for one-shot modifications to mdev config |
+--------------+---------------------------------------------------------------+
+| migratable | Indicates that live guest migration is supported for guests |
+| | to which crypto devices are passed through |
++--------------+---------------------------------------------------------------+
-Limitations
-===========
-Live guest migration is not supported for guests using AP devices without
-intervention by a system administrator. Before a KVM guest can be migrated,
-the vfio_ap mediated device must be removed. Unfortunately, it can not be
-removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
-the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
-its mdev can be hot unplugged from the guest in one of two ways:
-
-1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
- the following commands:
-
- virsh detach-device <guestname> <path-to-device-xml>
-
- For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
- the guest named 'my-guest':
-
- virsh detach-device my-guest ~/config/my-guest-hostdev.xml
-
- The contents of my-guest-hostdev.xml:
-
-.. code-block:: xml
-
- <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
- <source>
- <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
- </source>
- </hostdev>
-
-
- virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
-
- For example, to hot unplug the vfio_ap mediated device identified on the
- qemu command line with 'id=hostdev0' from the guest named 'my-guest':
-
-.. code-block:: sh
-
- virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
-
-2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
- to the guest and using the following qemu monitor command:
+Live Guest Migration
+====================
+The VFIO AP mediated device is not used to provide userspace with direct
+access to a device as is the case with other devices that use the VFIO
+framework for device pass-through. Instead, it manages AP
+configuration metadata identifying the adapters, domains, and control
+domains to which a guest will be granted access. These AP resources are
+configured by assigning them to a vfio-ap mediated device via its sysfs
+assignment interfaces. When the guest is started, the vfio_ap device driver
+sets the guest's AP configuration from the metadata stored with the mediated
+device. The AP devices are not accessed directly through the vfio_ap driver,
+so the driver does not migrate the state of the AP devices themselves.
+Instead, it migrates the AP configuration metadata. The vfio_ap device
+driver on the destination host will then ensure that the AP configuration
+metadata migrated from the source host is compatible with the AP
+configuration available on the destination host and if
+so, will set the destination guest's AP configuration accordingly.
- (QEMU) device-del id=<device-id>
+To be considered compatible, the AP configuration for the destination host must
+meet these requirements:
- For example, to hot unplug the vfio_ap mediated device that was specified
- on the qemu command line with 'id=hostdev0' when the guest was started:
+* Each AP adapter, domain and control domain number assigned to the source
+ guest's AP configuration must be in the destination host's AP
+ configuration. This can be verified as follows:
- (QEMU) device-del id=hostdev0
+ * On the source host, display the AP configuration of the source guest:
-After live migration of the KVM guest completes, an AP configuration can be
-restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
-system into the guest in one of two ways:
+ .. code-block:: bash
-1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
- device into the guest via the following virsh commands:
+ cat /sys/devices/vfio_ap/matrix/$UUID/guest_matrix
+ cat /sys/devices/vfio_ap/matrix/$UUID/control_domains
- virsh attach-device <guestname> <path-to-device-xml>
+ .. note::
+ Replace ``$UUID`` with the actual UUID of your mediated device.
- For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
- the guest named 'my-guest':
+ * Display the AP configuration of the destination host:
- virsh attach-device my-guest ~/config/my-guest-hostdev.xml
+ .. code-block:: bash
- The contents of my-guest-hostdev.xml:
+ lszcrypt -V
+ cat /sys/bus/ap/ap_control_domain_mask
-.. code-block:: xml
+ Note that each bit in the ap_control_domain_mask corresponds to
+ correspond to domain numbers 0-255. If a bit is set to one, then
+ the control domain is available on the destination host. For
+ example:
- <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
- <source>
- <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
- </source>
- </hostdev>
+ .. code-block:: bash
+ 0x0000000000001020000000000000000000000000000000000000000000000000
- virsh qemu-monitor-command <guest-name> --hmp \
- "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+ This mask indicates that AP adapters 51 (0x33) and 58 (0x58) are
+ available on the host system.
- For example, to hot plug the vfio_ap mediated device
- 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
- device-id hostdev0:
+ * Verify the configurations:
- virsh qemu-monitor-command my-guest --hmp \
- "device_add vfio-ap,\
- sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
- id=hostdev0"
+ * Each APQN in the guest's sysfs ``guest_matrix`` file must match an
+ APQN returned from the lszcrypt command on the destination host. It
+ is important to note the driver to which the queue is bound which
+ is displayed in the ``driver`` column. Queues not bound to the vfio_ap
+ driver are not available to the destination guest. That, however, does
+ not guarantee they can be used by the guest being migrated; for example,
+ migration will fail if any of these APQNs are assigned to a mediated
+ device other than the one being migrated.
-2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
- to the guest and using the following qemu monitor command:
+ * Each domain number from the source guest's sysfs
+ ``control_domains`` attribute must correspond to a bit set to one in
+ the destination host's ``ap_control_domain_mask``.
- (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+* Each APQN assigned to the source guest and destination guest must
+ reference a queue device with compatible hardware capabilities:
- For example, to plug the vfio_ap mediated device
- 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
- hostdev0:
++--------------+---------------------------------------------------------------+
+| Hardware | Description |
+| Capabilities | |
++==============+===============================================================+
+| [1] | * AP special command facility (APSC) |
+| Facilities | * AP 4096-bit ME PKU commands facility (AP4KM) |
+| | * AP 4096-bit CRT PKU commands (AP4KC) |
++--------------+---------------------------------------------------------------+
+| [1] Mode | * CCA-mode |
+| | * Accelerator-mode |
+| | * XCP-mode (EP11) |
++--------------+---------------------------------------------------------------+
+| [1] AP | APXA installed |
+| extended | |
+| addressing | |
++--------------+---------------------------------------------------------------+
+| [1] Command | Command filtering available |
+| filtering | |
++--------------+---------------------------------------------------------------+
+| [2] | * Full native card function |
+| Functional | * Only stateless functions |
+| capabilities | |
++--------------+---------------------------------------------------------------+
+| [3] AP type | * 10 (0x0a): CEX4 |
+| | * 11 (0x0b): CEX5 |
+| | * 12 (0x0c) CEX6 |
+| | * 13 (0x0d) CEX7 |
+| | * 14 (0x0e) CEX8 |
++--------------+---------------------------------------------------------------+
- (QEMU) device-add "vfio-ap,\
- sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
- id=hostdev0"
++-----------------------------------------------------------------------------+
+| Table Keys |
++=============================================================================+
+| |
+| [1] The hardware capability must be the same for both the source |
+| and destination queue device. |
+| |
+| [2] The source and destination queues must either both have |
+| full native card function or both have stateless functions. |
+| If the functional capabilities don't match, then the source |
+| queue can have stateless capabilities since full native card |
+| function can handle stateless functions; otherwise, migration |
+| will fail. |
+| |
+| [3] The AP type on the source and destination guests can |
+| differ if the queue passed through to the target guest is a |
+| newer model (backwards compatible) |
++--------------+--------------------------------------------------------------+
+
+* To verify the hardware capabilities are compatible:
+
+ * On both the source and destination hosts, display the hardware capabilities
+ for each AP adapter in the AP configuration for that host::
+
+ cat /sys/bus/ap/devices/card$APID/ap_functions
+
+ .. note::
+ The ``$APID`` is the two-character adapter number in hexidecimal format;
+ for example, card02 or card1f. The ``/sys/bus/ap/devices`` directory
+ also lists the APQNs of the queue devices installed in the respective
+ host system with the first two characters being the APID.
+
+ * Verify the hardware capabilities for each AP adapter device on the source
+ and destination host are compatible. The ``ap_functions`` attribute is a
+ bitmask. The bits in the mask read from left to right starting with bit 0.
+ Each bit that is set to one indicates the corresponding hardware capability
+ is installed:
+
+ * Bits 0-3 are the facilities bits. Each bit value must match for the AP
+ devices on the source and destination systems. The values indicate the
+ following:
+ * bit-0: APSC is installed
+ * bit-1: AP4KM is installed
+ * bit-2: AP4KC is installed
+
+ * Bits 3-5 specify the adapter Modes. Only one of these bits will be set to
+ one and must be the same for both the source and destination adapter
+ devices:
+ * bit-3: CCA-mode
+ * bit-4: Accelerator-mode
+ * bit-5: XCP-mode (EP11)
+
+ * Bit 6 specifies whether APXA is installed and must be the same for both
+ the source and destination adapter devices.
+
+ * Bit 7 specifies command whether command filtering is installed
+ and must be the same for both the source and destination adapter devices.
+
+ * Bits 8-15 are the classification bits. There are only two relevant
+ bits and only one of them will be set to one. For the source and destination
+ adapter devices to be considered compatible:
+
+ * Bit 8 may be set to 1 on both hosts (full native card function)
+ * Bit 9 may be set to 1 on both hosts (only stateless functions available)
+ * Bit 9 may be set to 1 on the source host and bit 8 may be set to 1 on
+ the destination host
+
+ * Bits 16-17 are the binding state bits which must be 00
+
+* To verify the AP types are compatible:
+
+ * On both the source and destination hosts, display the hardware type
+ for each AP adapter in the AP configuration for that host to ensure the
+ source and destination adapter types are the same, or the source type is
+ less than the destination type::
+
+ cat /sys/bus/ap/devices/card$APID/hwtype
+
+ .. note::
+ The ``$APID`` is the two-character adapter number in hexidecimal format;
+ for example, card02 or card1f.
+
+Setting up for live guest migration on the destination host
+-----------------------------------------------------------
+To set up a host as the destination for live guest migration, do the
+following:
+
+Create a mediated device
+~~~~~~~~~~~~~~~~~~~~~~~~
+Create a mediated device with the same name (UUID) as the mediated device
+used to supply the AP configuration to the source guest. For example, if
+the mediated device on the source guest is named
+62177883-f1bb-47f0-914d-32a22e3a8804:
+
+.. code-block:: bash
+
+ echo 62177883-f1bb-47f0-914d-32a22e3a8804 > \
+ /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create
+
+Reserve adapters and domains for the vfio_ap device driver's use
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Reserve the same adapters and domains passed through to the source guest
+for the vfio_ap device driver on the destination host by clearing the
+adapter numbers from the ``/sys/bus/ap/ap_mask`` and/or the domain numbers
+from the ``/sys/bus/ap/aqmask`` files.
+
+For example, suppose adapters 0, 20 (0x14) and 22 (0x16) as well as
+domains 51 (0x33) and 58 (0x3a) are passed through to the source guest. To
+ensure they can be passed through to the destination guest, they must be
+made available to the vfio_ap device driver. To make them available, do one
+or both of the following on the destination host:
+
+.. code-block:: bash
+
+ echo -0,-0x14,-22 > /sys/bus/ap/apmask
+ echo -0,-0x33,-0x3a > /sys/bus/ap/aqmask
+
+.. note::
+
+ * It is not necessary to reserve both the adapters and domains.
+ Reserving an adapter implicitly allows the vfio_ap driver to assign
+ each of the domains available on the system along with the reserved
+ adapters to a guest's AP configuration and vice versa.
+
+ * Reserving adapters and/or domains will result in binding the queue
+ devices referenced by the APQNs derived from the Cartesian product of
+ the adapters and domains made available to the vfio_ap device driver
+ with one caveat; an APQN does not necessarily have to reference a
+ queue device installed in the system in order to reserve it for the
+ vfio_ap device driver; however, if the APQN does not reference a
+ queue device installed on the destination host, migration will fail.
+
+ * It is not necessary to configure the destination mediated device with
+ adapters, domains and control domains because the source guest's AP
+ configuration will overlay the destination guest's during migration.
+
+ * To verify that the queue devices are reserved for the vfio_ap
+ device driver, issue the ``lszcrypt -V`` command and you should
+ see ``vfio_ap`` in the ``driver`` column of the output.
+
+Live guest migration failures due to AP configuration errors
+------------------------------------------------------------
+This section describes the errors that may occur during live guest migration
+when the AP configuration of the source guest and destination host are not
+compatible and where to look for problem determination.
+
+The destination host is missing the mediated device
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This error occurs when a mediated device with the same name (UUID) as the
+mediated device being migrated has not been created on the destination host.
+
+**Source host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | error: device not found: mediated device '$UUID' not |
+| | found |
++--------------+----------------------------------------------------------------+
+| QEMU log: | initiating migration |
+| | qemu-system-s390x: Sibling indicated error 1 |
++--------------+----------------------------------------------------------------+
+
+.. note::
+ $UUID is the UUID of the mediated device being migrated from the source host.
+
+
+One or more queues are not bound to the vfio_ap driver on the destination host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The queue devices passed through to the source guest are not installed in the
+destination host or are not bound to its vfio_ap device driver.
+
+**Source host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | error: operation failed: migration failed. Message from the |
+| | source host: operation failed: job 'migration out' failed: |
+| | Sibling indicated error 1. Message from the destination host: |
+| | operation failed: job 'migration in' failed: load of migration |
+| | failed: No such device: Failed to load vmstate version_id: 1, |
+| | ret: -19 |
++--------------+----------------------------------------------------------------+
+
+**Destinaton host**
+
++--------------+----------------------------------------------------------------+
+| Kernel log: | vfio_ap_mdev $UUID: Queue $APID.$APQI not available to |
+| | vfio_ap driver on target host |
++--------------+----------------------------------------------------------------+
+| QEMU log: | shutting down, reason=failed |
+| | qemu-system-s390x: terminating on signal $SIGNAL_NUM from |
+| | pid $PID $DAEMON |
++--------------+----------------------------------------------------------------+
+
+
+.. note::
+ * $UUID is the UUID of the mediated device being migrated from the source host.
+ * $APID.$APQI is adapter number and queue index (domain number) comprising the
+ $APQN (for example, 00.0033) of the queue device that is not available.
+ * $SIGNAL_NUM is the signal number that caused the QEMU process to terminate.
+ * $PID is the process ID of the QEMU process on the destination host.
+ * $DAEMON is the name of the QEMU daemon process.
+ * This message will be logged for each queue for which this error is detected.
+
+Failure to get hardware information for a queue device on destination host
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The vfio_ap device driver on the destination host will execute an AP instruction
+to get the hardware information for a queue device to perform queue compatibility
+checks. The instruction can fail for a number of reasons in which case the
+migration will be terminated.
+
+**Source host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | error: operation failed: migration failed. Message from the |
+| | source host: operation failed: job 'migration out' failed: |
+| | Sibling indicated error 1. Message from the destination host: |
+| | operation failed: job 'migration in' failed: load of migration |
+| | failed: failed to get hardware information for queue device |
+| | $APQN |
++--------------+----------------------------------------------------------------+
+
+**Destination host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | vfio_ap_mdev $UUID: migration failed: Failed to get hwinfo for |
+| | queue $APQN on target host: rc=$RESPONSE-CODE" |
++--------------+----------------------------------------------------------------+
+| QEMU log: | shutting down, reason=failed |
+| | qemu-system-s390x: terminating on signal $SIGNAL-ID from pid |
+| | $PID $DAEMON |
++--------------+----------------------------------------------------------------+
+
+.. note::
+ * $UUID is the UUID of the mediated device being migrated from the source host.
+ * $APQN (for example, 00.0033) of the queue device that is not compatible.
+ * $RESPONSE_CODE is the response code from the AP instruction.
+ * $SIGNAL_NUM is the signal number of the signal that caused the QEMU process
+ to terminate.
+ * $PID is the process ID of the QEMU process on the destination host.
+ * $DAEMON is the name of the QEMU daemon process.
+ * This message will be logged for each queue for which this error is detected.
+
+The AP configurations of the source guest and destination host not incompatible
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The functional capabilities or facilities available on one or more of the queues
+passed through to the source guest are not compatible with the queue device with
+the same APQN on the destination system (see the Hardware Capabilities table above)
+
+**Source host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | error: operation failed: migration failed. Message from the |
+| | source host: operation failed: job 'migration out' failed: |
+| | Sibling indicated error 1. Message from the destination host: |
+| | operation failed: job 'migration in' failed: load of migration |
+| | failed: No such device: Failed to load vmstate version_id: 1, |
+| | ret: -19 |
++--------------+----------------------------------------------------------------+
+| Kernel log: | N/A |
++--------------+----------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: Sibling indicated error 1* |
++--------------+----------------------------------------------------------------+
+
+**Destination host**
+
++--------------+----------------------------------------------------------------+
+| Log | Message |
++==============+================================================================+
+| Console log: | N/A |
++--------------+----------------------------------------------------------------+
+| Kernel log: | *vfio_ap_mdev $UUID: Migration failed: Source and target |
+| | queue ($APQN) not compatible* |
+| | |
+| | The message above will be followed by one or more messages |
+| | enumerating the incompatible features; for example: |
+| | |
+| | ``vfio_ap_mdev $UUID: APSC facility installed in source queue |
+| | $APQN`` |
+| | ``vfio_ap_mdev $UUID: APSC facility not installed in target |
+| | queue $APQN`` |
+| | |
+| | ``AP type of source ($APTYPE1) not compatible with target |
+| | ($APTYPE2)`` ... |
++--------------+----------------------------------------------------------------+
+| QEMU log: | *initiating migration* |
+| | |
+| | *qemu-system-s390x: error while loading state section id ...* |
+| | |
+| | *shutting down, reason=failed* |
+| | |
+| | *terminating on signal $SIGNAL-ID from pid $PID $DAEMON* |
++--------------+----------------------------------------------------------------+
+
+.. note::
+ * $UUID is the UUID of the mediated device being migrated from the source host.
+ * $APQN (for example, 00.0033) of the queue device that is incompatible.
+ * $APTYPE1 is the type of the source queue device.
+ * $APTYPE2 is the type of the target queue device.
+ * $SIGNAL-ID is the signal number of the signal that caused the QEMU process
+ to terminate.
+ * $PROCESS-ID is the process ID of the QEMU process on the source host.
+ * $PID is the process ID of the QEMU process on the destination host.
+ * $DAEMON is the name of the QEMU daemon process.
+ * Each follow-on message logging the incompatibility will be logged for each
+ error detected for the specified queue device.
+
+AP Configuration Management
+---------------------------
+The AP configurations of the source guest and destination host must be kept
+in synchronization or live guest migration will likely fail due to potential
+incompatibilities introduced by such changes. In particular, changing adapter
+or domain reservations via the sysfs ``apmask`` or ``aqmask`` attributes on
+either the source or destination host without coordinating such changes between
+the two hosts -especially while a migration is taking place - can result in
+migration failures.
+
+Configuration stability is an orchestration-layer or system administrator
+responsibility, which is consistent with other VFIO device types. The vfio_ap
+device driver will validate source and destination configurations and provide
+clear diagnostics when incompatibilities are detected, enabling orchestration
+tools to implement appropriate policies; for example, blocking migration when
+such changes are to be made.
+
+Master Key administration
+--------------------------
+It is imperative that s390 Common Cryptographic Architecture (CCA) master key
+administration be performed on both the source and destination AP
+devices to synchronize the key values prior to allowing live guest
+migration. If the master keys do not match, then crypto applications that
+rely on secure keys wrapped by a CCA master key will fail when the guest on
+which they are running is migrated to the destination host. Again, this is
+an orchestration-layer or system administrator responsibility and migration must
+be blocked while master key administration is being performed.
--
2.53.0