Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices
From: Anthony Krowiak
Date: Tue Mar 31 2026 - 08:08:52 EST
On 3/30/26 12:27 PM, Alex Williamson wrote:
On Wed, 25 Mar 2026 17:00:47 -0400
Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
This patch series implements live guest migration of a guest to which APAFAIK, same UUID is not a requirement for out-of-tree mdev drivers
devices have been passed through. To better comprehend this design, one has
to understand that VFIO AP mediated device is not used to provide userspace
with direct access to a device as is the case with other devices that use
the VFIO framework to pass them through to a guest. The sole purpose of the
VFIO AP mediated device is to manage an AP configuration for a guest. An AP
configuration is comprised of the AP adapter IDs (APID), AP queue
indexes (APQI) and domain numbers of the control domains to which a guest
will be granted access. Once the VFIO AP mediated device is attached to a
guest, its AP configuration is set by the vfio_ap device driver. Once set,
all access to the AP devices is handled by the s390 Interpretive Execution
facility; in other words, the vfio_ap device driver plays no role in
providing direct access to the AP devices in the guest's AP configuration.
The only role that the vfio_ap device driver plays in the migration
process is to verify that the AP configuration for the source guest is
compatible with the AP configuration of the destination guest.
Incompatibility will result in a live guest migration failure.
In order to be compatible, the following requirements must be met:
1. The destination guest will be started with the same QEMU command line
as the source guest, so the mediated device supplying the AP
configuration on both guests must have the same name (UUID).
supporting migration. You're really concerned more with the
configuration/composition of the mdev device, so requiring the same
UUID seems a bit arbitrary.
2. The AP configuration assigned via the VFIO AP mediated device on bothWhy isn't this sufficient vs also requiring the same UUID?
guests must be compatible. As such, each AP configuration must meet
the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the
destination guest
* Each APQN assigned to both guests must reference an AP queue with the
same hardware capabilities
I explained why in my previous response.
Note: There is a forthcoming consumer of this series which will be a QEMUThis seems inherently racy. What happens if the device becomes
patch series is entitled:
'hw/vfio/ap: Implement live guest migration of guests using AP
devices'
This design also adds a use case for enabling and disabling
migration of guests to which AP devices have been passed through. To
facilitate this, a new read/write sysfs 'migratable' attribute is added to
the mediated device. This attribute specifies whether the vfio device is
migratable (1) or not (0). When the value of this attribute is changed, the
vfio_ap device driver will signal an eventfd to userspace. It is up to
userspace to respond to the change by enabling or disabling migration of
the guest to which the mediated device is attached. The operation will be
rejected with a 'Device or resource busy' message if a migration is in
progress.
unmigratable while it's being migrated?
If userspace is deciding that the device is now unmigratable, why does
it need to signal this through the kernel driver rather than with the
userspace orchestration agent? The entire path seems unnecessary.
I am not familiar with what a userspace orchestration agent is, so
I can't address that. Can you please describe how that would work?
Maybe it would help to provide the reason for this. For certain types
of crypto operations, a master key must be configured for the crypto
card domain being used. This master key must be synchronized
between the source and destination crypto device so that in-flight
crypto operations can be completed during migration. If these master
keys must be changed, migration must be blocked until the master
key changes can be synchronized between the source and destination
system(s).
Userspace must also have a means for retrieving the value of the sysfsThis is just broken, it's redundant to our current device feature
'migratable' attribute when the guest is started to initialize whether it
can be migrated. For this, The VFIO_DEVICE_GET_INFO ioctl is used. The
struct vfio_device_info object passed to the ioctl will be extended with a
capability specifying the vfio device attributes. One of the attributes
will contain the value of the mediated device's 'migratable' attribute.
mechanism for exposing migration support. If you want the capability
to create unmigratable devices statically, can't that be encompassed
within the definition of the mdev type? Dynamic migration support just
seems like it's involving the kernel in orchestration it shouldn't be a
part of.
So, it appears you are suggesting the creation of a new mdev type
for unmigratable crypto devices. I don't see the value in that.
As I stated above, there is a valid reason for wanting to prevent
migration while master key synchronization is taking place.
If this feature violates the implicit rules of vfio device migration,
then so be it. Maybe we have to figure out another way to ensure
migration is not initiated during master key synchronization.
If we can't find an acceptable means to do this programmatically,
then maybe it will come down to a matter of documenting the
need to ensure migration is not initiated while master key
synchronization is taking place. This would put the onus on the
various system administrators responsible for host, guest and
master key administration to communicate out of band to
ensure they are all on the same page with regard to migration.
It would be preferable to be able to do this with a userspace
interface, so any suggestions would be greatly appreciated.
Thanks,
Alex
Thank you for your review.
Anthony Krowiak (23):
s390/vfio-ap: Store queue hardware info when probed
s390/vfio-ap: Provide access to queue objects and related info
s390/vfio-ap: Add header file for xfer of vfio device caps to
userspace
MAINTAINERS: Add new header file for the S390 VFIO AP DRIVER
maintainers
s390/vfio-ap: A sysfs 'migratable' attribute to enable/disable
migration of guest
s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
s390/vfio-ap: Signal event to enable/disable live guest migration
s390/vfio-ap: Return value of sysfs migratable attribute from
VFIO_DEVICE_GET_INFO ioctl
s390/vfio-ap: Data structures for facilitating vfio device migration
s390/vfio-ap: Initialize/release vfio device migration data
s390-vfio-ap: Callback to set vfio device mig state during guest
migration
s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
s390/vfio-ap: File ops called to save the vfio device migration state
s390/vfio-ap: Transition device migration state from STOP to RESUMING
s390/vfio-ap: File ops called to resume the vfio device migration
s390/vfio-ap: Transition device migration state from RESUMING to STOP
s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
s390/vfio-ap: Transition device migration state from STOP to RUNNING
and vice versa
s390-vfio-ap: Callback to get the current vfio device migration state
s390/vfio-ap: Callback to get the size of data to be migrated during
guest migration
s390/vfio-ap: Provide API to query whether migration is in progress
s390/vfio-ap: Disallow blocking migration in progress
s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst
Documentation/arch/s390/vfio-ap.rst | 339 +++++--
MAINTAINERS | 1 +
drivers/s390/crypto/Makefile | 2 +-
drivers/s390/crypto/vfio_ap_drv.c | 4 +-
drivers/s390/crypto/vfio_ap_migration.c | 1131 +++++++++++++++++++++++
drivers/s390/crypto/vfio_ap_ops.c | 263 +++++-
drivers/s390/crypto/vfio_ap_private.h | 20 +
include/uapi/linux/vfio.h | 2 +
include/uapi/linux/vfio_ap.h | 34 +
9 files changed, 1685 insertions(+), 111 deletions(-)
create mode 100644 drivers/s390/crypto/vfio_ap_migration.c
create mode 100644 include/uapi/linux/vfio_ap.h