Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices
From: Anthony Krowiak
Date: Thu Apr 02 2026 - 08:10:53 EST
On 4/1/26 12:57 PM, Alex Williamson wrote:
On Wed, 1 Apr 2026 09:38:59 -0400
Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
On 3/31/26 1:40 PM, Alex Williamson wrote:Yes, AIUI this is an artifact of your tooling. The destination QEMU
On Tue, 31 Mar 2026 08:07:06 -0400Maybe the problem here is stating that having the same UUID is a
Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
On 3/30/26 12:27 PM, Alex Williamson wrote:Combining replies:
On Wed, 25 Mar 2026 17:00:47 -0400
Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
This patch series implements live guest migration of a guest to which APAFAIK, same UUID is not a requirement for out-of-tree mdev drivers
devices have been passed through. To better comprehend this design, one has
to understand that VFIO AP mediated device is not used to provide userspace
with direct access to a device as is the case with other devices that use
the VFIO framework to pass them through to a guest. The sole purpose of the
VFIO AP mediated device is to manage an AP configuration for a guest. An AP
configuration is comprised of the AP adapter IDs (APID), AP queue
indexes (APQI) and domain numbers of the control domains to which a guest
will be granted access. Once the VFIO AP mediated device is attached to a
guest, its AP configuration is set by the vfio_ap device driver. Once set,
all access to the AP devices is handled by the s390 Interpretive Execution
facility; in other words, the vfio_ap device driver plays no role in
providing direct access to the AP devices in the guest's AP configuration.
The only role that the vfio_ap device driver plays in the migration
process is to verify that the AP configuration for the source guest is
compatible with the AP configuration of the destination guest.
Incompatibility will result in a live guest migration failure.
In order to be compatible, the following requirements must be met:
1. The destination guest will be started with the same QEMU command line
as the source guest, so the mediated device supplying the AP
configuration on both guests must have the same name (UUID).
supporting migration. You're really concerned more with the
configuration/composition of the mdev device, so requiring the same
UUID seems a bit arbitrary.
On Tue, 31 Mar 2026 07:17:08 -0400
Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
As stated above, the destination guest will be started with the sameThen this is a requirement of your tooling, not a kernel requirement, not
QEMU command line as the source guest. Within that command line
will be a '-device' parameter like the following:
-device
'{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
Note that sysfsdev is the path to the mdev named
62177883-f1bb-47f0-914d-32a22e3a8804;
therefore, the mdev with that name must exist on the destination guest or
the migration will fail with the following error:
error: device not found: mediated device
'62177883-f1bb-47f0-914d-32a22e3a8804' not found
something the kernel should care about. QEMU matches devices by their
virtual bus path, not the sysfsdev or host attributes. In the case of
VF migration with vfio-pci variant drivers we cannot require that the
source and target devices exist at the same bus address. Ideally the
pre-copy data from the source device to the target will include relevant
configuration information to validate that the source and target are
compatible, regardless of the uuid.
requirement in the patch series description. I agree this is
not a kernel requirement; however, a live guest migration will not
succeed unless the destination host has a mediated device with
a UUID that matches that of the UUID of the mediated device used
to pass through s390 crypto devices to the source guest for the
reason I stated above. Would it help if I removed item #1 as a
requirement?
can be started with any arbitrary UUID and should accept an incoming
migration, routing the migration to the correct device by the virtual
path, independent of the UUID.
If by tooling you are referring to libvirt and/or quemu, then I agree;
this is an artifact of tooling. According to
https://www.linux-kvm.org/page/Migration#Requirements,
"the guest on the destination must be started the same way it was
started on the source". s390 guests are run under qemu, so the
qemu command line used to start the source guest will also be used
to start the destination guest. To pass through
AP devices to the guest, the mediated device that provides the guest's
AP configuration - i.e., the crypto devices to which the guest is granted
access - must be identified on the qemu command line. As stated
previously, the following qemu command line parameter is included
for this purpose:
-device '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
When the qemu command is used to start the destination guest, the same
qemu command line will be used to start it. As such, qemu will try to open
an fd for that device. If the mdev with UUID does not exist, the migration
will fail. In other words, I don't think the guest can be started with any
arbitrary UUID. Am I wrong about this?
In any case, it is probably not relevant to mention this for this patch series
since it is not a KVM/kernel requirement, so I will remove it from the
cover-letter description.
The cover letter has one sentence describing the Interpretive ExecutionGot it.See above, userspace tooling requirements don't imply kernelI explained why in my previous response.2. The AP configuration assigned via the VFIO AP mediated device on bothWhy isn't this sufficient vs also requiring the same UUID?
guests must be compatible. As such, each AP configuration must meet
the following requirements:
* Both guests must have the same number of APQNs
* Each APQN assigned to the source guest must also be assigned to the
destination guest
* Each APQN assigned to both guests must reference an AP queue with the
same hardware capabilities
requirement.
A userspace interface can still exist, I just don't find this pathAs I stated in the cover letter description, all access to the AP
through the driver to the VM acceptable with this justification.
Mechanisms already exist for the device to refuse a state transition or
generate an error for a migration already in progress. IMHO, it would
be acceptable for the device to block a key change if the migration is
already in progress. If the key change cannot be represented in the
migration data stream, then it's up to the orchestration of the
migration to make sure they stay synchronized, but I don't see that
the vfio uAPI needs to be involved. Thanks,
devices is handled by the s390 Interpretive Execution facility, so
without a complete redesign of the vfio_ap device driver, there is no
way to know that a master key change is being requested. I do get your
point, however, so I will figure out how best to handle this in
userspace. Thanks for you review.
facility handling access to the device. There are a lot of subtleties
there that are not obvious to laypersons.
It seems like the architecture really doesn't have the ability
to monitor state changes of the device, and therefore represent those
changes in the migration data stream, so you want to fallback to a
cooperative scheme with a side-band channel to interrupt or block
migration when an asynchronous configuration change happens.
I just don't see that such cooperation needs a new uAPI through the
device; either the device itself could perform the same within the
existing uAPI or the agent changing the configuration could coordinate
with the migration orchestration. Thanks,
I understand your view on this and have already agreed to remove
the new uAPI.
Alex