Re: [PATCH v1 00/23] s390/vfio-ap: Implement live guest migration of guests using AP devices

[Anthony Krowiak]

On 3/31/26 1:40 PM, Alex Williamson wrote:
> On Tue, 31 Mar 2026 08:07:06 -0400
> Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
>
> > On 3/30/26 12:27 PM, Alex Williamson wrote:
> > > On Wed, 25 Mar 2026 17:00:47 -0400
> > > Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
> > >   
> > > > This patch series implements live guest migration of a guest to which AP
> > > > devices have been passed through. To better comprehend this design, one has
> > > > to understand that VFIO AP mediated device is not used to provide userspace
> > > > with direct access to a device as is the case with other devices that use
> > > > the VFIO framework to pass them through to a guest. The sole purpose of the
> > > > VFIO AP mediated device is to manage an AP configuration for a guest. An AP
> > > > configuration is comprised of the AP adapter IDs (APID), AP queue
> > > > indexes (APQI) and domain numbers of the control domains to which a guest
> > > > will be granted access. Once the VFIO AP mediated device is attached to a
> > > > guest, its AP configuration is set by the vfio_ap device driver. Once set,
> > > > all access to the AP devices is handled by the s390 Interpretive Execution
> > > > facility; in other words, the vfio_ap device driver plays no role in
> > > > providing direct access to the AP devices in the guest's AP configuration.
> > > >
> > > > The only role that the vfio_ap device driver plays in the migration
> > > > process is to verify that the AP configuration for the source guest is
> > > > compatible with the AP configuration of the destination guest.
> > > > Incompatibility will result in a live guest migration failure.
> > > > In order to be compatible, the following requirements must be met:
> > > >
> > > > 1. The destination guest will be started with the same QEMU command line
> > > >      as the source guest, so the mediated device supplying the AP
> > > >      configuration on both guests must have the same name (UUID).
> > > AFAIK, same UUID is not a requirement for out-of-tree mdev drivers
> > > supporting migration.  You're really concerned more with the
> > > configuration/composition of the mdev device, so requiring the same
> > > UUID seems a bit arbitrary.
> Combining replies:
>
> On Tue, 31 Mar 2026 07:17:08 -0400
> Anthony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
> > As stated above, the destination guest will be started with the same
> > QEMU command line as the source guest. Within that command line
> > will be a '-device' parameter like the following:
> >
> > -device
> > '{"driver":"vfio-ap","id":"hostdev0","sysfsdev":"/sys/bus/mdev/devices/62177883-f1bb-47f0-914d-32a22e3a8804"}
> >
> > Note that sysfsdev is the path to the mdev named
> > 62177883-f1bb-47f0-914d-32a22e3a8804;
> > therefore, the mdev with that name must exist on the destination guest or
> > the migration will fail with the following error:
> >
> > error: device not found: mediated device
> > '62177883-f1bb-47f0-914d-32a22e3a8804' not found
> Then this is a requirement of your tooling, not a kernel requirement, not
> something the kernel should care about.  QEMU matches devices by their
> virtual bus path, not the sysfsdev or host attributes.  In the case of
> VF migration with vfio-pci variant drivers we cannot require that the
> source and target devices exist at the same bus address.  Ideally the
> pre-copy data from the source device to the target will include relevant
> configuration information to validate that the source and target are
> compatible, regardless of the uuid.

Maybe the problem here is stating that having the same UUID is a
requirement in the patch series description. I agree this is
not a kernel requirement; however, a live guest migration will not
succeed unless the destination host has a mediated device with
a UUID that matches that of the UUID of the mediated device used
to pass through s390 crypto devices to the source guest for the
reason I stated above. Would it help if I removed item #1 as a
requirement?

> > > > 2. The AP configuration assigned via the VFIO AP mediated device on both
> > > >      guests must be compatible. As such, each AP configuration must meet
> > > >      the following requirements:
> > > >
> > > >      * Both guests must have the same number of APQNs
> > > >
> > > >      * Each APQN assigned to the source guest must also be assigned to the
> > > >        destination guest
> > > >
> > > >      * Each APQN assigned to both guests must reference an AP queue with the
> > > >        same hardware capabilities
> > > Why isn't this sufficient vs also requiring the same UUID?
> > I explained why in my previous response.
> See above, userspace tooling requirements don't imply kernel
> requirement.

Got it.

> A userspace interface can still exist, I just don't find this path
> through the driver to the VM acceptable with this justification.
> Mechanisms already exist for the device to refuse a state transition or
> generate an error for a migration already in progress.  IMHO, it would
> be acceptable for the device to block a key change if the migration is
> already in progress.  If the key change cannot be represented in the
> migration data stream, then it's up to the orchestration of the
> migration to make sure they stay synchronized, but I don't see that
> the vfio uAPI needs to be involved.  Thanks,

As I stated in the cover letter description, all access to the AP 
devices is handled by the s390 Interpretive Execution facility, so 
without a complete redesign of the vfio_ap device driver, there is no 
way to know that a master key change is being requested. I do get your 
point, however, so I will figure out how best to handle this in 
userspace. Thanks for you review.
> Alex