[PATCH v2 00/16] s390/vfio-ap: Add live guest migration support

[Anthony Krowiak]

This patch series implements live guest migration support for KVM guests
with s390 AP (Adjunct Processor) devices passed through via the VFIO
mediated device framework.

Background
----------

The vfio-ap device driver differs from typical VFIO device drivers in that
it does not virtualize a physical device. Instead, it manages AP
configuration metadata identifying the AP adapters, domains, and control
domains to which a guest will be granted access. These AP resources are 
configured by assigning them to a vfio-ap mediated device via its sysfs
assignment interfaces. When the fd for the VFIO device is opened by 
userspace, the vfio_ap device driver sets the guest's AP configuration
from the metadata stored with the mediated device. As such, the AP devices
are not accessed directly through the vfio_ap driver, so the driver has no
internal device state to migrate. It's sole purpose during migration is to
ensure that the AP configurations of the source and destination guests are
compatible.

Implementation Approach
-----------------------

This series implements the VFIO migration protocol using the STOP_COPY
migration flow. The key aspects are:

1. Hardware Information Capture (Patches 1-2)
   - Store AP queue hardware characteristics at probe time
   - Provide access to queue objects for validation

2. Migration Infrastructure (Patches 3-5)
   - Define migration data structures
   - Initialize/release migration data on device open/close

3. State Machine Implementation (Patches 6-13)
   - Implement required VFIO migration callbacks
   - Handle state transitions: STOP → STOP_COPY → STOP (source)
                               STOP → RESUMING → STOP (destination)
   - Use file streams for AP configuration data transfer

4. Validation and Callbacks (Patches 14-15)
   - Implement migration state and data size callbacks
   - Required for VFIO_DEVICE_FEATURE_MIGRATION support

5. Documentation (Patch 16)
   - Add live guest migration chapter to vfio-ap.rst

Compatibility Validation
------------------------

The series includes comprehensive validation to ensure source and
destination AP configurations are compatible. For each queue, the following
characteristics must match:

- AP type (target must be same or newer than source)
- Installed facilities (APSC, APQKM, AP4KC, SLCF)
- Operating mode (CCA, Accelerator, XCP)
- APXA facility setting
- Classification (native vs stateless functions)
- Queue usability (binding/associated state)

When incompatibilities are detected, migration fails with detailed error
messages identifying the specific queue and characteristic that caused
the failure.

Configuration Management
------------------------

This implementation does not prevent configuration changes during
migration. Configuration stability is an orchestration-layer
responsibility, consistent with other VFIO device types. The driver's
role is to validate configurations and provide clear diagnostics when
incompatibilities are detected, enabling orchestration tools to implement
appropriate policies.

Change log v1 to v2
-------------------

- Removed patches that attempted to block configuration changes during
  migration due to inherent race conditions and incomplete protection
- Simplified approach focuses on validation and error reporting
- Reduced series from 18 to 16 patches
- Rewrote the description in the cover letter to better describe the
  patch series, remove confusing comments as well as references to
  function provided by the patches that were removed.

Anthony Krowiak (16):
  s390/vfio-ap: Store queue hardware info when probed
  s390/vfio-ap: Provide access to queue objects and related info
  s390/vfio-ap: Data structures for facilitating vfio device migration
  s390/vfio-ap: Initialize/release vfio device migration data
  s390-vfio-ap: Callback to set vfio device mig state during guest
    migration
  s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY
  s390/vfio-ap: File ops called to save the vfio device migration state
  s390/vfio-ap: Transition device migration state from STOP to RESUMING
  s390/vfio-ap: File ops called to resume the vfio device migration
  s390/vfio-ap: Transition device migration state from RESUMING to STOP
  s390/vfio-ap: Transition device migration state from STOP_COPY to STOP
  s390/vfio-ap: Transition device migration state from STOP to RUNNING
    and vice versa
  s390-vfio-ap: Callback to get the current vfio device migration state
  s390/vfio-ap: Callback to get the size of data to be migrated during
    guest migration
  s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute
  s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst

 Documentation/arch/s390/vfio-ap.rst     |  325 +++++--
 drivers/s390/crypto/Makefile            |    2 +-
 drivers/s390/crypto/vfio_ap_drv.c       |    4 +-
 drivers/s390/crypto/vfio_ap_migration.c | 1095 +++++++++++++++++++++++
 drivers/s390/crypto/vfio_ap_ops.c       |   66 +-
 drivers/s390/crypto/vfio_ap_private.h   |   10 +
 6 files changed, 1395 insertions(+), 107 deletions(-)
 create mode 100644 drivers/s390/crypto/vfio_ap_migration.c

-- 
2.52.0