Re: [PATCH v10 16/16] s390/vfio-ap: update docs to include dynamic config support

[Tony Krowiak]

On 9/27/20 10:48 PM, Halil Pasic wrote:
> On Fri, 21 Aug 2020 15:56:16 -0400
> Tony Krowiak <akrowiak@xxxxxxxxxxxxx> wrote:
>
> > Update the documentation in vfio-ap.rst to include information about the
> > AP dynamic configuration support (i.e., hot plug of adapters, domains
> > and control domains via the matrix mediated device's sysfs assignment
> > attributes).
> If you don't mind I would like to skip out on commenting on the
> documentation update, because of the design issues I've raised. I think
> we should first clear that up. Is that OK with you?

That's fine since the docs will likely have to change.

> > Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> > ---
> >   Documentation/s390/vfio-ap.rst | 362 ++++++++++++++++++++++++++-------
> >   1 file changed, 285 insertions(+), 77 deletions(-)
> >
> > diff --git a/Documentation/s390/vfio-ap.rst b/Documentation/s390/vfio-ap.rst
> > index e15436599086..8907aeca8fb7 100644
> > --- a/Documentation/s390/vfio-ap.rst
> > +++ b/Documentation/s390/vfio-ap.rst
> > @@ -253,7 +253,7 @@ The process for reserving an AP queue for use by a KVM guest is:
> >   1. The administrator loads the vfio_ap device driver
> >   2. The vfio-ap driver during its initialization will register a single 'matrix'
> >      device with the device core. This will serve as the parent device for
> > -   all mediated matrix devices used to configure an AP matrix for a guest.
> > +   all matrix mediated devices used to configure an AP matrix for a guest.
> >   3. The /sys/devices/vfio_ap/matrix device is created by the device core
> >   4. The vfio_ap device driver will register with the AP bus for AP queue devices
> >      of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
> > @@ -269,7 +269,7 @@ The process for reserving an AP queue for use by a KVM guest is:
> >      default zcrypt cex4queue driver.
> >   8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
> >      it.
> > -9. The administrator creates a passthrough type mediated matrix device to be
> > +9. The administrator creates a passthrough type matrix mediated device to be
> >      used by a guest
> >   10. The administrator assigns the adapters, usage domains and control domains
> >       to be exclusively used by a guest.
> > @@ -279,14 +279,14 @@ Set up the VFIO mediated device interfaces
> >   The VFIO AP device driver utilizes the common interface of the VFIO mediated
> >   device core driver to:
> >   
> > -* Register an AP mediated bus driver to add a mediated matrix device to and
> > +* Register an AP mediated bus driver to add a matrix mediated device to and
> >     remove it from a VFIO group.
> > -* Create and destroy a mediated matrix device
> > -* Add a mediated matrix device to and remove it from the AP mediated bus driver
> > -* Add a mediated matrix device to and remove it from an IOMMU group
> > +* Create and destroy a matrix mediated device
> > +* Add a matrix mediated device to and remove it from the AP mediated bus driver
> > +* Add a matrix mediated device to and remove it from an IOMMU group
> >   
> >   The following high-level block diagram shows the main components and interfaces
> > -of the VFIO AP mediated matrix device driver::
> > +of the VFIO AP matrix mediated device driver::
> >   
> >      +-------------+
> >      |             |
> > @@ -351,29 +351,37 @@ matrix device.
> >       This attribute group identifies the user-defined sysfs attributes of the
> >       mediated device. When a device is registered with the VFIO mediated device
> >       framework, the sysfs attribute files identified in the 'mdev_attr_groups'
> > -    structure will be created in the mediated matrix device's directory. The
> > -    sysfs attributes for a mediated matrix device are:
> > +    structure will be created in the matrix mediated device's directory. The
> > +    sysfs attributes for a matrix mediated device are:
> >   
> >       assign_adapter / unassign_adapter:
> >         Write-only attributes for assigning/unassigning an AP adapter to/from the
> > -      mediated matrix device. To assign/unassign an adapter, the APID of the
> > +      matrix mediated device. To assign/unassign an adapter, the APID of the
> >         adapter is echoed to the respective attribute file.
> >       assign_domain / unassign_domain:
> >         Write-only attributes for assigning/unassigning an AP usage domain to/from
> > -      the mediated matrix device. To assign/unassign a domain, the domain
> > +      the matrix mediated device. To assign/unassign a domain, the domain
> >         number of the usage domain is echoed to the respective attribute
> >         file.
> >       matrix:
> > -      A read-only file for displaying the APQNs derived from the cross product
> > -      of the adapter and domain numbers assigned to the mediated matrix device.
> > +      A read-only file for displaying the APQNs derived from the Cartesian
> > +      product of the adapter and domain numbers assigned to the mediated matrix
> > +      device.
> > +    guest_matrix:
> > +      A read-only file for displaying the APQNs derived from the Cartesian
> > +      product of the adapter and domain numbers assigned to the APM and AQM
> > +      fields respectively of the KVM guest's CRYCB. This will differ from the
> > +      matrix if any APQNs assigned to the matrix mediated device do not
> > +      reference a queue device bound to the vfio_ap device driver (i.e., the
> > +      queue is not in the AP configuration).
> >       assign_control_domain / unassign_control_domain:
> >         Write-only attributes for assigning/unassigning an AP control domain
> > -      to/from the mediated matrix device. To assign/unassign a control domain,
> > +      to/from the matrix mediated device. To assign/unassign a control domain,
> >         the ID of the domain to be assigned/unassigned is echoed to the respective
> >         attribute file.
> >       control_domains:
> >         A read-only file for displaying the control domain numbers assigned to the
> > -      mediated matrix device.
> > +      matrix mediated device.
> >   
> >   * functions:
> >   
> > @@ -385,7 +393,7 @@ matrix device.
> >         domains assigned via the corresponding sysfs attributes files
> >   
> >     remove:
> > -    deallocates the mediated matrix device's ap_matrix_mdev structure. This will
> > +    deallocates the matrix mediated device's ap_matrix_mdev structure. This will
> >       be allowed only if a running guest is not using the mdev.
> >   
> >   * callback interfaces
> > @@ -397,7 +405,7 @@ matrix device.
> >       for the mdev matrix device to the MDEV bus. Access to the KVM structure used
> >       to configure the KVM guest is provided via this callback. The KVM structure,
> >       is used to configure the guest's access to the AP matrix defined via the
> > -    mediated matrix device's sysfs attribute files.
> > +    matrix mediated device's sysfs attribute files.
> >     release:
> >       unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
> >       mdev matrix device and deconfigures the guest's AP matrix.
> > @@ -410,11 +418,49 @@ function is called when QEMU connects to KVM. The guest's AP matrix is
> >   configured via it's CRYCB by:
> >   
> >   * Setting the bits in the APM corresponding to the APIDs assigned to the
> > -  mediated matrix device via its 'assign_adapter' interface.
> > +  matrix mediated device via its 'assign_adapter' interface.
> >   * Setting the bits in the AQM corresponding to the domains assigned to the
> > -  mediated matrix device via its 'assign_domain' interface.
> > +  matrix mediated device via its 'assign_domain' interface.
> >   * Setting the bits in the ADM corresponding to the domain dIDs assigned to the
> > -  mediated matrix device via its 'assign_control_domains' interface.
> > +  matrix mediated device via its 'assign_control_domains' interface.
> > +
> > +The linux device model precludes passing a device through to a KVM guest that
> > +is not bound to the device driver facilitating its pass-through. Consequently,
> > +an APQN that does not reference a queue device bound to the vfio_ap device
> > +driver will not be assigned to a KVM guest's CRYCB. The AP architecture,
> > +however, does not provide a means to filter individual APQNs from the guest's
> > +CRYCB, so the following logic is employed to filter them:
> > +
> > +* Filter the APQNs assigned to the matrix mediated device by APID.
> > +
> > +  To filter APQNs by APID, each APQN derived from the Cartesian product of the
> > +  adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is
> > +  examined and if any one of them does not reference a queue device bound to the
> > +  vfio_ap device driver, the adapter will not be plugged into the guest (i.e.,
> > +  the bit corresponding to its APID will not be set in the APM of the guest's
> > +  CRYCB).
> > +
> > +  If at least one adapter is plugged into the guest, then all domains assigned
> > +  to the mdev will also be plugged into the guest (i.e., the bits corresponding
> > +  to the APQIs of the domains assigned to the mdev will be set in the AQM field
> > +  of the guest's CRYCB).
> > +
> > +* Filter the APQNs assigned to the matrix mediated device by APQI.
> > +
> > +  The APQNs will be filtered by APQI if filtering by APID does not result in any
> > +  adapters or domains getting plugged into the guest.
> > +
> > +  To filter APQNs by APQI, each APQN derived from the Cartesian product of the
> > +  adapter numbers (APID) and domain numbers (APQI) assigned to the mdev is
> > +  examined and if any one of them does not reference a queue device bound to the
> > +  vfio_ap device driver, the domain will not be plugged into the guest (i.e.,
> > +  the bit corresponding to its APQI will not be set in the AQM of the guest's
> > +  CRYCB).
> > +
> > +  If at least one domain is plugged into the guest, then all adapters assigned
> > +  to the mdev will also be plugged into the guest (i.e., the bits corresponding
> > +  to the APIDs of the adapters assigned to the mdev will be set in the APM field
> > +  of the guest's CRYCB).
> >   
> >   The CPU model features for AP
> >   -----------------------------
> > @@ -435,6 +481,10 @@ available to a KVM guest via the following CPU model features:
> >      can be made available to the guest only if it is available on the host (i.e.,
> >      facility bit 12 is set).
> >   
> > +4. apqi: Indicates AP queue interrupts are available on the guest. This facility
> > +   can be made available to the guest only if it is available on the host (i.e.,
> > +   facility bit 65 is set).
> > +
> >   Note: If the user chooses to specify a CPU model different than the 'host'
> >   model to QEMU, the CPU model features and facilities need to be turned on
> >   explicitly; for example::
> > @@ -444,7 +494,7 @@ explicitly; for example::
> >   A guest can be precluded from using AP features/facilities by turning them off
> >   explicitly; for example::
> >   
> > -     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
> > +     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off
> >   
> >   Note: If the APFT facility is turned off (apft=off) for the guest, the guest
> >   will not see any AP devices. The zcrypt device drivers that register for type 10
> > @@ -530,40 +580,56 @@ These are the steps:
> >   
> >   2. Secure the AP queues to be used by the three guests so that the host can not
> >      access them. To secure them, there are two sysfs files that specify
> > -   bitmasks marking a subset of the APQN range as 'usable by the default AP
> > -   queue device drivers' or 'not usable by the default device drivers' and thus
> > -   available for use by the vfio_ap device driver'. The location of the sysfs
> > -   files containing the masks are::
> > +   bitmasks marking a subset of the APQN range as usable only by the default AP
> > +   queue device drivers. All remaining APQNs are available available for use by
> > +   any other device driver. The vfio_ap device driver is currently the only
> > +   non-default device driver. The location of the sysfs files containing the
> > +   masks are::
> >   
> >        /sys/bus/ap/apmask
> >        /sys/bus/ap/aqmask
> >   
> >      The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
> > -   (APID). Each bit in the mask, from left to right (i.e., from most significant
> > -   to least significant bit in big endian order), corresponds to an APID from
> > -   0-255. If a bit is set, the APID is marked as usable only by the default AP
> > -   queue device drivers; otherwise, the APID is usable by the vfio_ap
> > -   device driver.
> > +   (APID). Each bit in the mask, from left to right corresponds to an APID from
> > +   0-255. If a bit is set, the APID is marked as available to the default AP
> > +   queue device drivers.
> >   
> >      The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
> > -   (APQI). Each bit in the mask, from left to right (i.e., from most significant
> > -   to least significant bit in big endian order), corresponds to an APQI from
> > -   0-255. If a bit is set, the APQI is marked as usable only by the default AP
> > -   queue device drivers; otherwise, the APQI is usable by the vfio_ap device
> > -   driver.
> > +   (APQI). Each bit in the mask, from left to right corresponds to an APQI from
> > +   0-255. If a bit is set, the APQI is marked as available to the default AP
> > +   queue device drivers.
> > +
> > +   The Cartesian product of the APIDs corresponding to the bits set in the
> > +   apmask and the APQIs corresponding to the bits set in the aqmask comprise
> > +   the subset of APQNs that can be used only by the host default device drivers.
> > +   All other APQNs are available to the non-default device drivers such as the
> > +   vfio_ap driver.
> > +
> > +   Take, for example, the following masks::
> > +
> > +      apmask:
> > +      0x7d00000000000000000000000000000000000000000000000000000000000000
> > +
> > +      aqmask:
> > +      0x8000000000000000000000000000000000000000000000000000000000000000
> > +
> > +   The masks indicate:
> >   
> > -   Take, for example, the following mask::
> > +   * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default
> > +     device drivers.
> >   
> > -      0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
> > +   * Domain 0 is available for use by the host default device drivers
> >   
> > -    It indicates:
> > +   * The subset of APQNs available for use only by the default host device
> > +     drivers are:
> >   
> > -      1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
> > -      belong to the vfio_ap device driver's pool.
> > +     (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)
> > +
> > +   * All other APQNs are available for use by the non-default device drivers.
> >   
> >      The APQN of each AP queue device assigned to the linux host is checked by the
> > -   AP bus against the set of APQNs derived from the cross product of APIDs
> > -   and APQIs marked as usable only by the default AP queue device drivers. If a
> > +   AP bus against the set of APQNs derived from the Cartesian product of APIDs
> > +   and APQIs marked as available to the default AP queue device drivers. If a
> >      match is detected,  only the default AP queue device drivers will be probed;
> >      otherwise, the vfio_ap device driver will be probed.
> >   
> > @@ -627,6 +693,16 @@ These are the steps:
> >   	    default drivers pool:    adapter 0-15, domain 1
> >   	    alternate drivers pool:  adapter 16-255, domains 0, 2-255
> >   
> > +   Note ***:
> > +   Changing a mask such that one or more APQNs will be taken from a matrix
> > +   mediated device (see below) will fail with an error (EADDRINUSE). The error
> > +   is logged to the kernel ring buffer which can be viewed with the 'dmesg'
> > +   command. The output identifies each APQN flagged as 'in use' and the matrix
> > +   mediated device to which it is assigned; for example:
> > +
> > +   Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
> > +   Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4
> > +
> >   Securing the APQNs for our example
> >   ----------------------------------
> >      To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
> > @@ -684,7 +760,7 @@ Securing the APQNs for our example
> >   
> >        /sys/devices/vfio_ap/matrix/
> >        --- [mdev_supported_types]
> > -     ------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
> > +     ------ [vfio_ap-passthrough] (passthrough matrix mediated device type)
> >        --------- create
> >        --------- [devices]
> >   
> > @@ -775,17 +851,18 @@ Securing the APQNs for our example
> >        higher than the maximum is specified, the operation will terminate with
> >        an error (ENODEV).
> >   
> > -   * All APQNs that can be derived from the adapter ID and the IDs of
> > -     the previously assigned domains must be bound to the vfio_ap device
> > -     driver. If no domains have yet been assigned, then there must be at least
> > -     one APQN with the specified APID bound to the vfio_ap driver. If no such
> > -     APQNs are bound to the driver, the operation will terminate with an
> > -     error (EADDRNOTAVAIL).
> > +   * All APQNs that can be derived from the Cartesian product of the APID of the
> > +     adapter being assigned and the APQIs of the previously assigned domains
> > +     must be available to the vfio_ap device driver as specified in the sysfs
> > +     /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN
> > +     is reserved for use by the host device driver, the operation will terminate
> > +     with an error (EADDRNOTAVAIL).
> >   
> > -     No APQN that can be derived from the adapter ID and the IDs of the
> > -     previously assigned domains can be assigned to another mediated matrix
> > -     device. If an APQN is assigned to another mediated matrix device, the
> > -     operation will terminate with an error (EADDRINUSE).
> > +   * No APQN that can be derived from the Cartesian product of the APID of the
> > +     adapter being assigned and the APQIs of the previously assigned domains can
> > +     be assigned to another matrix mediated device. If even one APQN is assigned
> > +     to another matrix mediated device, the operation will terminate with an
> > +     error (EADDRINUSE).
> >   
> >      In order to successfully assign a domain:
> >   
> > @@ -794,17 +871,18 @@ Securing the APQNs for our example
> >        higher than the maximum is specified, the operation will terminate with
> >        an error (ENODEV).
> >   
> > -   * All APQNs that can be derived from the domain ID and the IDs of
> > -     the previously assigned adapters must be bound to the vfio_ap device
> > -     driver. If no domains have yet been assigned, then there must be at least
> > -     one APQN with the specified APQI bound to the vfio_ap driver. If no such
> > -     APQNs are bound to the driver, the operation will terminate with an
> > -     error (EADDRNOTAVAIL).
> > +   * All APQNs that can be derived from the Cartesian product of the APQI of the
> > +     domain being assigned and the APIDs of the previously assigned adapters
> > +     must be available to the vfio_ap device driver as specified in the sysfs
> > +     /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even one APQN
> > +     is reserved for use by the host device driver, the operation will terminate
> > +     with an error (EADDRNOTAVAIL).
> >   
> > -     No APQN that can be derived from the domain ID and the IDs of the
> > -     previously assigned adapters can be assigned to another mediated matrix
> > -     device. If an APQN is assigned to another mediated matrix device, the
> > -     operation will terminate with an error (EADDRINUSE).
> > +   * No APQN that can be derived from the Cartesian product of the APQI of the
> > +     domain being assigned and the APIDs of the previously assigned adapters can
> > +     be assigned to another matrix mediated device. If even one APQN is assigned
> > +     to another matrix mediated device, the operation will terminate with an
> > +     error (EADDRINUSE).
> >   
> >      In order to successfully assign a control domain, the domain number
> >      specified must represent a value from 0 up to the maximum domain number
> > @@ -813,22 +891,22 @@ Securing the APQNs for our example
> >   
> >   5. Start Guest1::
> >   
> > -     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
> > +     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
> >   	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
> >   
> >   7. Start Guest2::
> >   
> > -     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
> > +     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
> >   	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
> >   
> >   7. Start Guest3::
> >   
> > -     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
> > +     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
> >   	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
> >   
> > -When the guest is shut down, the mediated matrix devices may be removed.
> > +When the guest is shut down, the matrix mediated devices may be removed.
> >   
> > -Using our example again, to remove the mediated matrix device $uuid1::
> > +Using our example again, to remove the matrix mediated device $uuid1::
> >   
> >      /sys/devices/vfio_ap/matrix/
> >         --- [mdev_supported_types]
> > @@ -851,16 +929,146 @@ remove it if no guest will use it during the remaining lifetime of the linux
> >   host. If the mdev matrix device is removed, one may want to also reconfigure
> >   the pool of adapters and queues reserved for use by the default drivers.
> >   
> > +Hot plug support:
> > +================
> > +An adapter, domain or control domain may be hot plugged into a running KVM
> > +guest by assigning it to the matrix mediated device being used by the guest.
> > +Control domains will always be hot plugged; however, an adapter or domain will
> > +be hot plugged only if each new APQN resulting from its assignment
> > +references a queue device bound to the vfio_ap device driver as described
> > +below.
> > +
> > +When an adapter is assigned to a matrix mediated device in use by a KVM guest:
> > +
> > +* If no domains have yet been plugged into the KVM guest:
> > +
> > +  Hot plug the adapter and every domain previously assigned to the mdev if each
> > +  APQN derived from the Cartesian product of the APID of the adapter being
> > +  assigned and the APQIs of the domains previously assigned references a queue
> > +  device bound to the vfio_ap device driver.
> > +
> > +* If one or more domains have previously been plugged into the guest:
> > +
> > +  Hot plug the adapter if each APQN derived from the Cartesian product of the
> > +  APID of the adapter being assigned and the APQIs of the domains already
> > +  plugged into the guest references a queue device bound to the vfio_ap device
> > +  driver.
> > +
> > +When a domain is assigned to a matrix mediated device in use by a KVM guest:
> > +
> > +* If no adapters have yet been plugged into the KVM guest:
> > +
> > +  Hot plug the domain and every adapter previously assigned to the mdev if each
> > +  APQN derived from the Cartesian product of the APIDs of the adapters
> > +  previously assigned and the APQI of the domain being assigned references a
> > +  queue device bound to the vfio_ap device driver.
> > +
> > +* If one or more adapters have previously been plugged into the guest:
> > +
> > +  Hot plug the domain if each APQN derived from the Cartesian product of the
> > +  APIDs of the adapters already plugged into the guest and the APQI of the
> > +  domain being assigned references a queue device bound to the vfio_ap device
> > +  driver.
> > +
> > +Over-provisioning of AP queues for a KVM guest:
> > +==============================================
> > +Over-provisioning is defined herein as the assignment of adapters or domains to
> > +a matrix mediated device that do not reference AP devices in the host's AP
> > +configuration. The idea here is that when the adapter or domain becomes
> > +available, it will be automatically hot-plugged into the KVM guest using
> > +the matrix mediated device to which it is assigned as long as each new APQN
> > +resulting from plugging it in references a queue device bound to the vfio_ap
> > +device driver.
> > +
> >   Limitations
> >   ===========
> > -* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
> > -  to the default drivers pool of a queue that is still assigned to a mediated
> > -  device in use by a guest. It is incumbent upon the administrator to
> > -  ensure there is no mediated device in use by a guest to which the APQN is
> > -  assigned lest the host be given access to the private data of the AP queue
> > -  device such as a private key configured specifically for the guest.
> > +Live guest migration is not supported for guests using AP devices without
> > +intervention by a system administrator. Before a KVM guest can be migrated,
> > +the matrix mediated device must be removed. Unfortunately, it can not be
> > +removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
> > +the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
> > +its mdev can be hot unplugged from the guest in one of two ways:
> > +
> > +1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
> > +   the following commands:
> > +
> > +      virsh detach-device <guestname> <path-to-device-xml>
> > +
> > +      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
> > +      the guest named 'my-guest':
> > +
> > +         virsh detach-device my-guest ~/config/my-guest-hostdev.xml
> > +
> > +            The contents of my-guest-hostdev.xml:
> > +
> > +            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
> > +              <source>
> > +                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
> > +              </source>
> > +            </hostdev>
> > +
> > +
> > +      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
> > +
> > +      For example, to hot unplug the matrix mediated device identified on the
> > +      qemu command line with 'id=hostdev0' from the guest named 'my-guest':
> > +
> > +         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
> > +
> > +2. A matrix mediated device can be hot unplugged by attaching the qemu monitor
> > +   to the guest and using the following qemu monitor command:
> > +
> > +      (QEMU) device-del id=<device-id>
> > +
> > +      For example, to hot unplug the matrix mediated device that was specified
> > +      on the qemu command line with 'id=hostdev0' when the guest was started:
> > +
> > +         (QEMU) device-del id=hostdev0
> > +
> > +After live migration of the KVM guest completes, an AP configuration can be
> > +restored to the KVM guest by hot plugging a matrix mediated device on the target
> > +system into the guest in one of two ways:
> > +
> > +1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
> > +   device into the guest via the following virsh commands:
> > +
> > +   virsh attach-device <guestname> <path-to-device-xml>
> > +
> > +      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
> > +      the guest named 'my-guest':
> > +
> > +         virsh attach-device my-guest ~/config/my-guest-hostdev.xml
> > +
> > +            The contents of my-guest-hostdev.xml:
> > +
> > +            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
> > +              <source>
> > +                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
> > +              </source>
> > +            </hostdev>
> > +
> > +
> > +   virsh qemu-monitor-command <guest-name> --hmp \
> > +   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
> > +
> > +      For example, to hot plug the matrix mediated device
> > +      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
> > +      device-id hostdev0:
> > +
> > +      virsh qemu-monitor-command my-guest --hmp \
> > +      "device_add vfio-ap,\
> > +      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
> > +      id=hostdev0"
> > +
> > +2. A matrix mediated device can be hot plugged by attaching the qemu monitor
> > +   to the guest and using the following qemu monitor command:
> > +
> > +      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
> >   
> > -* Dynamically modifying the AP matrix for a running guest (which would amount to
> > -  hot(un)plug of AP devices for the guest) is currently not supported
> > +      For example, to plug the matrix mediated device
> > +      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
> > +      hostdev0:
> >   
> > -* Live guest migration is not supported for guests using AP devices.
> > +         (QEMU) device-add "vfio-ap,\
> > +         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
> > +         id=hostdev0"
> > \ No newline at end of file