Re: [PATCH v6 21/21] s390: doc: detailed specifications for AP virtualization

[Tony Krowiak]

On 07/03/2018 07:52 AM, Cornelia Huck wrote:
> On Tue, 3 Jul 2018 11:22:10 +0200
> Halil Pasic <pasic@xxxxxxxxxxxxx> wrote:
>
> > On 07/03/2018 09:46 AM, Harald Freudenberger wrote:
> > > On 02.07.2018 18:28, Halil Pasic wrote:
> > > > On 06/29/2018 11:11 PM, Tony Krowiak wrote:
> > > > > This patch provides documentation describing the AP architecture and
> > > > > design concepts behind the virtualization of AP devices. It also
> > > > > includes an example of how to configure AP devices for exclusive
> > > > > use of KVM guests.
> > > > >
> > > > > Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> > > > > ---
> > > > [..]
> > > > > +
> > > > > +Reserve APQNs for exclusive use of KVM guests
> > > > > +---------------------------------------------
> > > > > +The following block diagram illustrates the mechanism by which APQNs are
> > > > > +reserved:
> > > > > +
> > > > > +                              +------------------+
> > > > > +                 remove       |                  |   unbind
> > > > > +         +------------------->+ cex4queue driver +<-----------+
> > > > > +         |                    |                  |            |
> > > > > +         |                    +------------------+            |
> > > > > +         |                                                    |
> > > > > +         |                                                    |
> > > > > +         |                                                    |
> > > > > ++--------+---------+ register +------------------+      +-----+------+
> > > > > +|                  +<---------+                  | bind |            |
> > > > > +|      ap_bus      |          |  vfio_ap driver  +<-----+    admin   |
> > > > > +|                  +--------->+                  |      |            |
> > > > > ++------------------+  probe   +---+--------+-----+      +------------+
> > > > > +                                  |        |
> > > > > +                           create |        | store APQN
> > > > > +                                  |        |
> > > > > +                                  v        v
> > > > > +                              +---+--------+-----+
> > > > > +                              |                  |
> > > > > +                              |  matrix device   |
> > > > > +                              |                  |
> > > > > +                              +------------------+
> > > > > +
> > > > > +The process for reserving an AP queue for use by a KVM guest is:
> > > > > +
> > > > > +* The vfio-ap driver during its initialization will perform the following:
> > > > > +  * Create the 'vfio_ap' root device - /sys/devices/vfio_ap
> > > > > +  * Create the 'matrix' device in the 'vfio_ap' root
> > > > > +  * Register the matrix device with the device core
> > > > > +* Register with the ap_bus for AP queue devices of type 10 devices (CEX4 and
> > > > > +  newer) and to provide the vfio_ap driver's probe and remove callback
> > > > > +  interfaces. The reason why older devices are not supported is because there
> > > > > +  are no systems available on which to test.
> > > > > +* The admin unbinds queue cc.qqqq from the cex4queue device driver. This results
> > > > > +  in the ap_bus calling the the device driver's remove interface which
> > > > > +  unbinds the cc.qqqq queue device from the driver.
> > > > What if the queue cc.qqqq is already in use? AFAIU unbind is almost as radical as
> > > > pulling a cable. What is the proper procedure an admin should follow before doing
> > > > the unbind?
> > > What do you mean on this level with 'in use'? A unbind destroys the association
> > > between device and driver. There is no awareness of 'in use' or 'not in use' on this
> > > level. This is a hard unbind.
> > > >   
> >
> > Let me try to invoke the DASD analogy. If one for some reason wants to detach
> > a DASD the procedure to follow seems to be (see
> > https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lgdd/lgdd_t_dasd_online.html)
> > the following:
> > 1) Unmount.
> > 2) Offline possibly using safe_offline.
> > 3) Detach.
> >
> > Detaching a disk that is currently doing I/O asks for trouble, so the admin is encouraged
> > to make sure there is no pending I/O.
> I don't think we can use dasd (block devices) as a good analogy for
> every kind of device (for starters, consider network devices).
>
> > In case of AP you can interpret my 'in use' as the queue is not empty. In my understanding
> > unbind is supposed to be hard (I used the word radical). That's why I compared it to pulling
> > a cable. So that's why I ask is there stuff the admin is supposed to do before doing the
> > unbind.
> Are you asking for a kind of 'quiescing' operation? I would hope that
> the crypto drivers already can deal with that via flushing the queue,
> not allowing new requests, or whatever. This is not the block device
> case.

As I stated in Message ID: 
<89df60be-63d6-3ed9-4724-321e5b55d50d@xxxxxxxxxxxxx>,
I believe the queue is flushed when the remove callback is invoked on the
driver.

> Anyway, this is an administrative issue. If you don't have a clear
> concept which devices are for host usage and which for guest usage, you
> already have problems.
>
> Speaking of administrative issues, is there libvirt support for vfio-ap
> under development? It would be helpful to validate the approach.

There is libvirt support under development although it is not very far
along at this point.

>