Re: [PATCH v6 21/21] s390: doc: detailed specifications for AP virtualization

[Cornelia Huck]

On Tue, 3 Jul 2018 11:22:10 +0200
Halil Pasic <pasic@xxxxxxxxxxxxx> wrote:

> On 07/03/2018 09:46 AM, Harald Freudenberger wrote:
> > On 02.07.2018 18:28, Halil Pasic wrote:  
> >>
> >>
> >> On 06/29/2018 11:11 PM, Tony Krowiak wrote:  
> >>> This patch provides documentation describing the AP architecture and
> >>> design concepts behind the virtualization of AP devices. It also
> >>> includes an example of how to configure AP devices for exclusive
> >>> use of KVM guests.
> >>>
> >>> Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> >>> ---  
> >> [..]  
> >>> +
> >>> +Reserve APQNs for exclusive use of KVM guests
> >>> +---------------------------------------------
> >>> +The following block diagram illustrates the mechanism by which APQNs are
> >>> +reserved:
> >>> +
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------------------+
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ removeÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂ unbind
> >>> +ÂÂÂÂÂÂÂÂ +------------------->+ cex4queue driver +<-----------+
> >>> +ÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------------------+ÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>> ++--------+---------+ register +------------------+ÂÂÂÂÂ +-----+------+
> >>> +|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +<---------+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | bind |ÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +| ap_bus | | vfio_ap driver +<-----+ admin |
> >>> +|ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +--------->+ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ |
> >>> ++------------------+Â probeÂÂ +---+--------+-----+ÂÂÂÂÂ +------------+
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ create |ÂÂÂÂÂÂÂ | store APQN
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ vÂÂÂÂÂÂÂ v
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +---+--------+-----+
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |Â matrix deviceÂÂ |
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
> >>> +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ +------------------+
> >>> +
> >>> +The process for reserving an AP queue for use by a KVM guest is:
> >>> +
> >>> +* The vfio-ap driver during its initialization will perform the following:
> >>> +Â * Create the 'vfio_ap' root device - /sys/devices/vfio_ap
> >>> +Â * Create the 'matrix' device in the 'vfio_ap' root
> >>> +Â * Register the matrix device with the device core
> >>> +* Register with the ap_bus for AP queue devices of type 10 devices (CEX4 and
> >>> +Â newer) and to provide the vfio_ap driver's probe and remove callback
> >>> +Â interfaces. The reason why older devices are not supported is because there
> >>> +Â are no systems available on which to test.
> >>> +* The admin unbinds queue cc.qqqq from the cex4queue device driver. This results
> >>> +Â in the ap_bus calling the the device driver's remove interface which
> >>> +Â unbinds the cc.qqqq queue device from the driver.  
> >>
> >> What if the queue cc.qqqq is already in use? AFAIU unbind is almost as radical as
> >> pulling a cable. What is the proper procedure an admin should follow before doing
> >> the unbind?  
> > What do you mean on this level with 'in use'? A unbind destroys the association
> > between device and driver. There is no awareness of 'in use' or 'not in use' on this
> > level. This is a hard unbind.  
> >>  
> 
> 
> Let me try to invoke the DASD analogy. If one for some reason wants to detach
> a DASD the procedure to follow seems to be (see
> https://www.ibm.com/support/knowledgecenter/en/linuxonibm/com.ibm.linux.z.lgdd/lgdd_t_dasd_online.html)
> the following:
> 1) Unmount.
> 2) Offline possibly using safe_offline.
> 3) Detach.
> 
> Detaching a disk that is currently doing I/O asks for trouble, so the admin is encouraged
> to make sure there is no pending I/O.

I don't think we can use dasd (block devices) as a good analogy for
every kind of device (for starters, consider network devices).

> In case of AP you can interpret my 'in use' as the queue is not empty. In my understanding
> unbind is supposed to be hard (I used the word radical). That's why I compared it to pulling
> a cable. So that's why I ask is there stuff the admin is supposed to do before doing the
> unbind.

Are you asking for a kind of 'quiescing' operation? I would hope that
the crypto drivers already can deal with that via flushing the queue,
not allowing new requests, or whatever. This is not the block device
case.

Anyway, this is an administrative issue. If you don't have a clear
concept which devices are for host usage and which for guest usage, you
already have problems.

Speaking of administrative issues, is there libvirt support for vfio-ap
under development? It would be helpful to validate the approach.