Re: [PATCH v2 4/7] s390: vfio-ap: allow assignment of unavailable AP resources to mdev device

[Tony Krowiak]

On 5/6/19 3:05 AM, Pierre Morel wrote:
> On 03/05/2019 23:14, Tony Krowiak wrote:
> > The AP architecture does not preclude assignment of AP resources that are
> > not yet in the AP configuration (i.e., not available or not online).
> > Let's go ahead and implement this facet of the AP architecture for linux
> > guests.
> >
> > The current implementation does not allow assignment of AP resources to
> > an mdev device if the AP queue devices identified by the assignment are
> > not bound to the vfio_ap device driver. This patch allows assignment 
> > of AP
> > resources to the mdev device even if the AP queue devices are not 
> > bound to
> > the vfio_ap device driver, as long as the AP queue devices are not
> > reserved by the AP BUS for use by the zcrypt device drivers.
>
> or another mediated device.

Right you are!!

> > Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> > ---
> > Â drivers/s390/crypto/vfio_ap_ops.c | 231 
> > ++++++++------------------------------
> > Â 1 file changed, 44 insertions(+), 187 deletions(-)
> >
> > diff --git a/drivers/s390/crypto/vfio_ap_ops.c 
> > b/drivers/s390/crypto/vfio_ap_ops.c
> > index 1021466cb661..ea24caf17a16 100644
> > --- a/drivers/s390/crypto/vfio_ap_ops.c
> > +++ b/drivers/s390/crypto/vfio_ap_ops.c
> > @@ -113,122 +113,6 @@ static struct attribute_group 
> > *vfio_ap_mdev_type_groups[] = {
> > ÂÂÂÂÂ NULL,
> > Â };
> > -struct vfio_ap_queue_reserved {
> > -ÂÂÂ unsigned long *apid;
> > -ÂÂÂ unsigned long *apqi;
> > -ÂÂÂ bool reserved;
> > -};
> > -
> > -/**
> > - * vfio_ap_has_queue
> > - *
> > - * @dev: an AP queue device
> > - * @data: a struct vfio_ap_queue_reserved reference
> > - *
> > - * Flags whether the AP queue device (@dev) has a queue ID containing 
> > the APQN,
> > - * apid or apqi specified in @data:
> > - *
> > - * - If @data contains both an apid and apqi value, then @data will 
> > be flagged
> > - *ÂÂ as reserved if the APID and APQI fields for the AP queue device 
> > matches
> > - *
> > - * - If @data contains only an apid value, @data will be flagged as
> > - *ÂÂ reserved if the APID field in the AP queue device matches
> > - *
> > - * - If @data contains only an apqi value, @data will be flagged as
> > - *ÂÂ reserved if the APQI field in the AP queue device matches
> > - *
> > - * Returns 0 to indicate the input to function succeeded. Returns 
> > -EINVAL if
> > - * @data does not contain either an apid or apqi.
> > - */
> > -static int vfio_ap_has_queue(struct device *dev, void *data)
> > -{
> > -ÂÂÂ struct vfio_ap_queue_reserved *qres = data;
> > -ÂÂÂ struct ap_queue *ap_queue = to_ap_queue(dev);
> > -ÂÂÂ ap_qid_t qid;
> > -ÂÂÂ unsigned long id;
> > -
> > -ÂÂÂ if (qres->apid && qres->apqi) {
> > -ÂÂÂÂÂÂÂ qid = AP_MKQID(*qres->apid, *qres->apqi);
> > -ÂÂÂÂÂÂÂ if (qid == ap_queue->qid)
> > -ÂÂÂÂÂÂÂÂÂÂÂ qres->reserved = true;
> > -ÂÂÂ } else if (qres->apid && !qres->apqi) {
> > -ÂÂÂÂÂÂÂ id = AP_QID_CARD(ap_queue->qid);
> > -ÂÂÂÂÂÂÂ if (id == *qres->apid)
> > -ÂÂÂÂÂÂÂÂÂÂÂ qres->reserved = true;
> > -ÂÂÂ } else if (!qres->apid && qres->apqi) {
> > -ÂÂÂÂÂÂÂ id = AP_QID_QUEUE(ap_queue->qid);
> > -ÂÂÂÂÂÂÂ if (id == *qres->apqi)
> > -ÂÂÂÂÂÂÂÂÂÂÂ qres->reserved = true;
> > -ÂÂÂ } else {
> > -ÂÂÂÂÂÂÂ return -EINVAL;
> > -ÂÂÂ }
> > -
> > -ÂÂÂ return 0;
> > -}
> > -
> > -/**
> > - * vfio_ap_verify_queue_reserved
> > - *
> > - * @matrix_dev: a mediated matrix device
> > - * @apid: an AP adapter ID
> > - * @apqi: an AP queue index
> > - *
> > - * Verifies that the AP queue with @apid/@apqi is reserved by the 
> > VFIO AP device
> > - * driver according to the following rules:
> > - *
> > - * - If both @apid and @apqi are not NULL, then there must be an AP 
> > queue
> > - *ÂÂ device bound to the vfio_ap driver with the APQN identified by 
> > @apid and
> > - *ÂÂ @apqi
> > - *
> > - * - If only @apid is not NULL, then there must be an AP queue device 
> > bound
> > - *ÂÂ to the vfio_ap driver with an APQN containing @apid
> > - *
> > - * - If only @apqi is not NULL, then there must be an AP queue device 
> > bound
> > - *ÂÂ to the vfio_ap driver with an APQN containing @apqi
> > - *
> > - * Returns 0 if the AP queue is reserved; otherwise, returns 
> > -EADDRNOTAVAIL.
> > - */
> > -static int vfio_ap_verify_queue_reserved(unsigned long *apid,
> > -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long *apqi)
> > -{
> > -ÂÂÂ int ret;
> > -ÂÂÂ struct vfio_ap_queue_reserved qres;
> > -
> > -ÂÂÂ qres.apid = apid;
> > -ÂÂÂ qres.apqi = apqi;
> > -ÂÂÂ qres.reserved = false;
> > -
> > -ÂÂÂ ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL,
> > -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ &qres, vfio_ap_has_queue);
> > -ÂÂÂ if (ret)
> > -ÂÂÂÂÂÂÂ return ret;
> > -
> > -ÂÂÂ if (qres.reserved)
> > -ÂÂÂÂÂÂÂ return 0;
> > -
> > -ÂÂÂ return -EADDRNOTAVAIL;
> > -}
> > -
> > -static int
> > -vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev 
> > *matrix_mdev,
> > -ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long apid)
> > -{
> > -ÂÂÂ int ret;
> > -ÂÂÂ unsigned long apqi;
> > -ÂÂÂ unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
> > -
> > -ÂÂÂ if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
> > -ÂÂÂÂÂÂÂ return vfio_ap_verify_queue_reserved(&apid, NULL);
> > -
> > -ÂÂÂ for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
> > -ÂÂÂÂÂÂÂ ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
> > -ÂÂÂÂÂÂÂ if (ret)
> > -ÂÂÂÂÂÂÂÂÂÂÂ return ret;
> > -ÂÂÂ }
> > -
> > -ÂÂÂ return 0;
> > -}
> > -
> > Â /**
> > ÂÂ * vfio_ap_mdev_verify_no_sharing
> > ÂÂ *
> > @@ -236,18 +120,26 @@ 
> > vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev 
> > *matrix_mdev,
> > ÂÂ * and AP queue indexes comprising the AP matrix are not configured 
> > for another
> > ÂÂ * mediated device. AP queue sharing is not allowed.
> > ÂÂ *
> > - * @matrix_mdev: the mediated matrix device
> > + * @mdev_apm: the mask identifying the adapters assigned to mdev
> > + * @mdev_apm: the mask identifying the adapters assigned to mdev
> > ÂÂ *
> > ÂÂ * Returns 0 if the APQNs are not shared, otherwise; returns 
> > -EADDRINUSE.
> > ÂÂ */
> > -static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev 
> > *matrix_mdev)
> > +static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm,
> > +ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ unsigned long *mdev_aqm)
> > Â {
> > ÂÂÂÂÂ struct ap_matrix_mdev *lstdev;
> > ÂÂÂÂÂ DECLARE_BITMAP(apm, AP_DEVICES);
> > ÂÂÂÂÂ DECLARE_BITMAP(aqm, AP_DOMAINS);
> > ÂÂÂÂÂ list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
> > -ÂÂÂÂÂÂÂ if (matrix_mdev == lstdev)
> > +ÂÂÂÂÂÂÂ /*
> > +ÂÂÂÂÂÂÂÂ * If either of the input masks belongs to the mdev to which an
> > +ÂÂÂÂÂÂÂÂ * AP resource is being assigned, then we don't need to verify
> > +ÂÂÂÂÂÂÂÂ * that mdev's masks.
> > +ÂÂÂÂÂÂÂÂ */
> > +ÂÂÂÂÂÂÂ if ((mdev_apm == lstdev->matrix.apm) ||
> > +ÂÂÂÂÂÂÂÂÂÂÂ (mdev_aqm == lstdev->matrix.aqm))
> > ÂÂÂÂÂÂÂÂÂÂÂÂÂ continue;
>
> Is it possible that mdev_apm and mdev_aqm do not belong to the same 
> mediated device?

The mdev_apm and the mdev_aqm will not both belong to the same mdev
device. Either the mdev_apm OR the mdev_aqm will belong to the mdev
device to which the adapter or domain is being assigned.

When an adapter is assigned, the mdev_apm is allocated in the
assign_adapter_store function setting only the bit corresponding to
the APID of the adapter being assigned. The mdev_aqm address is the
address of the matrix_mdev->matrix.aqm.

When a domain is assigned, the mdev_aqm is allocated in the
assign_adapter_store function setting only the bit corresponding to
the APQI of the domain being assigned. The mdev_apm address is the
address of the matrix_mdev->matrix.apm.

> ...snip...