Re: [PATCH v11 26/26] s390: doc: detailed specifications for AP virtualization

[Christian Borntraeger]

On 09/27/2018 09:19 PM, Tony Krowiak wrote:

> The following fixup attempts to clarify the bit ordering confusion,
> hopefully this is acceptable.
> 

looks good to me, I will fold in.

> -----------------------------------8<-----------------------------------
> 
> From: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> Date: Thu, 27 Sep 2018 14:51:12 -0400
> Subject: [FIXUP v10] fixup! s390: doc: detailed specifications for AP
> Âvirtualization
> 
> Better explains mask bit ordering.
> 
> Signed-off-by: Tony Krowiak <akrowiak@xxxxxxxxxxxxx>
> ---
> ÂDocumentation/s390/vfio-ap.txt | 127 +++++++++++++++++++++++----------
> Â1 file changed, 91 insertions(+), 36 deletions(-)
> 
> diff --git a/Documentation/s390/vfio-ap.txt b/Documentation/s390/vfio-ap.txt
> index bec67eb7141c..599eb0f75c07 100644
> --- a/Documentation/s390/vfio-ap.txt
> +++ b/Documentation/s390/vfio-ap.txt
> @@ -123,21 +123,24 @@ to identify the adapters, usage domains and control domains assigned to the KVM
> Âguest:
> 
> Â* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
> -Â to the KVM guest. Each bit in the mask, from most significant to least
> -Â significant bit, corresponds to an APID from 0-255. If a bit is set, the
> -Â corresponding adapter is valid for use by the KVM guest.
> +Â to the KVM guest. Each bit in the mask, from left to right (i.e. from most
> +Â significant to least significant bit in big endian order), corresponds to
> +Â an APID from 0-255. If a bit is set, the corresponding adapter is valid for
> +Â use by the KVM guest.
> 
> Â* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
> -Â assigned to the KVM guest. Each bit in the mask, from most significant to
> -Â least significant bit, corresponds to an AP queue index (APQI) from 0-255. If
> -Â a bit is set, the corresponding queue is valid for use by the KVM guest.
> +Â assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from
> +Â most significant to least significant bit in big endian order), corresponds to
> +Â an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue
> +Â is valid for use by the KVM guest.
> 
> Â* The AP Domain Mask field is a bit mask that identifies the AP control domains
> ÂÂ assigned to the KVM guest. The ADM bit mask controls which domains can be
> ÂÂ changed by an AP command-request message sent to a usage domain from the
> -Â guest. Each bit in the mask, from least significant to most significant bit,
> -Â corresponds to a domain from 0-255. If a bit is set, the corresponding domain
> -Â can be modified by an AP command-request message sent to a usage domain.
> +Â guest. Each bit in the mask, from left to right (i.e. from most significant to
> +Â least significant bit in big endian order), corresponds to a domain from
> +Â 0-255. If a bit is set, the corresponding domain can be modified by an AP
> +Â command-request message sent to a usage domain.
> 
> ÂIf you recall from the description of an AP Queue, AP instructions include
> Âan APQN to identify the AP queue to which an AP command-request message is to be
> @@ -503,23 +506,34 @@ These are the steps:
> ÂÂÂ access them. To secure them, there are two sysfs files that specify
> ÂÂÂ bitmasks marking a subset of the APQN range as 'usable by the default AP
> ÂÂÂ queue device drivers' or 'not usable by the default device drivers' and thus
> -ÂÂ available for use by the vfio_ap device driver'. The sysfs files containing
> -ÂÂ the sysfs locations of the masks are:
> +ÂÂ available for use by the vfio_ap device driver'. The location of the sysfs
> +ÂÂ files containing the masks are:
> 
> ÂÂÂ /sys/bus/ap/apmask
> ÂÂÂ /sys/bus/ap/aqmask
> 
> ÂÂÂ The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
> -ÂÂ (APID). Each bit in the mask, from most significant to least significant bit,
> -ÂÂ corresponds to an APID from 0-255. If a bit is set, the APID is marked as
> -ÂÂ usable only by the default AP queue device drivers; otherwise, the APID is
> -ÂÂ usable by the vfio_ap device driver.
> +ÂÂ (APID). Each bit in the mask, from left to right (i.e., from most significant
> +ÂÂ to least significant bit in big endian order), corresponds to an APID from
> +ÂÂ 0-255. If a bit is set, the APID is marked as usable only by the default AP
> +ÂÂ queue device drivers; otherwise, the APID is usable by the vfio_ap
> +ÂÂ device driver.
> 
> ÂÂÂ The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
> -ÂÂ (APQI). Each bit in the mask, from most significant to least significant bit,
> -ÂÂ corresponds to an APQI from 0-255. If a bit is set, the APQI is marked as
> -ÂÂ usable only by the default AP queue device drivers; otherwise, the APQI is
> -ÂÂ usable by the vfio_ap device driver.
> +ÂÂ (APQI). Each bit in the mask, from left to right (i.e., from most significant
> +ÂÂ to least significant bit in big endian order), corresponds to an APQI from
> +ÂÂ 0-255. If a bit is set, the APQI is marked as usable only by the default AP
> +ÂÂ queue device drivers; otherwise, the APQI is usable by the vfio_ap device
> +ÂÂ driver.
> +
> +ÂÂ Take, for example, the following mask:
> +
> +ÂÂÂÂÂ 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
> +
> +ÂÂÂ It indicates:
> +
> +ÂÂÂÂÂ 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
> +ÂÂÂÂÂ belong to the vfio_ap device driver's pool.
> 
> ÂÂÂ The APQN of each AP queue device assigned to the linux host is checked by the
> ÂÂÂ AP bus against the set of APQNs derived from the cross product of APIDs
> @@ -530,38 +544,79 @@ These are the steps:
> ÂÂÂ By default, the two masks are set to reserve all APQNs for use by the default
> ÂÂÂ AP queue device drivers. There are two ways the default masks can be changed:
> 
> -ÂÂ 1. The masks can be changed at boot time with the kernel command line
> -ÂÂÂÂÂ like this:
> +ÂÂ 1. The sysfs mask files can be edited by echoing a string into the
> +ÂÂÂÂÂ respective sysfs mask file in one of two formats:
> +
> +ÂÂÂÂÂ * An absolute hex string starting with 0x - like "0x12345678" - sets
> +ÂÂÂÂÂÂÂ the mask. If the given string is shorter than the mask, it is padded
> +ÂÂÂÂÂÂÂ with 0s on the right; for example, specifying a mask value of 0x41 is
> +ÂÂÂÂÂÂÂ the same as specifying:
> +
> + 0x4100000000000000000000000000000000000000000000000000000000000000
> +
> +ÂÂÂÂÂÂÂ Keep in mind that the mask reads from left to right (i.e., most
> +ÂÂÂÂÂÂÂ significant to least significant bit in big endian order), so the mask
> +ÂÂÂÂÂÂÂ above identifies device numbers 1 and 7 (01000001).
> +
> +ÂÂÂÂÂÂÂ If the string is longer than the mask, the operation is terminated with
> +ÂÂÂÂÂÂÂ an error (EINVAL).
> +
> +ÂÂÂÂÂ * Individual bits in the mask can be switched on and off by specifying
> +ÂÂÂÂÂÂÂ each bit number to be switched in a comma separated list. Each bit
> +ÂÂÂÂÂÂÂ number string must be prepended with a ('+') or minus ('-') to indicate
> +ÂÂÂÂÂÂÂ the corresponding bit is to be switched on ('+') or off ('-'). Some
> +ÂÂÂÂÂÂÂ valid values are:
> +
> +ÂÂÂÂÂÂÂÂÂÂ "+0"ÂÂÂ switches bit 0 on
> +ÂÂÂÂÂÂÂÂÂÂ "-13"ÂÂ switches bit 13 off
> +ÂÂÂÂÂÂÂÂÂÂ "+0x41" switches bit 65 on
> +ÂÂÂÂÂÂÂÂÂÂ "-0xff" switches bit 255 off
> +
> +ÂÂÂÂÂÂÂÂÂÂ The following example:
> +ÂÂÂÂÂÂÂÂÂÂÂÂÂ +0,-6,+0x47,-0xf0
> +
> +ÂÂÂÂÂÂÂÂÂÂÂÂÂ Switches bits 0 and 71 (0x47) on
> +ÂÂÂÂÂÂÂÂÂÂÂÂÂ Switches bits 6 and 240 (0xf0) off
> +
> +ÂÂÂÂÂÂÂ Note that the bits not specified in the list remain as they were before
> +ÂÂÂÂÂÂÂ the operation.
> +
> +ÂÂ 2. The masks can also be changed at boot time via parameters on the kernel
> +ÂÂÂÂÂ command line like this:
> 
> ÂÂÂÂÂÂÂÂÂ ap.apmask=0xffff ap.aqmask=0x40
> 
> -ÂÂÂÂÂÂÂÂ This would give these two pools:
> +ÂÂÂÂÂÂÂÂ This would create the following masks:
> 
> -ÂÂÂÂÂÂÂÂÂÂÂ default drivers pool:ÂÂÂ adapter 0-15, domain 1
> -ÂÂÂÂÂÂÂÂÂÂÂ alternate drivers pool:Â adapter 16-255, domains 2-255
> +ÂÂÂÂÂÂÂÂÂÂÂ apmask:
> + 0xffff000000000000000000000000000000000000000000000000000000000000
> 
> -ÂÂ 2. The sysfs mask files can also be edited by echoing a string into the
> -ÂÂÂÂÂ respective file in one of two formats:
> +ÂÂÂÂÂÂÂÂÂÂÂ aqmask:
> + 0x4000000000000000000000000000000000000000000000000000000000000000
> 
> -ÂÂÂÂÂ * An absolute hex string starting with 0x - like "0x12345678" - sets
> -ÂÂÂÂÂÂÂ the mask. If the given string is shorter than the mask, it is padded
> -ÂÂÂÂÂÂÂ with 0s on the right. If the string is longer than the mask, the
> -ÂÂÂÂÂÂÂ operation is terminated with an error (EINVAL).
> +ÂÂÂÂÂÂÂÂ Resulting in these two pools:
> 
> -ÂÂÂÂÂ * A plus ('+') or minus ('-') followed by a numerical value. Valid
> -ÂÂÂÂÂÂÂ examples are "+1", "-13", "+0x41", "-0xff" and even "+0" and "-0". Only
> -ÂÂÂÂÂÂÂ the corresponding bit in the mask is switched on ('+') or off ('-'). The
> -ÂÂÂÂÂÂÂ values may also be specified in a comma-separated list to switch more
> -ÂÂÂÂÂÂÂ than one bit on or off.
> +ÂÂÂÂÂÂÂÂÂÂÂ default drivers pool:ÂÂÂ adapter 0-15, domain 1
> +ÂÂÂÂÂÂÂÂÂÂÂ alternate drivers pool:Â adapter 16-255, domains 0, 2-255
> 
> +ÂÂ Securing the APQNs for our example:
> +ÂÂ ----------------------------------
> ÂÂÂ To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
> ÂÂÂ 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
> -ÂÂ APQNs must be removed from the masks as follows:
> +ÂÂ APQNs can either be removed from the default masks:
> 
> ÂÂÂÂÂÂ echo -5,-6 > /sys/bus/ap/apmask
> 
> ÂÂÂÂÂÂ echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
> 
> +ÂÂ Or the masks can be set as follows:
> +
> +ÂÂÂÂÂ echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
> +ÂÂÂÂÂ > apmask
> +
> +ÂÂÂÂÂ echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
> +ÂÂÂÂÂ > aqmask
> +
> ÂÂÂ This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
> ÂÂÂ 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
> ÂÂÂ sysfs directory for the vfio_ap device driver will now contain symbolic links