[PATCH v11 0/8] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 3/3: vfio changes

From: Eric Auger
Date: Tue Jul 19 2016 - 09:11:50 EST

This series allows the user-space to register a reserved IOVA domain.
This completes the kernel integration of the whole functionality on top
of the 2 previous parts (v11).

We reuse the VFIO DMA MAP ioctl with a new flag to bridge to the
msi-iommu API. The need for provisioning such MSI IOVA range is reported
through capability chain, using VFIO_IOMMU_TYPE1_INFO_CAP_MSI_GEOMETRY.

vfio_iommu_type1 checks if the MSI mapping is safe when attaching the
vfio group to the container (allow_unsafe_interrupts modality). This is
done in a very coarse way, looking at all the registered doorbells and
returning assignment is safe wrt IRQ if all the doorbells are safe.

More details & context can be found at:

Best Regards


- functional on ARM64 AMD Overdrive HW (single GICv2m frame) with
Intel I350T2 (SR-IOV capable, igb/igbvf) and Intel 82574L (e1000e)
- functional on Cavium ThunderX (ARM GICv3 ITS) with Intel 82574L (e1000e)

[1] [RFC 0/2] VFIO: Add virtual MSI doorbell support
[2] [RFC PATCH 0/6] vfio: Add interface to map MSI pages
[3] [PATCH v2 0/3] Introduce MSI hardware mapping for VFIO

Git: complete series available at

the above branch includes a temporary patch to work around a ThunderX pci
bus reset crash (which I think unrelated to this series):
"vfio: pci: HACK! workaround thunderx pci_try_reset_bus crash"
Do not take this for other platforms.
This issue will be handled separately on the ML.

previous version at

v10 -> v11:
- no change to this series, just incremented for consistency
- eventually tested with GICv3 ITS on Cavium

v9 -> v10:
Took into account Alex' comments:
- split "vfio/type1: vfio_find_dma accepting a type argument" into 2 patches
- properly implement replay of MSI DMA slots (by setting the aperture on the
new domain); fixes the assignment of several devices
- rework user api for vfio_iommu_type1_info capability chains (cap_offset
at the end of the struct and fix padding issues)
- explicit dma->type setting to VFIO_IOVA_USER

v8 -> v9:
- report MSI geometry through capability chain (last patch only);
with the current limitation that an arbitrary number of 16 page
requirement is reported. To be improved later on.

v7 -> v8:
- use renamed msi-iommu API
- VFIO only responsible for setting the IOVA aperture
- use new DOMAIN_ATTR_MSI_GEOMETRY iommu domain attribute

v6 -> v7:
- vfio_find_dma now accepts a dma_type argument.
- should have recovered the capability to unmap the whole user IOVA range
- remove computation of nb IOVA pages -> will post a separate RFC for that
while respinning the QEMU part

RFC v5 -> patch v6:
- split to ease the review process

RFC v4 -> RFC v5:
- take into account Thomas' comments on MSI related patches
- split "msi: IOMMU map the doorbell address when needed"
- increase readability and add comments
- fix style issues
- split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
- platform ITS now advertises IOMMU_CAP_INTR_REMAP
- fix compilation issue with CONFIG_IOMMU API unset
- arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING

RFC v3 -> v4:
- Move doorbell mapping/unmapping in msi.c
- fix ref count issue on set_affinity: in case of a change in the address
the previous address is decremented
- doorbell map/unmap now is done on msi composition. Should allow the use
case for platform MSI controllers
- create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
to reserved IOVA management (looking like dma-iommu glue)
- series reordering to ease the review:
- first part is related to IOMMU
- second related to MSI sub-system
- third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
- expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
[this partially addresses Marc's comments on iommu_get/put_single_reserved
size/alignment problematic - which I did not ignore - but I don't know
how much I can do at the moment]

RFC v2 -> RFC v3:
- should fix wrong handling of some CONFIG combinations:
- fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)

PATCH v1 -> RFC v2:
- reverted to RFC since it looks more reasonable ;-) the code is split
between VFIO, IOMMU, MSI controller and I am not sure I did the right
choices. Also API need to be further discussed.
- iova API usage in arm-smmu.c.
- MSI controller natively programs the MSI addr with either the PA or IOVA.
This is not done anymore in vfio-pci driver as suggested by Alex.
- check irq remapping capability of the group

RFC v1 [2] -> PATCH v1:
- use the existing dma map/unmap ioctl interface with a flag to register a
reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
- a single reserved IOVA contiguous region now is allowed
- use of an RB tree indexed by PA to store allocated reserved slots
- use of a vfio_domain iova_domain to manage iova allocation within the
window provided by the userspace
- vfio alloc_map/unmap_free take a vfio_group handle
- vfio_group handle is cached in vfio_pci_device
- add ref counting to bindings
- user modality enabled at the end of the series

Eric Auger (8):
vfio: introduce a vfio_dma type field
vfio/type1: vfio_find_dma accepting a type argument
vfio/type1: implement recursive vfio_find_dma_from_node
vfio/type1: handle unmap/unpin and replay for VFIO_IOVA_RESERVED slots
vfio: allow reserved msi iova registration
vfio/type1: check doorbell safety
iommu/arm-smmu: do not advertise IOMMU_CAP_INTR_REMAP
vfio/type1: return MSI geometry through VFIO_IOMMU_GET_INFO capability

drivers/iommu/arm-smmu-v3.c | 3 +-
drivers/iommu/arm-smmu.c | 3 +-
drivers/vfio/vfio_iommu_type1.c | 246 +++++++++++++++++++++++++++++++++++++---
include/uapi/linux/vfio.h | 42 ++++++-
4 files changed, 276 insertions(+), 18 deletions(-)