Re: [PATCH v11 0/8] KVM PCIe/MSI passthrough on ARM/ARM64: kernel part 1/3: iommu changes
From: Dennis Chen
Date: Wed Jul 20 2016 - 07:52:53 EST
On Wed, Jul 20, 2016 at 01:03:00PM +0200, Auger Eric wrote:
> Hi Dennis
> On 20/07/2016 11:56, Dennis Chen wrote:
> > Hi Eric,
> > On Tue, Jul 19, 2016 at 12:55:03PM +0000, Eric Auger wrote:
> >> This series introduces the msi-iommu api used to:
> >> - allocate/free resources for MSI IOMMU mapping
> >> - set the MSI iova window aperture
> >> - map/unmap physical addresses onto MSI IOVAs.
> >> - determine whether an msi needs to be iommu mapped
> >> - overwrite an msi_msg PA address with its pre-allocated/mapped IOVA
> >> Also a new iommu domain attribute, DOMAIN_ATTR_MSI_GEOMETRY is introduced
> >> to report the MSI iova window geometry (aperture and iommu-msi API support).
> >> Currently:
> >> - iommu driver is supposed to allocate/free MSI mapping resources
> >> - VFIO subsystem is supposed to set the MSI IOVA aperture.
> >> - The MSI layer is supposed to allocate/free iova mappings and overwrite
> >> msi_msg with IOVA at composition time
> >> More details & context can be found at:
> >> http://www.linaro.org/blog/core-dump/kvm-pciemsi-passthrough-armarm64/
> >> Best Regards
> >> Eric
> >> Git: complete series available at
> >> https://github.com/eauger/linux/tree/v4.7-rc7-passthrough-v11
> > Why can't I find this new series on your git tree:
> > https://git.linaro.org/people/eric.auger/linux.git
> you are not looking at the right git repo: see github one above.
> > ?
> > Also, do I need to download all the 3-part patches to test the PCIe NIC passthru
> > as I did on your v9 series?
> Yes you need to take the 3 parts. You should have everything that is
> needed on the above branch. In case you do not work on Cavium, you
> should not cherry-pick
> "vfio: pci: HACK! workaround thunderx pci_try_reset_bus crash"
Thanks Eric, I've got all the 3 parts from the github git tree in my local repos.
Currently I think the major platform of mine is AMD overdrive. So I will test this patch set
on that platform first.
> > Thanks,
> > Dennis
> >> see part III for wrap-up details.
> >> History:
> >> v10 -> v11:
> >> - no change in the series, just incremented for consistency
> >> - added a temporary patch in the branch:
> >> "iommu/iova: FIXUP! validate iova_domain input to put_iova_domain"
> >> originally sent by Nate and adapted for this use case. This is currently
> >> under discussion on the ML. The crash typically occurs in case unsafe
> >> interrupts are discovered while allow_unsafe_interrupts is not set.
> >> v9 -> v10:
> >> - split error management in iommu_msi_set_aperture
> >> v8 -> v9:
> >> - rename iommu_domain_msi_geometry programmable flag into iommu_msi_supported
> >> - introduce msi_apperture_valid helper and use this instead of is_aperture_set
> >> v7 -> v8:
> >> - The API is retargetted for MSI: renamed msi-iommu
> >> all "dma-reserved" namings removed
> >> - now implemented upon dma-iommu (get, put, init), ie. reuse iova_cookie,
> >> and iova API
> >> - msi mapping resources now are guaranteed to exist during the whole iommu
> >> domain's lifetime. No need to lock to garantee the cookie integrity
> >> - removed alloc/free_reserved_reserved_iova_domain. We now have a single
> >> function that sets the aperture, looking like iommu_dma_init_domain.
> >> - we now use a list instead of an RB-tree
> >> - prot is not propagated anymore at domain creation due to the retargetting
> >> for MSI
> >> - iommu_domain pointer removed from doorbell_mapping struct
> >> - replaced DOMAIN_ATTR_MSI_MAPPING by DOMAIN_ATTR_MSI_GEOMETRY
> >> v6 -> v7:
> >> - fixed known lock bugs and multiple page sized slots matching
> >> (I only have a single MSI frame made of a single page)
> >> - reserved_iova_cookie now pointing to a struct that encapsulates the
> >> iova domain handle + protection attribute passed from VFIO (Alex' req)
> >> - 2 new functions exposed: iommu_msi_mapping_translate_msg,
> >> iommu_msi_mapping_desc_to_domain: not sure this is the right location/proto
> >> though
> >> - iommu_put_reserved_iova now takes a phys_addr_t
> >> - everything now is cleanup on iommu_domain destruction
> >> RFC v5 -> patch v6:
> >> - split to ease the review process
> >> - in dma-reserved-api use a spin lock instead of a mutex (reported by
> >> Jean-Philippe)
> >> - revisit iommu_get_reserved_iova API to pass a size parameter upon
> >> Marc's request
> >> - Consistently use the page order passed when creating the iova domain.
> >> - init reserved_binding_list (reported by Julien)
> >> RFC v4 -> RFC v5:
> >> - take into account Thomas' comments on MSI related patches
> >> - split "msi: IOMMU map the doorbell address when needed"
> >> - increase readability and add comments
> >> - fix style issues
> >> - split "iommu: Add DOMAIN_ATTR_MSI_MAPPING attribute"
> >> - platform ITS now advertises IOMMU_CAP_INTR_REMAP
> >> - fix compilation issue with CONFIG_IOMMU API unset
> >> - arm-smmu-v3 now advertises DOMAIN_ATTR_MSI_MAPPING
> >> RFC v3 -> v4:
> >> - Move doorbell mapping/unmapping in msi.c
> >> - fix ref count issue on set_affinity: in case of a change in the address
> >> the previous address is decremented
> >> - doorbell map/unmap now is done on msi composition. Should allow the use
> >> case for platform MSI controllers
> >> - create dma-reserved-iommu.h/c exposing/implementing a new API dedicated
> >> to reserved IOVA management (looking like dma-iommu glue)
> >> - series reordering to ease the review:
> >> - first part is related to IOMMU
> >> - second related to MSI sub-system
> >> - third related to VFIO (except arm-smmu IOMMU_CAP_INTR_REMAP removal)
> >> - expose the number of requested IOVA pages through VFIO_IOMMU_GET_INFO
> >> [this partially addresses Marc's comments on iommu_get/put_single_reserved
> >> size/alignment problematic - which I did not ignore - but I don't know
> >> how much I can do at the moment]
> >> RFC v2 -> RFC v3:
> >> - should fix wrong handling of some CONFIG combinations:
> >> CONFIG_IOVA, CONFIG_IOMMU_API, CONFIG_PCI_MSI_IRQ_DOMAIN
> >> - fix MSI_FLAG_IRQ_REMAPPING setting in GICv3 ITS (although not tested)
> >> PATCH v1 -> RFC v2:
> >> - reverted to RFC since it looks more reasonable ;-) the code is split
> >> between VFIO, IOMMU, MSI controller and I am not sure I did the right
> >> choices. Also API need to be further discussed.
> >> - iova API usage in arm-smmu.c.
> >> - MSI controller natively programs the MSI addr with either the PA or IOVA.
> >> This is not done anymore in vfio-pci driver as suggested by Alex.
> >> - check irq remapping capability of the group
> >> RFC v1  -> PATCH v1:
> >> - use the existing dma map/unmap ioctl interface with a flag to register a
> >> reserved IOVA range. Use the legacy Rb to store this special vfio_dma.
> >> - a single reserved IOVA contiguous region now is allowed
> >> - use of an RB tree indexed by PA to store allocated reserved slots
> >> - use of a vfio_domain iova_domain to manage iova allocation within the
> >> window provided by the userspace
> >> - vfio alloc_map/unmap_free take a vfio_group handle
> >> - vfio_group handle is cached in vfio_pci_device
> >> - add ref counting to bindings
> >> - user modality enabled at the end of the series
> >> Eric Auger (8):
> >> iommu: Add iommu_domain_msi_geometry and DOMAIN_ATTR_MSI_GEOMETRY
> >> iommu/arm-smmu: initialize the msi geometry and advertise iommu-msi
> >> support
> >> iommu: introduce an msi cookie
> >> iommu/msi-iommu: initialization
> >> iommu/msi-iommu: iommu_msi_[get,put]_doorbell_iova
> >> iommu/msi-iommu: iommu_msi_domain
> >> iommu/msi-iommu: iommu_msi_msg_pa_to_va
> >> iommu/arm-smmu: get/put the msi cookie
> >> drivers/iommu/Kconfig | 7 +
> >> drivers/iommu/Makefile | 1 +
> >> drivers/iommu/arm-smmu-v3.c | 18 ++-
> >> drivers/iommu/arm-smmu.c | 18 ++-
> >> drivers/iommu/iommu.c | 5 +
> >> drivers/iommu/msi-iommu.c | 322 ++++++++++++++++++++++++++++++++++++++++++++
> >> include/linux/iommu.h | 15 +++
> >> include/linux/msi-iommu.h | 144 ++++++++++++++++++++
> >> 8 files changed, 522 insertions(+), 8 deletions(-)
> >> create mode 100644 drivers/iommu/msi-iommu.c
> >> create mode 100644 include/linux/msi-iommu.h
> >> --
> >> 1.9.1
> >> _______________________________________________
> >> kvmarm mailing list
> >> kvmarm@xxxxxxxxxxxxxxxxxxxxx
> >> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm