Re: [PATCH kernel v7 00/20] powerpc/powernv/npu, vfio: NVIDIA V100 + P9 passthrough

From: Alexey Kardashevskiy
Date: Thu Dec 20 2018 - 06:28:22 EST




On 20/12/2018 20:38, Michael Ellerman wrote:
> Alexey Kardashevskiy <aik@xxxxxxxxx> writes:
>
>> My bad, I was not cc-ing everyone but now with v7 I am, sorry about that.
>
> I've already applied v6, I'll assume this is unchanged from that unless
> you tell me otherwise.

14/20 has fixed warning about uninitialized npdev, 20/20 has fixed
comment about one capability:

[fstn1-p1 kernel]$ git diff 7e04f09 9128bd1
diff --git a/arch/powerpc/platforms/powernv/npu-dma.c
b/arch/powerpc/platforms/powernv/npu-dma.c
index ed81426..12b8421 100644
--- a/arch/powerpc/platforms/powernv/npu-dma.c
+++ b/arch/powerpc/platforms/powernv/npu-dma.c
@@ -540,7 +540,7 @@ struct iommu_table_group
*pnv_try_setup_npu_table_group(struct pnv_ioda_pe *pe)
struct npu_comp *npucomp;
struct pci_dev *gpdev = NULL;
struct pci_controller *hose;
- struct pci_dev *npdev;
+ struct pci_dev *npdev = NULL;

list_for_each_entry(gpdev, &pe->pbus->devices, bus_list) {
npdev = pnv_pci_get_npu_dev(gpdev, 0);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 22b825c..5562587 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -390,8 +390,7 @@ struct vfio_region_info_cap_nvlink2_ssatgt {
};

/*
- * Capability with compressed real address (aka SSA - small system
address),
- * used to match the NVLink bridge with a GPU. Also contains a link speed.
+ * Capability with an NVLink link speed.
*/
#define VFIO_REGION_INFO_CAP_NVLINK2_LNKSPD 5



>
> cheers
>
>> This is for passing through NVIDIA V100 GPUs on POWER9 systems.
>> 20/20 has the details of hardware setup.
>>
>> This implements support for NVIDIA V100 GPU with coherent memory and
>> NPU/ATS support available in the POWER9 CPU. The aim is to support
>> unmodified vendor driver in the guest.
>>
>> This is pushed to (both guest and host kernels):
>> https://github.com/aik/linux/tree/nv2
>>
>> Matching qemu is pushed to github:
>> https://github.com/aik/qemu/tree/nv2
>>
>> Skiboot bits are here:
>> https://github.com/aik/skiboot/tree/nv2
>>
>> The individual patches have changelogs. v7 fixes compile warning
>> and updates a VFIO capability comment in 20/20.
>>
>> Please comment. Thanks.
>>
>>
>>
>> Alexey Kardashevskiy (20):
>> powerpc/ioda/npu: Call skiboot's hot reset hook when disabling NPU2
>> powerpc/mm/iommu/vfio_spapr_tce: Change mm_iommu_get to reference a
>> region
>> powerpc/vfio/iommu/kvm: Do not pin device memory
>> powerpc/powernv: Move npu struct from pnv_phb to pci_controller
>> powerpc/powernv/npu: Move OPAL calls away from context manipulation
>> powerpc/pseries/iommu: Use memory@ nodes in max RAM address
>> calculation
>> powerpc/pseries/npu: Enable platform support
>> powerpc/pseries: Remove IOMMU API support for non-LPAR systems
>> powerpc/powernv/pseries: Rework device adding to IOMMU groups
>> powerpc/iommu_api: Move IOMMU groups setup to a single place
>> powerpc/powernv: Reference iommu_table while it is linked to a group
>> powerpc/powernv/npu: Move single TVE handling to NPU PE
>> powerpc/powernv/npu: Convert NPU IOMMU helpers to
>> iommu_table_group_ops
>> powerpc/powernv/npu: Add compound IOMMU groups
>> powerpc/powernv/npu: Add release_ownership hook
>> powerpc/powernv/npu: Check mmio_atsd array bounds when populating
>> powerpc/powernv/npu: Fault user page into the hypervisor's pagetable
>> vfio_pci: Allow mapping extra regions
>> vfio_pci: Allow regions to add own capabilities
>> vfio_pci: Add NVIDIA GV100GL [Tesla V100 SXM2] subdriver
>>
>> drivers/vfio/pci/Makefile | 1 +
>> arch/powerpc/include/asm/iommu.h | 17 +-
>> arch/powerpc/include/asm/mmu_context.h | 15 +-
>> arch/powerpc/include/asm/pci-bridge.h | 1 +
>> arch/powerpc/include/asm/pci.h | 4 +
>> arch/powerpc/platforms/powernv/pci.h | 30 +-
>> drivers/vfio/pci/trace.h | 102 ++++
>> drivers/vfio/pci/vfio_pci_private.h | 20 +
>> include/uapi/linux/vfio.h | 37 ++
>> arch/powerpc/kernel/iommu.c | 69 +--
>> arch/powerpc/kvm/book3s_64_vio.c | 18 +-
>> arch/powerpc/mm/mmu_context_iommu.c | 110 +++-
>> arch/powerpc/platforms/powernv/npu-dma.c | 549 +++++++++++++++---
>> arch/powerpc/platforms/powernv/pci-ioda-tce.c | 3 +-
>> arch/powerpc/platforms/powernv/pci-ioda.c | 237 ++++----
>> arch/powerpc/platforms/powernv/pci.c | 43 +-
>> arch/powerpc/platforms/pseries/iommu.c | 88 ++-
>> arch/powerpc/platforms/pseries/pci.c | 22 +
>> drivers/vfio/pci/vfio_pci.c | 42 +-
>> drivers/vfio/pci/vfio_pci_nvlink2.c | 482 +++++++++++++++
>> drivers/vfio/vfio_iommu_spapr_tce.c | 64 +-
>> drivers/vfio/pci/Kconfig | 6 +
>> 22 files changed, 1569 insertions(+), 391 deletions(-)
>> create mode 100644 drivers/vfio/pci/trace.h
>> create mode 100644 drivers/vfio/pci/vfio_pci_nvlink2.c
>>
>> --
>> 2.17.1

--
Alexey