[PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver.

[Dave Jiang]

The actual code is independent of the stage 2 driver code submission that adds
support for SVM, ENQCMD(S), PASID, and shared workqueues. This code series will
support dedicated workqueue on a guest with no vIOMMU.
  
A new device type "mdev" is introduced for the idxd driver. This allows the wq
to be dedicated to the usage of a VFIO mediated device (mdev). Once the work
queue (wq) is enabled, an uuid generated by the user can be added to the wq
through the uuid sysfs attribute for the wq.  After the association, a mdev can
be created using this UUID. The mdev driver code will associate the uuid and
setup the mdev on the driver side. When the create operation is successful, the
uuid can be passed to qemu. When the guest boots up, it should discover a DSA
device when doing PCI discovery.

For example:
1. Enable wq with âmdevâ wq type
2. A user generated UUID is associated with a wq:
echo $UUID > /sys/bus/dsa/devices/wq0.0/uuid
3. The uuid is written to the mdev class sysfs path:
echo $UUID > /sys/class/mdev_bus/0000\:00\:0a.0/mdev_supported_types/idxd-wq/create
4. Pass the following parameter to qemu:
"-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:00:0a.0/$UUID"
 
Since the mdev is an emulated device with a single wq, the guest will see a DSA
device with a single wq. With no vIOMMU support, the behavior will be the same
as the stage 1 driver running with no IOMMU turned on on the bare metal host. 
The difference is that the wq exported through mdev will have the read only
config bit set for configuration. This means that the device does not require
the typical configuration. After enabling the device, the user must set the WQ
type and name. That is all is necessary to enable the WQ and start using it.
The single wq configuration is not the only way to create the mdev. Multi wq
support for mdev will be in the future works.
 
The mdev utilizes Interrupt Message Store or IMS[3] instead of MSIX for
interrupts for the guest. This preserves MSIX for host usages and also allows a
significantly larger number of interrupt vectors for guest usage.

The idxd driver implements IMS as on-device memory mapped unified storage. Each
interrupt message is stored as a DWORD size data payload and a 64-bit address
(same as MSI-X). Access to the IMS is through the host idxd driver. All the IMS
interrupt messages are stored in the remappable format. Hence, if the driver
enables IMS, interrupt remapping is also enabled by default. 
 
This patchset extends the existing platfrom-msi.c which already provides a
generic mechanism to support non-PCI compliant MSI interrupts for platform
devices to provide the IMS infrastructure. 

More details about IMS, its implementation in the the kernel, common
misconceptions about IMS and the basic driver changes required to support IMS
can be found under Documentations/interrupt_message_store.txt

[1]: https://lore.kernel.org/lkml/157965011794.73301.15960052071729101309.stgit@xxxxxxxxxxxxxxxxxxxxxxxxxx/
[2]: https://software.intel.com/en-us/articles/intel-sdm
[3]: https://software.intel.com/en-us/download/intel-scalable-io-virtualization-technical-specification
[4]: https://software.intel.com/en-us/download/intel-data-streaming-accelerator-preliminary-architecture-specification
[5]: https://01.org/blogs/2019/introducing-intel-data-streaming-accelerator
[6]: https://intel.github.io/idxd/
[7]: https://github.com/intel/idxd-driver idxd-stage3

---

Dave Jiang (5):
      dmaengine: idxd: add config support for readonly devices
      dmaengine: idxd: add IMS support in base driver
      dmaengine: idxd: add device support functions in prep for mdev
      dmaengine: idxd: add support for VFIO mediated device
      dmaengine: idxd: add error notification from host driver to mediated device

Jing Lin (1):
      dmaengine: idxd: add ABI documentation for mediated device support

Lu Baolu (2):
      vfio/mdev: Add a member for iommu domain in mdev_device
      vfio/type1: Save domain when attach domain to mdev

Megha Dey (7):
      drivers/base: Introduce platform_msi_ops
      drivers/base: Introduce a new platform-msi list
      drivers/base: Allocate/free platform-msi interrupts by group
      drivers/base: Add support for a new IMS irq domain
      ims-msi: Add mask/unmask routines
      ims-msi: Enable IMS interrupts
      Documentation: Interrupt Message store


 Documentation/ABI/stable/sysfs-driver-dma-idxd |   18 
 Documentation/ims-howto.rst                    |  210 +++
 arch/x86/include/asm/hw_irq.h                  |    7 
 arch/x86/include/asm/irq_remapping.h           |    6 
 drivers/base/Kconfig                           |    9 
 drivers/base/Makefile                          |    1 
 drivers/base/core.c                            |    1 
 drivers/base/ims-msi.c                         |  162 ++
 drivers/base/platform-msi.c                    |  202 ++-
 drivers/dma/Kconfig                            |    4 
 drivers/dma/idxd/Makefile                      |    2 
 drivers/dma/idxd/cdev.c                        |    3 
 drivers/dma/idxd/device.c                      |  325 ++++-
 drivers/dma/idxd/dma.c                         |    9 
 drivers/dma/idxd/idxd.h                        |   55 +
 drivers/dma/idxd/init.c                        |   81 +
 drivers/dma/idxd/irq.c                         |    6 
 drivers/dma/idxd/mdev.c                        | 1727 ++++++++++++++++++++++++
 drivers/dma/idxd/mdev.h                        |  105 +
 drivers/dma/idxd/registers.h                   |   10 
 drivers/dma/idxd/submit.c                      |   31 
 drivers/dma/idxd/sysfs.c                       |  199 ++-
 drivers/dma/idxd/vdev.c                        |  603 ++++++++
 drivers/dma/idxd/vdev.h                        |   43 +
 drivers/dma/mv_xor_v2.c                        |    6 
 drivers/dma/qcom/hidma.c                       |    6 
 drivers/iommu/arm-smmu-v3.c                    |    6 
 drivers/iommu/intel-iommu.c                    |    2 
 drivers/iommu/intel_irq_remapping.c            |   31 
 drivers/irqchip/irq-mbigen.c                   |    8 
 drivers/irqchip/irq-mvebu-icu.c                |    6 
 drivers/mailbox/bcm-flexrm-mailbox.c           |    6 
 drivers/perf/arm_smmuv3_pmu.c                  |    6 
 drivers/vfio/mdev/mdev_core.c                  |   22 
 drivers/vfio/mdev/mdev_private.h               |    2 
 drivers/vfio/vfio_iommu_type1.c                |   52 +
 include/linux/device.h                         |    3 
 include/linux/intel-iommu.h                    |    3 
 include/linux/list.h                           |   36 +
 include/linux/mdev.h                           |   13 
 include/linux/msi.h                            |   93 +
 kernel/irq/msi.c                               |   43 -
 42 files changed, 4009 insertions(+), 154 deletions(-)
 create mode 100644 Documentation/ims-howto.rst
 create mode 100644 drivers/base/ims-msi.c
 create mode 100644 drivers/dma/idxd/mdev.c
 create mode 100644 drivers/dma/idxd/mdev.h
 create mode 100644 drivers/dma/idxd/vdev.c
 create mode 100644 drivers/dma/idxd/vdev.h

--