Re: [PATCH v6 00/20] Add VFIO mediated device support and DEV-MSI support for the idxd driver

[Jason Gunthorpe]

On Fri, Jun 11, 2021 at 11:21:42AM -0700, Dave Jiang wrote:
> 
> On 6/8/2021 9:02 AM, Dave Jiang wrote:
> > 
> > On 6/7/2021 12:11 PM, Jason Gunthorpe wrote:
> > > On Mon, Jun 07, 2021 at 11:13:04AM -0700, Dave Jiang wrote:
> > > 
> > > > So in step 1, we 'tag' the wq to be dedicated to guest usage and
> > > > put the
> > > > hardware wq into enable state. For a dedicated mode wq, we can
> > > > definitely
> > > > just register directly and skip the mdev step. For a shared wq
> > > > mode, we can
> > > > have multiple mdev running on top of a single wq. So we need
> > > > some way to
> > > > create more mdevs. We can either go with the existing
> > > > established creation
> > > > path by mdev, or invent something custom for the driver as Jason
> > > > suggested
> > > > to accomodate additional virtual devices for guests. We
> > > > implemented the mdev
> > > > path originally with consideration of mdev is established and
> > > > has a known
> > > > interface already.
> > > It sounds like you could just as easially have a 'create new vfio'
> > > file under the idxd sysfs.. Especially since you already have a bus
> > > and dynamic vfio specific things being created on this bus.
> > 
> > Will explore this and using of 'struct vfio_device' without mdev.
> > 
> Hi Jason. I hacked the idxd driver to remove mdev association and use
> vfio_device directly. Ran into some issues. Specifically mdev does some
> special handling when it comes to iommu domain.

Yes, I know of this, it this needs fixing.

> effect of special handling for iommu_attach_group. And in addition, it ends
> up switching the bus to pci_bus_type before iommu_domain_alloc() is called. 
> Do we need to provide similar type of handling for vfio_device that are not
> backed by an entire PCI device like vfio_pci? Not sure it's the right thing
> to do to attach these devices to pci_bus_type directly.

Yes, type1 needs to be changed so it somehow knows that the struct
device is 'sw only', for instance because it has no direct HW IOMMU
connection, then all the mdev hackery should be deleted from type1.

I haven't investigated closely exactly how to do this.

Jason