Re: [PATCH v6 00/20] Add VFIO mediated device support and DEV-MSI support for the idxd driver

[Dave Jiang]

On 6/8/2021 9:02 AM, Dave Jiang wrote:
> On 6/7/2021 12:11 PM, Jason Gunthorpe wrote:
> > On Mon, Jun 07, 2021 at 11:13:04AM -0700, Dave Jiang wrote:
> >
> > > So in step 1, we 'tag' the wq to be dedicated to guest usage and put 
> > > the
> > > hardware wq into enable state. For a dedicated mode wq, we can 
> > > definitely
> > > just register directly and skip the mdev step. For a shared wq mode, 
> > > we can
> > > have multiple mdev running on top of a single wq. So we need some 
> > > way to
> > > create more mdevs. We can either go with the existing established 
> > > creation
> > > path by mdev, or invent something custom for the driver as Jason 
> > > suggested
> > > to accomodate additional virtual devices for guests. We implemented 
> > > the mdev
> > > path originally with consideration of mdev is established and has a 
> > > known
> > > interface already.
> > It sounds like you could just as easially have a 'create new vfio'
> > file under the idxd sysfs.. Especially since you already have a bus
> > and dynamic vfio specific things being created on this bus.
>
> Will explore this and using of 'struct vfio_device' without mdev.
Hi Jason. I hacked the idxd driver to remove mdev association and use 
vfio_device directly. Ran into some issues. Specifically mdev does some 
special handling when it comes to iommu domain. When we hit 
vfio_iommu_type1_attach_group(), there's a branch in there for 
mdev_bus_type. It sets the group with mdev_group flag, which later has 
effect of special handling for iommu_attach_group. And in addition, it 
ends up switching the bus to pci_bus_type before iommu_domain_alloc() is 
called.  Do we need to provide similar type of handling for vfio_device 
that are not backed by an entire PCI device like vfio_pci? Not sure it's 
the right thing to do to attach these devices to pci_bus_type directly.