Re: [PATCH 1/2] vfio/type1: Adopt fast IOTLB flush interface when unmap IOVAs

[Suravee Suthikulpanit]

Hi Alex,

On 11/18/17 11:20 AM, Alex Williamson wrote:
> On Fri, 17 Nov 2017 14:51:52 -0700
> Alex Williamson<alex.williamson@xxxxxxxxxx>  wrote:
>
> > On Fri, 17 Nov 2017 15:11:19 -0600
> > Suravee Suthikulpanit<Suravee.Suthikulpanit@xxxxxxx>  wrote:
> >
> > > From: Suravee Suthikulpanit<suravee.suthikulpanit@xxxxxxx>
> > >
> > > VFIO IOMMU type1 currently upmaps IOVA pages synchronously, which requires
> > > IOTLB flushing for every unmapping. This results in large IOTLB flushing
> > > overhead when handling pass-through devices with a large number of mapped
> > > IOVAs (e.g. GPUs).
> > Of course the type of device is really irrelevant, QEMU maps the entire
> > VM address space for any assigned device.
> >
> > > This can be avoided by using the new IOTLB flushing interface.
> > >
> > > Cc: Alex Williamson<alex.williamson@xxxxxxxxxx>
> > > Cc: Joerg Roedel<jroedel@xxxxxxx>
> > > Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@xxxxxxx>
> > > ---
> > >   drivers/vfio/vfio_iommu_type1.c | 12 +++++++++---
> > >   1 file changed, 9 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> > > index 92155cc..28a7ab6 100644
> > > --- a/drivers/vfio/vfio_iommu_type1.c
> > > +++ b/drivers/vfio/vfio_iommu_type1.c
> > > @@ -698,10 +698,12 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma,
> > >   				break;
> > >   		}
> > >   
> > > -		unmapped = iommu_unmap(domain->domain, iova, len);
> > > +		unmapped = iommu_unmap_fast(domain->domain, iova, len);
> > >   		if (WARN_ON(!unmapped))
> > >   			break;
> > >   
> > > +		iommu_tlb_range_add(domain->domain, iova, len);
> > > +
> > We should only add @unmapped, not @len, right?
> Actually, the problems are deeper than that, if we can't guarantee that
> the above iommu_unmap_fast has removed the iommu mapping, then we can't
> do the unpin below as that would potentially allow the device access to
> unknown memory.  Thus, to support this, the unpinning would need to be
> pushed until after the sync and we therefore need some mechanism of
> remembering the phys addresses that we've unmapped.  Thanks,
>
> Alex
>   

If so, I am planning to use a list to temporary store information for 
unmapped regions to be unpinned after sync.  Please lemme know if that 
would be alright.

Suravee