RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)

[Shameerali Kolothum Thodi]

Hi Eric,

> -----Original Message-----
> From: Shameerali Kolothum Thodi
> Sent: 30 April 2020 10:38
> To: 'Auger Eric' <eric.auger@xxxxxxxxxx>; Zhangfei Gao
> <zhangfei.gao@xxxxxxxxxx>; eric.auger.pro@xxxxxxxxx;
> iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> kvm@xxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; will@xxxxxxxxxx;
> joro@xxxxxxxxxx; maz@xxxxxxxxxx; robin.murphy@xxxxxxx
> Cc: jean-philippe@xxxxxxxxxx; alex.williamson@xxxxxxxxxx;
> jacob.jun.pan@xxxxxxxxxxxxxxx; yi.l.liu@xxxxxxxxx; peter.maydell@xxxxxxxxxx;
> tn@xxxxxxxxxxxx; bbhushan2@xxxxxxxxxxx
> Subject: RE: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
> 
> Hi Eric,
> 
> > -----Original Message-----
> > From: Auger Eric [mailto:eric.auger@xxxxxxxxxx]
> > Sent: 16 April 2020 08:45
> > To: Zhangfei Gao <zhangfei.gao@xxxxxxxxxx>; eric.auger.pro@xxxxxxxxx;
> > iommu@xxxxxxxxxxxxxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> > kvm@xxxxxxxxxxxxxxx; kvmarm@xxxxxxxxxxxxxxxxxxxxx; will@xxxxxxxxxx;
> > joro@xxxxxxxxxx; maz@xxxxxxxxxx; robin.murphy@xxxxxxx
> > Cc: jean-philippe@xxxxxxxxxx; Shameerali Kolothum Thodi
> > <shameerali.kolothum.thodi@xxxxxxxxxx>; alex.williamson@xxxxxxxxxx;
> > jacob.jun.pan@xxxxxxxxxxxxxxx; yi.l.liu@xxxxxxxxx; peter.maydell@xxxxxxxxxx;
> > tn@xxxxxxxxxxxx; bbhushan2@xxxxxxxxxxx
> > Subject: Re: [PATCH v11 00/13] SMMUv3 Nested Stage Setup (IOMMU part)
> >
> > Hi Zhangfei,
> >
> > On 4/16/20 6:25 AM, Zhangfei Gao wrote:
> > >
> > >
> > > On 2020/4/14 äå11:05, Eric Auger wrote:
> > >> This version fixes an issue observed by Shameer on an SMMU 3.2,
> > >> when moving from dual stage config to stage 1 only config.
> > >> The 2 high 64b of the STE now get reset. Otherwise, leaving the
> > >> S2TTB set may cause a C_BAD_STE error.
> > >>
> > >> This series can be found at:
> > >> https://github.com/eauger/linux/tree/v5.6-2stage-v11_10.1
> > >> (including the VFIO part)
> > >> The QEMU fellow series still can be found at:
> > >> https://github.com/eauger/qemu/tree/v4.2.0-2stage-rfcv6
> > >>
> > >> Users have expressed interest in that work and tested v9/v10:
> > >> - https://patchwork.kernel.org/cover/11039995/#23012381
> > >> - https://patchwork.kernel.org/cover/11039995/#23197235
> > >>
> > >> Background:
> > >>
> > >> This series brings the IOMMU part of HW nested paging support
> > >> in the SMMUv3. The VFIO part is submitted separately.
> > >>
> > >> The IOMMU API is extended to support 2 new API functionalities:
> > >> 1) pass the guest stage 1 configuration
> > >> 2) pass stage 1 MSI bindings
> > >>
> > >> Then those capabilities gets implemented in the SMMUv3 driver.
> > >>
> > >> The virtualizer passes information through the VFIO user API
> > >> which cascades them to the iommu subsystem. This allows the guest
> > >> to own stage 1 tables and context descriptors (so-called PASID
> > >> table) while the host owns stage 2 tables and main configuration
> > >> structures (STE).
> > >>
> > >>
> > >
> > > Thanks Eric
> > >
> > > Tested v11 on Hisilicon kunpeng920 board via hardware zip accelerator.
> > > 1. no-sva works, where guest app directly use physical address via ioctl.
> > Thank you for the testing. Glad it works for you.
> > > 2. vSVA still not work, same as v10,
> > Yes that's normal this series is not meant to support vSVM at this stage.
> >
> > I intend to add the missing pieces during the next weeks.
> 
> Thanks for that. I have made an attempt to add the vSVA based on
> your v10 + JPBs sva patches. The host kernel and Qemu changes can
> be found here[1][2].
> 
> This basically adds multiple pasid support on top of your changes.
> I have done some basic sanity testing and we have some initial success
> with the zip vf dev on our D06 platform. Please note that the STALL event is
> not yet supported though, but works fine if we mlock() guest usr mem.

I have added STALL support for our vSVA prototype and it seems to be
working(on our hardware). I have updated the kernel and qemu branches with
the same[1][2]. I should warn you though that these are prototype code and I am pretty
much re-using the VFIO_IOMMU_SET_PASID_TABLE interface for almost everything.
But thought of sharing, in case if it is useful somehow!.

Thanks,
Shameer

[1]https://github.com/hisilicon/kernel-dev/commits/vsva-prototype-host-v1

[2]https://github.com/hisilicon/qemu/tree/v4.2.0-2stage-rfcv6-vsva-prototype-v1