RE: [RFC PATCH 0/4] Use 1st-level for DMA remapping in guest

[Tian, Kevin]

> From: Peter Xu [mailto:peterx@xxxxxxxxxx]
> Sent: Wednesday, September 25, 2019 3:45 PM
> 
> On Wed, Sep 25, 2019 at 07:21:51AM +0000, Tian, Kevin wrote:
> > > From: Peter Xu [mailto:peterx@xxxxxxxxxx]
> > > Sent: Wednesday, September 25, 2019 2:57 PM
> > >
> > > On Wed, Sep 25, 2019 at 10:48:32AM +0800, Lu Baolu wrote:
> > > > Hi Kevin,
> > > >
> > > > On 9/24/19 3:00 PM, Tian, Kevin wrote:
> > > > > > > >       '-----------'
> > > > > > > >       '-----------'
> > > > > > > >
> > > > > > > > This patch series only aims to achieve the first goal, a.k.a using
> > > > > first goal? then what are other goals? I didn't spot such information.
> > > > >
> > > >
> > > > The overall goal is to use IOMMU nested mode to avoid shadow page
> > > table
> > > > and VMEXIT when map an gIOVA. This includes below 4 steps (maybe
> not
> > > > accurate, but you could get the point.)
> > > >
> > > > 1) GIOVA mappings over 1st-level page table;
> > > > 2) binding vIOMMU 1st level page table to the pIOMMU;
> > > > 3) using pIOMMU second level for GPA->HPA translation;
> > > > 4) enable nested (a.k.a. dual stage) translation in host.
> > > >
> > > > This patch set aims to achieve 1).
> > >
> > > Would it make sense to use 1st level even for bare-metal to replace
> > > the 2nd level?
> > >
> > > What I'm thinking is the DPDK apps - they have MMU page table already
> > > there for the huge pages, then if they can use 1st level as the
> > > default device page table then it even does not need to map, because
> > > it can simply bind the process root page table pointer to the 1st
> > > level page root pointer of the device contexts that it uses.
> > >
> >
> > Then you need bear with possible page faults from using CPU page
> > table, while most devices don't support it today.
> 
> Right, I was just thinking aloud.  After all neither do we have IOMMU
> hardware to support 1st level (or am I wrong?)...  It's just that when

You are right. Current VT-d supports only 2nd level.

> the 1st level is ready it should sound doable because IIUC PRI should
> be always with the 1st level support no matter on IOMMU side or the
> device side?

No. PRI is not tied to 1st or 2nd level. Actually from device p.o.v, it's
just a protocol to trigger page fault, but the device doesn't care whether
the page fault is on 1st or 2nd level in the IOMMU side. The only
relevant part is that a PRI request can have PASID tagged or cleared.
When it's tagged with PASID, the IOMMU will locate the translation
table under the given PASID (either 1st or 2nd level is fine, according
to PASID entry setting). When no PASID is included, the IOMMU locates
the translation from default entry (e.g. PASID#0 or any PASID contained
in RID2PASID in VT-d).

Your knowledge happened to be correct in deprecated ECS mode. At
that time, there is only one 2nd level per context entry which doesn't
support page fault, and there is only one 1st level per PASID entry which
supports page fault. Then PRI could be indirectly connected to 1st level,
but this just changed with new scalable mode.

Another note is that the PRI capability only indicates that a device is
capable of handling page faults, but not that a device can tolerate
page fault for any of its DMA access. If the latter is fasle, using CPU 
page table for DPDK usage is still risky (and specific to device behavior)

> 
> I'm actually not sure about whether my understanding here is
> correct... I thought the pasid binding previously was only for some
> vendor kernel drivers but not a general thing to userspace.  I feel
> like that should be doable in the future once we've got some new
> syscall interface ready to deliver 1st level page table (e.g., via
> vfio?) then applications like DPDK seems to be able to use that too
> even directly via bare metal.
> 

using 1st level for userspace is different from supporting DMA page
fault in userspace. The former is purely about which structure to
keep the mapping. I think we may do the same thing for both bare
metal and guest (using 2nd level only for GPA when nested is enabled
on the IOMMU). But reusing CPU page table for userspace is more
tricky. :-)

Thanks
Kevin