Re: [PATCH] iommu/dma: Setup iova_domain granule for IOMMU_DMA_MSI cookies

[Nate Watterson]

On 5/16/2017 3:55 PM, Auger Eric wrote:
> Hi,
>
> On 13/04/2017 21:38, Nate Watterson wrote:
> > Hi Robin,
> >
> > On 4/13/2017 7:21 AM, Robin Murphy wrote:
> > > Hi Nate,
> > >
> > > On 13/04/17 09:55, Nate Watterson wrote:
> > > > Currently, the __iommu_dma_{map/free} functions call iova_{offset/align}
> > > > making them unsuitable for use with iommu_domains having an
> > > > IOMMU_DMA_MSI
> > > > cookie since the cookie's iova_domain member, iovad, is uninitialized.
> > > >
> > > > Now that iommu_dma_get_msi_page() calls __iommu_dma_map() regardless
> > > > of cookie type, failures are being seen when mapping MSI target
> > > > addresses for devices attached to UNMANAGED domains. To work around
> > > > this issue, the iova_domain granule for IOMMU_DMA_MSI cookies is
> > > > initialized to the value returned by cookie_msi_granule().
> > >
> > > Oh bum. Thanks for the report.
> > >
> > > However, I really don't like bodging around it with deliberate undefined
> > > behaviour. Fixing things properly doesn't seem too hard:
> >
> > I was not especially please with my solution, but I wanted to avoid
> > potentially missing any other spots in the code where granule was
> > used uninitialized. The compile time check made me feel a little
> > less dirty about innappropriately using the iova_domain with MSI
> > cookies.
> >
> > > ----->8-----
> > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> > > index 8348f366ddd1..62618e77bedc 100644
> > > --- a/drivers/iommu/dma-iommu.c
> > > +++ b/drivers/iommu/dma-iommu.c
> > > @@ -396,13 +396,13 @@ static void iommu_dma_free_iova(struct
> > > iommu_dma_cookie *cookie,
> > >                   dma_addr_t iova, size_t size)
> > >    {
> > >           struct iova_domain *iovad = &cookie->iovad;
> > > -       unsigned long shift = iova_shift(iovad);
> > >
> > >           /* The MSI case is only ever cleaning up its most recent
> > > allocation */
> > >           if (cookie->type == IOMMU_DMA_MSI_COOKIE)
> > >                   cookie->msi_iova -= size;
> > >           else
> > > -               free_iova_fast(iovad, iova >> shift, size >> shift);
> > > +               free_iova_fast(iovad, iova_pfn(iovad, iova),
> > > +                               size >> iova_shift(iovad));
> > >    }
> > >
> > >    static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t
> > > dma_addr,
> > > @@ -617,11 +617,14 @@ static dma_addr_t __iommu_dma_map(struct device
> > > *dev, phys_addr_t phys,
> > >    {
> > >           struct iommu_domain *domain = iommu_get_domain_for_dev(dev);
> > >           struct iommu_dma_cookie *cookie = domain->iova_cookie;
> > > -       struct iova_domain *iovad = &cookie->iovad;
> > > -       size_t iova_off = iova_offset(iovad, phys);
> > > +       size_t iova_off = 0;
> > >           dma_addr_t iova;
> > >
> > > -       size = iova_align(iovad, size + iova_off);
> > > +       if (cookie->type == IOMMU_DMA_IOVA_COOKIE) {
> > > +               iova_off = iova_offset(&cookie->iovad, phys);
> > > +               size = iova_align(&cookie->iovad, size + iova_off);
> > > +       }
> > > +
> > >           iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev),
> > > dev);
> > >           if (!iova)
> > >                   return DMA_ERROR_CODE;
> > > -----8<-----
> > >
> > > Untested, and you'll probably want to double-check it anyway given that
> > > the original oversight was mine in the first place ;)
> >
> > This looks good to me. As Shanker has already mentioned, it does fix the
> > faults we were previously seeing with direct device assignment. I also
> > verified that there aren't any other obvious cases of a granule == 0
> > being used in the dma_iommu code by adding BUG_ON(!iovad->granule) to
> > iova_{mask/align/offset/...} and running a variety of tests without
> > issue.
> >
> > Are you going to post the patch?
>
> I also noticed PCIe passthrough/ARM is broken for me with 4.12-r1. I
> tested Robin's patch as well, on Cavium ThunderX, and this fixes the
> faults I have seen.
>
> Has anyone sent a formal patch?

iommu/dma: Don't touch invalid iova_domain members

> Thanks
>
> Eric
>
> > > Robin.
> > >
> > > > Fixes: a44e6657585b ("iommu/dma: Clean up MSI IOVA allocation")
> > > > Signed-off-by: Nate Watterson <nwatters@xxxxxxxxxxxxxx>
> > > > ---
> > > >    drivers/iommu/dma-iommu.c | 10 ++++++++++
> > > >    1 file changed, 10 insertions(+)
> > > >
> > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c
> > > > index 8348f366..d7b0816 100644
> > > > --- a/drivers/iommu/dma-iommu.c
> > > > +++ b/drivers/iommu/dma-iommu.c
> > > > @@ -127,6 +127,16 @@ int iommu_get_msi_cookie(struct iommu_domain
> > > > *domain, dma_addr_t base)
> > > >          cookie->msi_iova = base;
> > > >        domain->iova_cookie = cookie;
> > > > +
> > > > +    /*
> > > > +     * Setup granule for compatibility with __iommu_dma_{alloc/free}
> > > > and
> > > > +     * add a compile time check to ensure that writing granule won't
> > > > +     * clobber msi_iova.
> > > > +     */
> > > > +    cookie->iovad.granule = cookie_msi_granule(cookie);
> > > > +    BUILD_BUG_ON(offsetof(struct iova_domain, granule) <
> > > > +            sizeof(cookie->msi_iova));
> > > > +
> > > >        return 0;
> > > >    }
> > > >    EXPORT_SYMBOL(iommu_get_msi_cookie);

--
Qualcomm Datacenter Technologies as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.