Re: [PATCH v2 8/8] habanalabs: enable 64-bit DMA mask in POWER9

[Oded Gabbay]

On Tue, Jun 11, 2019 at 6:26 PM Greg KH <gregkh@xxxxxxxxxxxxxxxxxxx> wrote:
>
> On Tue, Jun 11, 2019 at 08:17:53AM -0700, Christoph Hellwig wrote:
> > On Tue, Jun 11, 2019 at 11:58:57AM +0200, Greg KH wrote:
> > > That feels like a big hack.  ppc doesn't have any "what arch am I
> > > running on?" runtime call?  Did you ask on the ppc64 mailing list?  I'm
> > > ok to take this for now, but odds are you need a better fix for this
> > > sometime...
> >
> > That isn't the worst part of it.  The whole idea of checking what I'm
> > running to set a dma mask just doesn't make any sense at all.
>
> Oded, I thought I asked if there was a dma call you should be making to
> keep this type of check from being needed.  What happened to that?  As
> Christoph points out, none of this should be needed, which is what I
> thought I originally said :)
>
> thanks,
>
> greg k-h

I'm sorry, but it seems I can't explain what's my problem because you
and Christoph keep mentioning the pci_set_dma_mask() but it doesn't
help me.
I'll try again to explain.

The main problem specifically for Goya device, is that I can't call
this function with *the same parameter* for POWER9 and x86-64, because
x86-64 supports dma mask of 48-bits while POWER9 supports only 32-bits
or 64-bits.

The main limitation in my Goya device is that it can generate PCI
outbound transactions with addresses from 0 to (2^50 - 1).
That's why when we first integrated it in x86-64, we used a DMA mask
of 48-bits, by calling pci_set_dma_mask(pdev, 48). That way, the
kernel ensures me that all the DMA addresses are from 0 to (2^48 - 1),
and that address range is accessible by my device.

If for some reason, the x86-64 machine doesn't support 48-bits, the
standard fallback code in ALL the drivers I have seen is to set the
DMA mask to 32-bits. And that's how my current driver's code is
written.

Now, when I tried to integrate Goya into a POWER9 machine, I got a
reject from the call to pci_set_dma_mask(pdev, 48). The standard code,
as I wrote above, is to call the same function with 32-bits. That
works BUT it is not practical, as our applications require much more
memory mapped then 32-bits. In addition, once you add more cards which
are all mapped to the same range, it is simply not usable at all.

Therefore, I consulted with POWER people and they told me I can call
to pci_set_dma_mask with the mask as 64, but I must make sure that ALL
outbound transactions from Goya will be with bit 59 set in the
address.
I can achieve that with a dedicated configuration I make in Goya's
PCIe controller. That's what I did and that works.

So, to summarize:
If I call pci_set_dma_mask with 48, then it fails on POWER9. However,
in runtime, I don't know if its POWER9 or not, so upon failure I will
call it again with 32, which makes our device pretty much unusable.
If I call pci_set_dma_mask with 64, and do the dedicated configuration
in Goya's PCIe controller, then it won't work on x86-64, because bit
59 will be set and the host won't like it (I checked it). In addition,
I might get addresses above 50 bits, which my device can't generate.

I hope this makes things more clear. Now, please explain to me how I
can call pci_set_dma_mask without any regard to whether I run on
x86-64 or POWER9, considering what I wrote above ?

Thanks,
Oded