[PATCH 0/8] Fixing DMA mask issues in habanalabs driver

[Oded Gabbay]

This patch-set changes the way the Goya internal CPU access memory on the
Host machine. This is needed to prevent the non-standard way the driver
used the PCI DMA set mask kernel API so far.

The DMA set mask should be called only once at the start of the driver.
This is because changing the DMA mask to a new value after allocations
were made using a previous mask value, might cause the previous allocations
to become unaccessible (usually if there is IOMMU present).

The driver did that because of a limitation in Goya's internal CPU. The
limitation was that the internal CPU can only access 40-bit addresses,
while the entire ASIC can access 50-bit addresses. Therefore, the driver
set the DMA mask to 39-bits, allocated memory for the internal CPU on the
host and then changed the DMA mask to 48-bits.

This patch-set eliminates the double DMA set by using Goya's MMU to
overcome the limitation. The driver now sets the DMA mask only once to
48-bits and allocates a single DMA region of 2MB for the internal CPU. It
then maps that region in Goya's MMU to a device virtual address under 40-bits.

In addition, this patch-set enables the use of 64-bit mask on POWER9
systems. POWER9 DMA mask can be set ONLY to 32-bit or 64-bit. To use
64-bit, the device must set bit 59 to 1 in all its outbound transactions. 
This is achieved by setting a special configuration in Goya's PCIe
controller. The configuration must be done only in POWER9 machines, as it
will make the device non-functional on other architectures 
(e.g. x86-64, ARM).

Thanks,
Oded

Oded Gabbay (8):
  habanalabs: initialize device CPU queues after MMU init
  habanalabs: de-couple MMU and VM module initialization
  habanalabs: initialize MMU context for driver
  habanalabs: add MMU mappings for Goya CPU
  habanalabs: set Goya CPU to use ASIC MMU
  habanalabs: remove DMA mask hack for Goya
  habanalabs: add WARN in case of bad MMU mapping
  habanalabs: enable 64-bit DMA mask in POWER9

 drivers/misc/habanalabs/asid.c           |   2 +-
 drivers/misc/habanalabs/context.c        |   7 +
 drivers/misc/habanalabs/debugfs.c        |   7 +-
 drivers/misc/habanalabs/device.c         |  45 +++--
 drivers/misc/habanalabs/goya/goya.c      | 234 +++++++++++++++++------
 drivers/misc/habanalabs/goya/goyaP.h     |  12 +-
 drivers/misc/habanalabs/habanalabs.h     |   9 +-
 drivers/misc/habanalabs/habanalabs_drv.c |   7 +
 drivers/misc/habanalabs/memory.c         |  13 +-
 drivers/misc/habanalabs/mmu.c            |  20 +-
 drivers/misc/habanalabs/pci.c            |   7 +-
 11 files changed, 259 insertions(+), 104 deletions(-)

-- 
2.17.1