HMM (Heterogeneous Memory Management)

[JÃrÃme Glisse]

Last time I spoke with Linus and Andrew, the requirement for getting
HMM upstream was having real hardware working with it beside Mellanox
(as Mellanox does not use all HMM features), both with closed source
driver and open source driver. Work on open source driver is underway,
and I anticipate we will get update from NVIDIA and other parties on
their efforts and plans shortly.

I am re-posting now because I want people to have time to look at HMM
again. The open source driver will stay behind close doors until
hardware is released. I can however have the upstream maintainer share
his progress here if anyone feels the need for that.

Other parties such as IBM and Mediatek are also interested in HMM. I
expect they will comment on their respective hardware when they can.

I hope that HMM can be considered for inclusion upstream soon.

This version is virtualy the same as the one since last post (modulo
rebase differences). Tree with the patchset:

git://people.freedesktop.org/~glisse/linux hmm-v12 branch

HMM (HMM (Heterogeneous Memory Management) is an helper layer
for device driver, its main features are :
   - Shadow CPU page table of a process into a device specific
     format page table and keep both page table synchronize.
   - Handle DMA mapping of system ram page on behalf of device
     (for shadowed page table entry).
   - Migrate private anonymous memory to private device memory
     and handle CPU page fault (which triggers a migration back
     to system memory so CPU can access it).

Benefits of HMM :
   - Avoid current model where device driver have to pin page
     which blocks several kernel features (KSM, migration, ...).
   - No impact on existing workload that do not use HMM (it only
     adds couple more if() to common code path).
   - Intended as common infrastructure for various hardware.
   - Allow userspace API to move away from explicit copy code
     path where application programmer has to manage manually
     memcpy to and from device memory.
   - Transparent to userspace, for instance allowing library to
     use GPU without involving application linked against it.

Change log :

v12:
  - Rebase

v11:
  - Fix PROT_NONE case
  - Fix missing page table walk callback
  - Add support for hugetlbfs

v10:
  - Minor fixes here and there.

v9:
  - Added new device driver helpers.
  - Added documentions.
  - Improved page table code claritity (minor architectural changes
    and better names).

v8:
  - Removed currently unuse fence code.
  - Added DMA mapping on behalf of device.

v7:
  - Redone and simplified page table code to match Linus suggestion
    http://article.gmane.org/gmane.linux.kernel.mm/125257

... Lost in translation ...


Why doing this ?

Mirroring a process address space is mandatory with OpenCL 2.0 and
with other GPU compute APIs. OpenCL 2.0 allows different level of
implementation and currently only the lowest 2 are supported on
Linux. To implement the highest level, where CPU and GPU access
can happen concurently and are cache coherent, HMM is needed, or
something providing same functionality, for instance through
platform hardware.

Hardware solution such as PCIE ATS/PASID is limited to mirroring
system memory and does not provide way to migrate memory to device
memory (which offer significantly more bandwidth, up to 10 times
faster than regular system memory with discrete GPU, also have
lower latency than PCIE transaction).

Current CPU with GPU on same die (AMD or Intel) use the ATS/PASID
and for Intel a special level of cache (backed by a large pool of
fast memory).

For foreseeable future, discrete GPUs will remain releveant as they
can have a large quantity of faster memory than integrated GPU.

Thus we believe HMM will allow us to leverage discrete GPUs memory
in a transparent fashion to the application, with minimum disruption
to the linux kernel mm code. Also HMM can work along hardware
solution such as PCIE ATS/PASID (leaving regular case to ATS/PASID
while HMM handles the migrated memory case).


Design :

The patch 1, 2, 3 and 4 augment the mmu notifier API with new
informations to more efficiently mirror CPU page table updates.

The first side of HMM, process address space mirroring, is
implemented in patch 5 through 14. This use a secondary page
table, in which HMM mirror memory actively use by the device.
HMM does not take a reference on any of the page, it use the
mmu notifier API to track changes to the CPU page table and to
update the mirror page table. All this while providing a simple
API to device driver.

To implement this we use a "generic" page table and not a radix
tree because we need to store more flags than radix tree allows
and we need to store dma address (sizeof(dma_addr_t) > sizeof(long)
on some platform).


(1) Previous patchset posting :
    v1 http://lwn.net/Articles/597289/
    v2 https://lkml.org/lkml/2014/6/12/559
    v3 https://lkml.org/lkml/2014/6/13/633
    v4 https://lkml.org/lkml/2014/8/29/423
    v5 https://lkml.org/lkml/2014/11/3/759
    v6 http://lwn.net/Articles/619737/
    v7 http://lwn.net/Articles/627316/
    v8 https://lwn.net/Articles/645515/
    v9 https://lwn.net/Articles/651553/
    v10 https://lwn.net/Articles/654430/
    v11 https://lkml.org/lkml/2015/10/21/739

Cheers,
JÃrÃme

To: "Andrew Morton" <akpm@xxxxxxxxxxxxxxxxxxxx>,
To: <linux-kernel@xxxxxxxxxxxxxxx>,
To: linux-mm <linux-mm@xxxxxxxxx>,
Cc: "Linus Torvalds" <torvalds@xxxxxxxxxxxxxxxxxxxx>,
Cc: "Mel Gorman" <mgorman@xxxxxxx>,
Cc: "H. Peter Anvin" <hpa@xxxxxxxxx>,
Cc: "Peter Zijlstra" <peterz@xxxxxxxxxxxxx>,
Cc: "Linda Wang" <lwang@xxxxxxxxxx>,
Cc: "Kevin E Martin" <kem@xxxxxxxxxx>,
Cc: "Andrea Arcangeli" <aarcange@xxxxxxxxxx>,
Cc: "Johannes Weiner" <jweiner@xxxxxxxxxx>,
Cc: "Larry Woodman" <lwoodman@xxxxxxxxxx>,
Cc: "Rik van Riel" <riel@xxxxxxxxxx>,
Cc: "Dave Airlie" <airlied@xxxxxxxxxx>,
Cc: "Jeff Law" <law@xxxxxxxxxx>,
Cc: "Brendan Conoboy" <blc@xxxxxxxxxx>,
Cc: "Joe Donohue" <jdonohue@xxxxxxxxxx>,
Cc: "Christophe Harle" <charle@xxxxxxxxxx>,
Cc: "Duncan Poole" <dpoole@xxxxxxxxxx>,
Cc: "Sherry Cheung" <SCheung@xxxxxxxxxx>,
Cc: "Subhash Gutti" <sgutti@xxxxxxxxxx>,
Cc: "John Hubbard" <jhubbard@xxxxxxxxxx>,
Cc: "Mark Hairgrove" <mhairgrove@xxxxxxxxxx>,
Cc: "Lucien Dunning" <ldunning@xxxxxxxxxx>,
Cc: "Cameron Buschardt" <cabuschardt@xxxxxxxxxx>,
Cc: "Arvind Gopalakrishnan" <arvindg@xxxxxxxxxx>,
Cc: "Haggai Eran" <haggaie@xxxxxxxxxxxx>,
Cc: "Or Gerlitz" <ogerlitz@xxxxxxxxxxxx>,
Cc: "Sagi Grimberg" <sagig@xxxxxxxxxxxx>
Cc: "Shachar Raindel" <raindel@xxxxxxxxxxxx>,
Cc: "Liran Liss" <liranl@xxxxxxxxxxxx>,
Cc: "Roland Dreier" <roland@xxxxxxxxxxxxxxx>,
Cc: "Sander, Ben" <ben.sander@xxxxxxx>,
Cc: "Stoner, Greg" <Greg.Stoner@xxxxxxx>,
Cc: "Bridgman, John" <John.Bridgman@xxxxxxx>,
Cc: "Mantor, Michael" <Michael.Mantor@xxxxxxx>,
Cc: "Blinzer, Paul" <Paul.Blinzer@xxxxxxx>,
Cc: "Morichetti, Laurent" <Laurent.Morichetti@xxxxxxx>,
Cc: "Deucher, Alexander" <Alexander.Deucher@xxxxxxx>,
Cc: "Leonid Shamis" <Leonid.Shamis@xxxxxxx>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@xxxxxxxxxxxxxxxxxx>