Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

[Jerome Glisse]

On Wed, Nov 15, 2017 at 07:29:10PM -0800, chetan L wrote:
> On Wed, Nov 15, 2017 at 7:23 PM, chetan L <loke.chetan@xxxxxxxxx> wrote:
> > On Wed, Nov 15, 2017 at 6:44 PM, Jerome Glisse <jglisse@xxxxxxxxxx> wrote:
> >> On Wed, Nov 15, 2017 at 06:10:08PM -0800, chet l wrote:
> >>> >> You may think it as a CCIX device or CAPI device.
> >>> >> The requirement is eliminate any extra copy.
> >>> >> A typical usecase/requirement is malloc() and madvise() allocate from
> >>> >> device memory, then CPU write data to device memory directly and
> >>> >> trigger device to read the data/do calculation.
> >>> >
> >>> > I suggest you rely on the device driver userspace API to do a migration after malloc
> >>> > then. Something like:
> >>> >   ptr = malloc(size);
> >>> >   my_device_migrate(ptr, size);
> >>> >
> >>> > Which would call an ioctl of the device driver which itself would migrate memory or
> >>> > allocate device memory for the range if pointer return by malloc is not yet back by
> >>> > any pages.
> >>> >
> >>>
> >>> So for CCIX, I don't think there is going to be an inline device
> >>> driver that would allocate any memory for you. The expansion memory
> >>> will become part of the system memory as part of the boot process. So,
> >>> if the host DDR is 256GB and the CCIX expansion memory is 4GB, the
> >>> total system mem will be 260GB.
> >>>
> >>> Assume that the 'mm' is taught to mark/anoint the ZONE_DEVICE(or
> >>> ZONE_XXX) range from 256 to 260 GB. Then, for kmalloc it(mm) won't use
> >>> the ZONE_DEV range. But for a malloc, it will/can use that range.
> >>
> >> HMM zone device memory would work with that, you just need to teach the
> >> platform to identify this memory zone and not hotplug it. Again you
> >> should rely on specific device driver API to allocate this memory.
> >>
> >
> > @Jerome - a new linux-accelerator's list has just been created. I have
> > CC'd that list since we have overlapping interests w.r.t CCIX.
> >
> > I cannot comment on surprise add/remove as of now ... will cross the
> > bridge later.

Note that this is not hotplug strictly speaking. Design today is that it
is the device driver that register the memory. From kernel point of view
this is an hotplug but for many of the target architecture there is no
real hotplug ie device and its memory was present at boot time.

Like i said i think for now we are better of having each device manage and
register its memory. HMM provide a toolbox for that. If we see common trend
accross multiple devices then we can think about making something more
generic.


For the NUMA discussion this is related to CPU less node ie not wanting
to add any more CPU less node (node with only memory) and they are other
aspect too. For instance you do not necessarily have good informations
from the device to know if a page is access a lot by the device (this
kind of information is often only accessible by the device driver). Thus
the automatic NUMA placement is useless here. Not mentioning that for it
to work we would need to change how it currently work (iirc there is
issue when you not have a CPU id you can use).

Cheers,
Jérôme