Re: [PATCH 0/6] Cache coherent device memory (CDM) with HMM v5

[chet l]

>> You may think it as a CCIX device or CAPI device.
>> The requirement is eliminate any extra copy.
>> A typical usecase/requirement is malloc() and madvise() allocate from
>> device memory, then CPU write data to device memory directly and
>> trigger device to read the data/do calculation.
>
> I suggest you rely on the device driver userspace API to do a migration after malloc
> then. Something like:
>   ptr = malloc(size);
>   my_device_migrate(ptr, size);
>
> Which would call an ioctl of the device driver which itself would migrate memory or
> allocate device memory for the range if pointer return by malloc is not yet back by
> any pages.
>

So for CCIX, I don't think there is going to be an inline device
driver that would allocate any memory for you. The expansion memory
will become part of the system memory as part of the boot process. So,
if the host DDR is 256GB and the CCIX expansion memory is 4GB, the
total system mem will be 260GB.

Assume that the 'mm' is taught to mark/anoint the ZONE_DEVICE(or
ZONE_XXX) range from 256 to 260 GB. Then, for kmalloc it(mm) won't use
the ZONE_DEV range. But for a malloc, it will/can use that range.


> There has been several discussions already about madvise/mbind/set_mempolicy/
> move_pages and at this time i don't think we want to add or change any of them to
> understand device memory. My personal opinion is that we first need to have enough

We will visit these APIs when we are more closer to building exotic
CCIX devices. And the plan is to present/express the CCIX proximity
attributes just like a NUMA node-proximity attribute today. That way
there would be minimal disruptions to the existing OS ecosystem.



Chetan