Re: [PATCH V3 0/4] Define coherent device memory node

[Anshuman Khandual]

On 02/23/2017 12:22 PM, Anshuman Khandual wrote:
> On 02/22/2017 03:20 PM, Michal Hocko wrote:
>> On Tue 21-02-17 19:09:18, Anshuman Khandual wrote:
>>> On 02/21/2017 04:41 PM, Michal Hocko wrote:
>>>> On Fri 17-02-17 17:11:57, Anshuman Khandual wrote:
>>>> [...]
>>>>> * User space using mbind() to get CDM memory is an additional benefit
>>>>>   we get by making the CDM plug in as a node and be part of the buddy
>>>>>   allocator. But the over all idea from the user space point of view
>>>>>   is that the application can allocate any generic buffer and try to
>>>>>   use the buffer either from the CPU side or from the device without
>>>>>   knowing about where the buffer is really mapped physically. That
>>>>>   gives a seamless and transparent view to the user space where CPU
>>>>>   compute and possible device based compute can work together. This
>>>>>   is not possible through a driver allocated buffer.
>>>>
>>>> But how are you going to define any policy around that. Who is allowed
>>>
>>> The user space VMA can define the policy with a mbind(MPOL_BIND) call
>>> with CDM/CDMs in the nodemask.
>>>
>>>> to allocate and how much of this "special memory". Is it possible that
>>>
>>> Any user space application with mbind(MPOL_BIND) call with CDM/CDMs in
>>> the nodemask can allocate from the CDM memory. "How much" gets controlled
>>> by how we fault from CPU and the default behavior of the buddy allocator.
>>
>> In other words the policy is implemented by the kernel. Why is this a
>> good thing?
> 
> Its controlled by the kernel only during page fault paths of either CPU
> or device. But the device driver will actually do the placements after
> wards after taking into consideration access patterns and relative
> performance. We dont want the driver to be involved during page fault
> path memory allocations which should naturally go through the buddy
> allocator.
> 
>>
>>>> we will eventually need some access control mechanism? If yes then mbind
>>>
>>> No access control mechanism is needed. If an application wants to use
>>> CDM memory by specifying in the mbind() it can. Nothing prevents it
>>> from using the CDM memory.
>>
>> What if we find out that an access control _is_ really needed? I can
>> easily imagine that some devices will come up with really fast and expensive
>> memory. You do not want some random user to steal it from you when you
>> want to use it for your workload.
> 
> Hmm, it makes sense but I think its not something we have to deal with
> right away. Later we may have to think about some generic access control
> mechanism for mbind() and then accommodate CDM with it.
> 
>>
>>>> is really not suitable interface to (ab)use. Also what should happen if
>>>> the mbind mentions only CDM memory and that is depleted?
>>>
>>> IIUC *only CDM* cannot be requested from user space as there are no user
>>> visible interface which can translate to __GFP_THISNODE.
>>
>> I do not understand what __GFP_THISNODE has to do with this. This is an
>> internal flag.
> 
> Right. My bad. I was just referring to the fact that there is nothing in
> user space which can make buddy allocator pick NOFALLBACK list instead of
> FALLBACK list.
> 
>>
>>> MPOL_BIND with
>>> CDM in the nodemask will eventually pick a FALLBACK zonelist which will
>>> have zones of the system including CDM ones. If the resultant CDM zones
>>> run out of memory, we fail the allocation request as usual.
>>
>> OK, so let's say you mbind to a single node which is CDM. You seem to be
>> saying that we will simply break the NUMA affinity in this special case?
> 
> Why ? It should simply follow what happens when we pick a single NUMA node
> in previous situations.
> 
>> Currently we invoke the OOM killer if nodes which the application binds
>> to are depleted and cannot be reclaimed.
> 
> Right, the same should happen here for CDM as well.
> 
>>  
>>>> Could you also explain why the transparent view is really better than
>>>> using a device specific mmap (aka CDM awareness)?
>>>
>>> Okay with a transparent view, we can achieve a control flow of application
>>> like the following.
>>>
>>> (1) Allocate a buffer:		alloc_buffer(buf, size)
>>> (2) CPU compute on buffer:	cpu_compute(buf, size)
>>> (3) Device compute on buffer:	device_compute(buf, size)
>>> (4) CPU compute on buffer:	cpu_compute(buf, size)
>>> (5) Release the buffer:		release_buffer(buf, size)
>>>
>>> With assistance from a device specific driver, the actual page mapping of
>>> the buffer can change between system RAM and device memory depending on
>>> which side is accessing at a given point. This will be achieved through
>>> driver initiated migrations.
>>
>> But then you do not need any NUMA affinity, right? The driver can do
>> all this automagically. How does the numa policy comes into the game in
>> your above example. Sorry for being dense, I might be really missing
>> something important here, but I really fail to see why the NUMA is the
>> proper interface here.
> 
> You are right. Driver can migrate any mapping in the userspace to any
> where on the system as long as cpuset does not prohibit it. But we still
> want the driver to conform to the applicable VMA memory policy set from
> the userspace. Hence a VMA policy needs to be set from the user space.
> NUMA VMA memory policy also restricts the allocations inside the
> applicable nodemask during page fault paths (CPU and device) as well.

Hello Michal,

Does that answer your question ?