Re: Interacting with coherent memory on external devices

From: Vlastimil Babka
Date: Wed May 13 2015 - 10:11:05 EST

Sorry for reviving oldish thread...

On 04/28/2015 01:54 AM, Benjamin Herrenschmidt wrote:
On Mon, 2015-04-27 at 11:48 -0500, Christoph Lameter wrote:
On Mon, 27 Apr 2015, Rik van Riel wrote:

Why would we want to avoid the sane approach that makes this thing
work with the fewest required changes to core code?

Becaus new ZONEs are a pretty invasive change to the memory management and
because there are other ways to handle references to device specific

ZONEs is just one option we put on the table.

I think we can mostly agree on the fundamentals that a good model of
such a co-processor is a NUMA node, possibly with a higher distance
than other nodes (but even that can be debated).

That gives us a lot of the basics we need such as struct page, ability
to use existing migration infrastructure, and is actually a reasonably
representation at high level as well.

The question is how do we additionally get the random stuff we don't
care about out of the way. The large distance will not help that much
under memory pressure for example.

Covering the entire device memory with a CMA goes a long way toward that
goal. It will avoid your ordinary kernel allocations.

I think ZONE_MOVABLE should be sufficient for this. CMA is basically for marking parts of zones as MOVABLE-only. You shouldn't need that for the whole zone. Although it might happen that CMA will be a special zone one day.

It also provides just what we need to be able to do large contiguous
"explicit" allocations for use by workloads that don't want the
transparent migration and by the driver for the device which might also
need such special allocations for its own internal management data

Plain zone compaction + reclaim should work as well in a ZONE_MOVABLE zone. CMA allocations might IIRC additionally migrate across zones, e.g. from the device to system memory (unlike plain compaction), which might be what you want, or not.

We still have the risk of pages in the CMA being pinned by something
like gup however, that's where the ZONE idea comes in, to ensure the
various kernel allocators will *never* allocate in that zone unless
explicitly specified, but that could possibly implemented differently.

Kernel allocations should ignore the ZONE_MOVABLE zone as they are not typically movable. Then it depends on how much control you want for userspace allocations.

Maybe a concept of "exclusive" NUMA node, where allocations never
fallback to that node unless explicitly asked to go there.

I guess that could be doable on the zonelist level, where the device memory node/zone wouldn't be part of the "normal" zonelists, so memory pressure calculations should be also fine. But sure there will be some corner cases :)

Of course that would have an impact on memory pressure calculations,
nothign comes completely for free, but at this stage, this is the goal
of this thread, ie, to swap ideas around and see what's most likely to
work in the long run before we even start implementing something.


To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx For more info on Linux MM,
see: .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at