Re: [RFC] Memory Tiering

From: David Hildenbrand
Date: Thu Oct 17 2019 - 04:07:11 EST

On 16.10.19 22:05, Dave Hansen wrote:
The memory hierarchy is getting more complicated and the kernel is
playing an increasing role in managing the different tiers. A few
different groups of folks described "migration" optimizations they were
doing in this area at LSF/MM earlier this year. One of the questions
folks asked was why autonuma wasn't being used.

At Intel, the primary new tier that we're looking at is persistent
memory (PMEM). We'd like to be able to use "persistent memory"
*without* using its persistence properties, treating it as slightly
slower DRAM. Keith Busch has some patches to use NUMA migration to
automatically migrate DRAM->PMEM instead of discarding it near the end
of the reclaim process. Huang Ying has some patches which use a
modified autonuma to migrate frequently-used data *back* from PMEM->DRAM.

Very interesting topic. I heard similar demand from HPC folks (especially involving other memory types ("tiers")). There, I think you often want to let the application manage that. But of course, for many applications an automatic management might already be beneficial.

Am I correct that you are using PMEM in this area along with ZONE_DEVICE and not by giving PMEM to the buddy (add_memory())?

We've tried to do this all generically so that it is not tied to
persistent memory and can be applied to any memory types in lots of

We've been running this code in various forms for the past few months,
comparing it to pure DRAM and hardware-based caching. The initial
results are encouraging and we thought others might want to take a look
at the code or run their own experiments. We're expecting to post the
individual patches soon. But, until then, the code is available here:

and is tagged with "tiering-0.2", aka. d8e31e81b1dca9.

Note that internally folks have been calling this "hmem" which is
terribly easy to confuse with the existing hmm. There are still some
"hmem"'s in the tree, but I don't expect them to live much longer.



David / dhildenb