Re: [LSF/MM TOPIC] Page Cache Flexibility for NVM

[Jerome Glisse]

On Thu, Feb 21, 2019 at 11:11:51PM +0000, Adam Manzanares wrote:
> Hello,
> 
> I would like to attend the LSF/MM Summit 2019. I'm interested in
> several MM topics that are mentioned below as well as Zoned Block
> Devices and any io determinism topics that come up in the storage
> track. 
> 
> I have been working on a caching layer, hmmap (heterogeneous memory
> map) [1], for emerging NVM and it is in spirit close to the page
> cache. The key difference being that the backend device and caching
> layer of hmmap is pluggable. In addition, hmmap supports DAX and write
> protection, which I believe are key features for emerging NVMs that may
> have write/read asymmetry as well as write endurance constraints.
> Lastly we can leverage hardware, such as a DMA engine, when moving
> pages between the cache while also allowing direct access if the device
> is capable.
> 
> I am proposing that as an alternative to using NVMs as a NUMA node
> we expose the NVM through the page cache or a viable alternative and
> have userspace applications mmap the NVM and hand out memory with
> their favorite userspace memory allocator.
> 
> This would isolate the NVMs to only applications that are well aware
> of the performance implications of accessing NVM. I believe that all
> of this work could be solved with the NUMA node approach, but the two
> approaches are seeming to blur together.
> 
> The main points I would like to discuss are:
> 
> * Is the page cache model a viable alternative to NVM as a NUMA NODE?
> * Can we add more flexibility to the page cache?
> * Should we force separation of NVM through an explicit mmap?
> 
> I believe this discussion could be merged with NUMA, memory hierarchy
> and device memory, Use NVDIMM as NUMA node and NUMA API, or memory
> reclaim with NUMA balancing.

What about cache coherency and atomic ? If device block are expose
through PCIE then there is no cache coherency or atomic and thus
direct mmap will not have the expected memory model which would
break program expectation of a mmap.

This is also one of the reasons i do not see a way forward with NUMA
and device memory. It can depart from the usual memory too much to
be drop in like that to unaware application.

In any case yes this kind of memory falls into the device memory i
wish to discuss during LSF/MM.

Cheers,
Jérôme