Re: [LSF/MM TOPIC] Page Cache Flexibility for NVM

[Adam Manzanares]

On Thu, 2019-02-21 at 19:27 -0500, Jerome Glisse wrote:
> On Thu, Feb 21, 2019 at 11:11:51PM +0000, Adam Manzanares wrote:
> > Hello,
> > 
> > I would like to attend the LSF/MM Summit 2019. I'm interested in
> > several MM topics that are mentioned below as well as Zoned Block
> > Devices and any io determinism topics that come up in the storage
> > track. 
> > 
> > I have been working on a caching layer, hmmap (heterogeneous memory
> > map) [1], for emerging NVM and it is in spirit close to the page
> > cache. The key difference being that the backend device and caching
> > layer of hmmap is pluggable. In addition, hmmap supports DAX and
> > write
> > protection, which I believe are key features for emerging NVMs that
> > may
> > have write/read asymmetry as well as write endurance constraints.
> > Lastly we can leverage hardware, such as a DMA engine, when moving
> > pages between the cache while also allowing direct access if the
> > device
> > is capable.
> > 
> > I am proposing that as an alternative to using NVMs as a NUMA node
> > we expose the NVM through the page cache or a viable alternative
> > and
> > have userspace applications mmap the NVM and hand out memory with
> > their favorite userspace memory allocator.
> > 
> > This would isolate the NVMs to only applications that are well
> > aware
> > of the performance implications of accessing NVM. I believe that
> > all
> > of this work could be solved with the NUMA node approach, but the
> > two
> > approaches are seeming to blur together.
> > 
> > The main points I would like to discuss are:
> > 
> > * Is the page cache model a viable alternative to NVM as a NUMA
> > NODE?
> > * Can we add more flexibility to the page cache?
> > * Should we force separation of NVM through an explicit mmap?
> > 
> > I believe this discussion could be merged with NUMA, memory
> > hierarchy
> > and device memory, Use NVDIMM as NUMA node and NUMA API, or memory
> > reclaim with NUMA balancing.
> 
> What about cache coherency and atomic ? If device block are expose
> through PCIE then there is no cache coherency or atomic and thus
> direct mmap will not have the expected memory model which would
> break program expectation of a mmap.

For the PCIE cache coherency case I would envision that you would map
the memory as read only into the process address space. Once a write
occurs I would then remap the PCIE memory to a page in the proposed
caching mechanism.

I have to think more about what this means for atomic operations.

> 
> This is also one of the reasons i do not see a way forward with NUMA
> and device memory. It can depart from the usual memory too much to
> be drop in like that to unaware application.

I have similar concerns and am trying to segregate the device memory to
aware applications.

> 
> In any case yes this kind of memory falls into the device memory i
> wish to discuss during LSF/MM.
> 
> Cheers,
> JÃrÃme
>