Re: [HMM 03/15] mm/unaddressable-memory: new type of ZONE_DEVICE for unaddressable memory

[John Hubbard]

On 4/23/17 6:13 AM, Dan Williams wrote:
> On Sat, Apr 22, 2017 at 11:11 AM, Jerome Glisse <jglisse@xxxxxxxxxx> wrote:
> > On Fri, Apr 21, 2017 at 10:30:01PM -0700, Dan Williams wrote:
> > > On Fri, Apr 21, 2017 at 8:30 PM, JÃrÃme Glisse <jglisse@xxxxxxxxxx> wrote:
> >
> > [...]
> >
> > > > +/*
> > > > + * Specialize ZONE_DEVICE memory into multiple types each having differents
> > > > + * usage.
> > > > + *
> > > > + * MEMORY_DEVICE_PERSISTENT:
> > > > + * Persistent device memory (pmem): struct page might be allocated in different
> > > > + * memory and architecture might want to perform special actions. It is similar
> > > > + * to regular memory, in that the CPU can access it transparently. However,
> > > > + * it is likely to have different bandwidth and latency than regular memory.
> > > > + * See Documentation/nvdimm/nvdimm.txt for more information.
> > > > + *
> > > > + * MEMORY_DEVICE_UNADDRESSABLE:
> > > > + * Device memory that is not directly addressable by the CPU: CPU can neither
> > > > + * read nor write _UNADDRESSABLE memory. In this case, we do still have struct
> > > > + * pages backing the device memory. Doing so simplifies the implementation, but
> > > > + * it is important to remember that there are certain points at which the struct
> > > > + * page must be treated as an opaque object, rather than a "normal" struct page.
> > > > + * A more complete discussion of unaddressable memory may be found in
> > > > + * include/linux/hmm.h and Documentation/vm/hmm.txt.
> > > > + */
> > > > +enum memory_type {
> > > > +       MEMORY_DEVICE_PERSISTENT = 0,
> > > > +       MEMORY_DEVICE_UNADDRESSABLE,
> > > > +};
> > >
> > > Ok, this is a bikeshed, but I think it is important. I think these
> > > should be called MEMORY_DEVICE_PUBLIC and MEMORY_DEVICE_PRIVATE. The
> > > reason is that persistence has nothing to do with the code paths that
> > > deal with the pmem use case of ZONE_DEVICE. The only property the mm
> > > cares about is that the address range behaves the same as host memory
> > > for dma and cpu accesses. The "unaddressable" designation always
> > > confuses me because a memory range isn't memory if it's
> > > "unaddressable". It is addressable, it's just "private" to the device.
> >
> > I can change the name but the memory is truely unaddressable, the CPU
> > can not access it whatsoever (well it can access a small window but
> > even that is not guaranteed).
>
> Understood, but that's still "addressable only by certain agents or
> through a proxy" which seems closer to "private" to me.

Actually, MEMORY_DEVICE_PRIVATE / _PUBLIC seems like a good choice to 
me, because the memory may not remain CPU-unaddressable in the future. 
By that, I mean that I know of at least one company (ours) that is 
working on products that will support hardware-based memory coherence 
(and access counters to go along with that). If someone were to enable 
HMM on such a system, then the device memory would be, in fact, directly 
addressable by a CPU--thus exactly contradicting the "unaddressable" name.

Yes, it is true that we would have to modify HMM anyway, in order to 
work in that situation, partly because HMM today relies on CPU and 
device page faults in order to work. And it's also true that we might 
want to take a different approach than HMM, to support that kind of 
device: for example, making it a NUMA node has been debated here, recently.

But even so, I think the potential for the "unaddressable" memory 
actually becoming "addressable" someday, is a good argument for using a 
different name.

thanks,

--
John Hubbard
NVIDIA