Re: "Directly mapped persistent memory page cache"

From: Ingo Molnar
Date: Sun May 10 2015 - 06:07:41 EST

* Eric W. Biederman <ebiederm@xxxxxxxxxxxx> wrote:

> > What do you think?
> The tricky bit is what happens when you reboot and run a different
> version of the kernel, especially a kernel with things debugging
> features like kmemcheck that increase the size of struct page.

Yes - but I think that's relatively easy to handle, as most 'weird'
page struct usages can be cordoned off:

I.e. we could define a 64-bit "core" struct page, denote it with a
single PG_ flag and stick with it: the only ABI is its size
essentially, as we (lazily) re-initialize it after every bootup.

The 'extended' (often debug) part of a struct page, such as
page->shadow on kmemcheck, can simply be handled in a special way
based on the PG_ flag:

- for example in the kmemcheck case no page->shadow means no leak
tracking: that's perfectly fine as these pages aren't part of the
buddy allocator and kmalloc() anyway.

- or NUMA_BALANCING's page->_last_cpupid can be 0 as well, as these
pages aren't (normally) NUMA-migrated.

The extended fields would have to be accessed via small wrappers,
which return 0 if the extended part is not present, but that's pretty
much all.

> I don't think we could have persistent struct page entries, as the
> exact contents of the struct page entries is too volatile and too
> different between architectures. [...]

Especially with the 2MB (and 1GB) granular lazy initialization
approach persisting them across reboots does not seem necessary

Even main RAM is already doing lazy initialization: Mel's patches that
do that just went into -mm.

> [...] Especially architecture changes that a pmem store is likely to
> see such as switching between a 32bit and a 64bit kernel.

We'd not want to ABI-restrict the layout of struct page. But to say
that there's a core 64-byte descriptor per 4K page is not an overly
strict promise to keep.

> Further I think where in the persistent memory the struct page
> arrays live is something we could leave up to the filesystem. We
> could have some reasonable constraints to make it fast but I think
> whoever decides where things live on the persistent memory can make
> that choice.

So the beauty of the scheme is that in its initial incarnation it's
filesystem independent: you can create any filesystem on top of it
seemlessly, the filesystem simply sees a linear block device that is
1.5% smaller than the underlying storage. It won't even (normally)
have access to the struct page areas. This kind of data space
separation also protects against filesystem originated data

Now in theory a filesystem might be aware of it, but I think it's far
more important to keep this scheme simple, robust, fast and

> For small persistent memories it probably make sense to allocate the
> struct page array describing them out of ordinary ram. For small
> memories I don't think we are talking enough memory to worry about.
> For TB+ persistent memories where you need 16GiB per TiB it makes
> sense to allocate a one or several regions to store your struct page
> arrays, as you can't count on ordinary ram having enough capacity,
> and you may not even be talking about a system that actually has
> ordinary ram at that point.

Correct - if there's no ordinary page cache in main DRAM then for many
appliances ordinary RAM could be something like SRAM: really fast and
not wasted on dirty state and IO caches - a huge, directly mapped L4
or L5 CPU cache in essence.


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at