Re: Struct page proposal

[Matthew Wilcox]

On Wed, Sep 22, 2021 at 09:21:31PM -0400, Kent Overstreet wrote:
> The fundamental reason for struct page is that we need memory to be self
> describing, without any context - we need to be able to go from a generic
> untyped struct page and figure out what it contains: handling physical memory
> failure is the most prominent example, but migration and compaction are more
> common. We need to be able to ask the thing that owns a page of memory "hey,
> stop using this and move your stuff here".

Yup, and another thing we need is to take any page mapped to userspace
and mark it as dirty (whatever that means for the owner of the page).

> Matthew's helpfully been coming up with a list of page types:
> https://kernelnewbies.org/MemoryTypes
> 
> But struct page could be a lot smaller than it is now. I think we can get it
> down to two pointers, which means it'll take up 0.4% of system memory. Both
> Matthew and Johannes have ideas for getting it down even further - the main
> thing to note is that virt_to_page() _should_ be an uncommon operation (most of
> the places we're currently using it are completely unnecessary, look at all the
> places we're using it on the zero page). Johannes is thinking two layer radix
> tree, Matthew was thinking about using maple trees - personally, I think that
> 0.4% of system memory is plenty good enough.

As with a lot of these future plans, I think the details can vary
slightly.  What I propose on the above wiki page is to take it
down to one pointer per page, but yes, I have a dream that eventually we
can take it down to one pointer + size per allocation (so 16 bytes)
rather than 16 bytes per page.

> Ok, but what do we do with the stuff currently in struct page?
> -------------------------------------------------------------
> 
> The main thing to note is that since in normal operation most folios are going
> to be describing many pages, not just one - and we'll be using _less_ memory
> overall if we allocate them separately. That's cool.
> 
> Of course, for this to make sense, we'll have to get all the other stuff in
> struct page moved into their own types, but file & anon pages are the big one,
> and that's already being tackled.

We can also allocate a far larger structure.  eg, we might decide that
a file page looks like this:

struct folio {
    unsigned long flags;
    unsigned long pfn;
    struct list_head lru;
    struct address_space *mapping;
    pgoff_t index;
    void *private;
    atomic_t _mapcount;
    atomic_t _refcount;
#ifdef CONFIG_MEMCG
    unsigned long memcg_data;
#endif
    unsigned char dtor;
    unsigned char order;
    atomic_t hmapcount;
    unsigned int nr_pages;
    atomic_t hpinned_count;
    struct list_head deferred_list;
};

(compiling that list reminds me that we'll need to sort out mapcount
on subpages when it comes time to do this.  ask me if you don't know
what i'm talking about here.)

> Why two ulongs/pointers, instead of just one?
> ---------------------------------------------
> 
> Because one of the things we really want and don't have now is a clean division
> between allocator and allocatee state. Allocator meaning either the buddy
> allocator or slab, allocatee state would be the folio or the network pool state
> or whatever actually called kmalloc() or alloc_pages().
> 
> Right now slab state sits in the same place in struct page where allocatee state
> does, and the reason this is bad is that slab/slub are a hell of a lot faster
> than the buddy allocator, and Johannes wants to move the boundary between slab
> allocations and buddy allocator allocations up to like 64k. If we fix where slab
> state lives, this will become completely trivial to do.
> 
> So if we have this:
> 
> struct page {
> 	unsigned long	allocator;
> 	unsigned long	allocatee;
> };
> 
> The allocator field would be used for either a pointer to slab/slub's state, if
> it's a slab page, or if it's a buddy allocator page it'd encode the order of the
> allocation - like compound order today, and probably whether or not the
> (compound group of) pages is free.
> 
> The allocatee field would be used for a type tagged (using the low bits of the
> pointer) to one of:
>  - struct folio
>  - struct anon_folio, if that becomes a thing
>  - struct network_pool_page
>  - struct pte_page
>  - struct zone_device_page

I think we /can/ do all this.  I don't know that it's the right thing to
do.  And I really mean that.  I genuinely don't know that "allocate
file pages from slab" will solve any problems at all.  And I kind of
don't want to investigate that until later.

By the way, another way we could do this is to put the 'allocator'
field into the allocatee's data structure.  eg the first word
in struct folio could point to the struct slab that contains it.

> Other notes & potential issues:
>  - page->compound_dtor needs to die

The reason we have it right now is that the last person to call
put_page() may not be the one who allocated it.  _maybe_ we can do
all-of-the-dtor-stuff when the person who allocates it frees it, and
have put_page() only free the memory.  TBD.

>  - page->rcu_head moves into the types that actually need it, no issues there

Hope so!

>  - page->refcount has question marks around it. I think we can also just move it
>    into the types that need it; with RCU derefing the pointer to the folio or
>    whatever and grabing a ref on folio->refcount can happen under a RCU read
>    lock - there's no real question about whether it's technically possible to
>    get it out of struct page, and I think it would be cleaner overall that way.
> 
>    However, depending on how it's used from code paths that go from generic
>    untyped pages, I could see it turning into more of a hassle than it's worth.
>    More investigation is needed.

I think this depends how far we go splitting everything apart.

>  - page->memcg_data - I don't know whether that one more properly belongs in
>    struct page or in the page subtypes - I'd love it if Johannes could talk
>    about that one.

Johannes certainly knows more about this than I do.  I think it's needed
for anon folios, file folios and slab, but maybe it's needed for page
tables too?

>  - page->flags - dealing with this is going to be a huge hassle but also where
>    we'll find some of the biggest gains in overall sanity and readability of the
>    code. Right now, PG_locked is super special and ad hoc and I have run into
>    situations multiple times (and Johannes was in vehement agreement on this
>    one) where I simply could not figure the behaviour of the current code re:
>    who is responsible for locking pages without instrumenting the code with
>    assertions.
> 
>    Meaning anything we do to create and enforce module boundaries between
>    different chunks of code is going to suck, but the end result should be
>    really worthwhile.
> 
> Matthew Wilcox and David Howells have been having conversations on IRC about
> what to do about other page bits. It appears we should be able to kill a lot of
> filesystem usage of both PG_private and PG_private_2 - filesystems in general
> hang state off of page->private, soon to be folio->private, and PG_private in
> current use just indicates whether page->private is nonzero - meaning it's
> completely redundant.

Also I want to kill PG_error.  PG_slab and PG_hwpoison become page
types (in the "allocatee" field in your parlance).  PG_head goes away
naturally.

Something we don't have to talk about right now is that there's no reason
several non-contiguous pages can't have the same 'allocatee' value.
I'm thinking that every page allocated to a given vmalloc allocation
would point to the same vm_struct.  So I'm thinking that the way all of
the above would work is we'd allocate a "struct folio" from slab, then
pass the (tagged) pointer to alloc_pages().  alloc_pages() would fill
in the 'allocatee' pointer for each of the struct pages with whatever
pointer it is given.

There's probably a bunch of holes in the above handwaving, but I'm
pretty confident we can fill them.