Re: State of the Page (August 2022)

[Kirill A. Shutemov]

On Thu, Aug 11, 2022 at 10:31:21PM +0100, Matthew Wilcox wrote:
> ==============================
> State Of The Page, August 2022
> ==============================
> 
> I thought I'd write down where we are with struct page and where
> we're going, just to make sure we're all (still?) pulling in a similar
> direction.
> 
> Destination
> ===========
> 
> For some users, the size of struct page is simply too large.  At 64
> bytes per 4KiB page, memmap occupies 1.6% of memory.  If we can get
> struct page down to an 8 byte tagged pointer, it will be 0.2% of memory,
> which is an acceptable overhead.

Right. This is attractive. But it brings cost of indirection.

It can be especially painful for physical memory scanning. I guess we can
derive some info from memdesc type itself, like if it can be movable. But
still looks like an expensive change.

Do you have any estimation on how much CPU time we will pay to reduce
memory (and cache) overhead? RAM size tend to grow faster than IPC.
We need to make sure it is the right direction.

>    struct page {
>       unsigned long mem_desc;
>    };
> 
> Types of memdesc
> ----------------
> 
> This is very much subject to change as new users present themselves.
> Here are the current ones in-plan:
> 
>  - Undescribed.  Instead of the rest of the word being a pointer,
>    there are 2^28 subtypes available:
>    - Unmappable.  Typically device drivers allocating private memory.
>    - Reserved.  These pages are not allocatable.
>    - HWPoison
>    - Offline (eg balloon)
>    - Guard (see debug_pagealloc)
>  - Slab
>  - Anon Folio
>  - File Folio
>  - Buddy (ie free -- also for PCP?)
>  - Page Table
>  - Vmalloc
>  - Net Pool
>  - Zsmalloc
>  - Z3Fold
>  - Mappable.  Typically device drivers mapping memory to userspace
> 
> That implies 4 bits needed for the tag, so all memdesc allocations
> must be 16-byte aligned.  That is not an undue burden.  Memdescs
> must also be TYPESAFE_BY_RCU if they are mappable to userspace or
> can be stored in a file's address_space.
> 
> It may be worth distinguishing between vmalloc-mappable and
> vmalloc-unmappable to prevent some things being mapped to userspace
> inadvertently.

Given that memdesc represents Slab too, how do we allocate them?

> 
> Contents of a memdesc
> ---------------------
> 
> At least initially, the first word of a memdesc must be identical to the
> current page flags.  That allows various functions (eg set_page_dirty())
> to work on any kind of page without needing to know whether it's a device
> driver page, a vmalloc page, anon or file folio.
> 
> Similarly, both anon and file folios must have the list_head in the
> same place so they can be placed on the same LRU list.  Whether anon
> and file folios become separate types is still unclear to me.
> 
> Mappable
> --------
> 
> All pages mapped to userspace must have:
> 
>  - A refcount
>  - A mapcount
> 
> Preferably in the same place in the memdesc so we can handle them without
> having separate cases for each type of memdesc.  It would be nice to have
> a pincount as well, but that's already an optional feature.
> 
> I propose:
> 
>    struct mappable {
>        unsigned long flags;	/* contains dirty flag */
>        atomic_t _refcount;
>        atomic_t _mapcount;
>    };
> 
>    struct folio {
>       union {
>          unsigned long flags;
>          struct mappable m;
>       };
>       ...
>    };

Hm. How does lockless page cache lookup would look like in this case?

Currently it relies on get_page_unless_zero() and to keep it work there's
should be guarantee that nothing else is allocated where mappable memdesc
was before. Would it require some RCU tricks on memdesc free?

-- 
  Kiryl Shutsemau / Kirill A. Shutemov