Re: [PATCH v1 0/5] mm, kpageflags: support folio and fix output for compound pages

[Naoya Horiguchi]

On Mon, Oct 16, 2023 at 12:36:22PM +0100, Ryan Roberts wrote:
> On 16/10/2023 11:13, David Hildenbrand wrote:
> >>>>> It does sound inconsistent. What exactly do you want to tell user space with
> >>>>> the new flag?
> >>>>
> >>>> The current most problematic behavior is to report folio as thp (order-2
> >>>> pagecache page is definitely a folio but not a thp), and this is what the
> >>>> new flag is intended to tell.
> >>>
> >>> We are currently considering calling these sub-PMD sized THPs "small-sized
> >>> THP". [1] Arguably, we're starting with the anon part where we won't get
> >>> around exposing them to the user in sysfs.
> >>>
> >>> So I wouldn't immediately say that these things are not THPs. They are not
> >>> PMD-sized THP. A slab/hugetlb is certainly not a thp but a folio. Whereby
> >>> slabs can also be order-0 folios, but hugetlb can't.
> >>
> >> I think this is a mistake.  Users expect THPs to be PMD sized.  We already
> >> have the term "large folio" in use for file-backed memory; why do we
> >> need to invent a new term for anon large folios?
> > 
> > I changed my opinion two times, but I stabilized at "these are just huge pages
> > of different size" when it comes to user-visible features.
> > 
> > Handling/calling them folios internally -- especially to abstract the page vs.
> > compound page and how we manage/handle the metadata -- is a reasonable thing to
> > do, because that's what we decided to pass around.
> > 
> > 
> > For future reference, here is a writeup about my findings and the reason for my
> > opinion:
> > 
> > 
> > (1) OS-independent concept
> > 
> > Ignoring how the OS manages metadata (e.g., "struct page", "struct folio",
> > compound head/tail, memdesc, ...), the common term to describe a "the smallest
> > fixed-length contiguous block of physical memory into which memory pages are
> > mapped by the operating system.["[1] is a page frame -- people usually simplify
> > by dropping the "frame" part, so do I.
> > 
> > Larger pages (which we call "huge pages", FreeBSD "superpages", Windows "large
> > pages") can come in different sizes and were traditionally based on architecture
> > support, whereby architectures can support multiple ones [1]; I think what we
> > see is that the OS might use intermediate sizes to manage memory more
> > efficiently, abstracting/evolving that concept from the actual hardware page
> > table mapping granularity.
> > 
> > But the foundation is that we are dealing with "blocks of physical memory" in a
> > unit that is larger than the smallest page sizes. Larger pages.
> > 
> > [the comment about SGI IRIX on [1] is an interesting read; so are "scattered
> > superpages"[3]]
> > 
> > Users learned the difference between a "page" and a "huge page". I'm confident
> > that they can learn the difference between a "traditional huge page" and a
> > "small-sized huge page", just like they did with hugetlb (below).
> > 
> > We just have to be careful with memory statistics and to default to the
> > traditional huge pages for now. Slowly, the term "THP" will become more generic.
> > Apart from that, I fail to see the big source of confusion.
> > 
> > Note: FreeBSD currently similarly calls these things on arm64 "medium-sized
> > superpages", and did not invent new terms for that so far [2].
> > 
> > 
> > (2) hugetlb
> > 
> > Traditional huge pages started out to be PMD-sized. Before 2008, we only
> > supported a single huge page size. Ever since, we added support for sizes larger
> > (gigantic) and smaller than that (cont-pte / cont-pmd).
> > 
> > So (a) users did not panic because we also supported huge pages that were not
> > PMD-sized; (b) we managed to integrate it into the existing environment,
> > defaulting to the old PMD-sized huge pages towards the user but still providing
> > configuration knobs and (c) it is natural today to have multiple huge page sizes
> > supported in hugetlb.
> > 
> > Nowadays, when somebody says that they are using hugetlb huge pages, the first
> > question frequently is "which huge page size?". The same will happen with
> > transparent huge pages I believe.
> > 
> > 
> > (3) THP preparation for multiple sizes
> > 
> > With
> >     /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
> > added in 2016, we already provided a way for users to query the PMD size for
> > THP, implying that there might be multiple sizes in the future.
> > 
> > Therefore, in commit 49920d28781d, Hugh already envisioned " some transparent
> > support for pud and pgd pages" and ended up calling it "_pmd_size". Turns out,
> > we want smaller THPs first, not larger ones.
> > 
> > 
> > (4) Metadata management
> > 
> > How the OS manages metadata for its memory -- and how it calls the involved
> > datastructures -- is IMHO an implementation detail (an important one regarding
> > performance, robustness and metadata overhead as we learned, though ;) ).
> > 
> > We were able to introduce folios without user-visible changes. We should be able
> > to implement memdesc (or memory type hierarchies) without user-visible changes
> > -- except for some interfaces that provide access to bare "struct page"
> > information (classifies as debugging interfaces IMHO).
> > 
> > 
> > Last but not least, we ended up consistently calling these "larger than a page"
> > things that we map into user space "(transparent) huge page" towards the user in
> > toggles, stats and documentation. Fortunately we didn't use the term "compound
> > page" back then; it would have been a mistake.
> > 
> > 
> > Regarding the pagecache, we managed to not expose any toggles towards the user,
> > because memory waste can be better controlled. So the term "folio" does not pop
> > up as a toggle in /sys and /proc.
> > 
> >     t14s: ~  $ find /sys -name "*folio*" 2> /dev/null
> >     t14s: ~  $ find /proc -name "*folio*" 2> /dev/null
> > 
> > Once we want to remove the (sub)page mapcount, we'll likely have to remove
> > _nr_pages_mapped. To make some workloads that are sensitive to memory
> > consumption [4] play along when not accounting only the actually mapped parts,
> > we might have to introduce other ways to control that, when
> > "/sys/kernel/debug/fault_around_bytes" no longer does the trick. I'm hoping we
> > can still find ways to avoid exposing any toggles for that; we'll see.
> > 
> > 
> > [1] https://en.wikipedia.org/wiki/Page_(computer_memory)
> > [2] https://www.freebsd.org/status/report-2022-04-2022-06/superpages/
> > [3] https://ieeexplore.ieee.org/document/6657040/similar#similar
> > [4] https://www.suse.com/support/kb/doc/?id=000019017
> 
> +1 for David's reasoning.
> 
> FWIW, the way I see it, everything is a folio; a folio is an implementation
> detail that neatly abstracts a physically contiguous, power-of-2 number of pages
> (including the single page case). So I'm not sure how useful it is to add the
> proposed KPF_FOLIO flag. The only real thing I can imagine user space using it
> for would be to tell if some extent of virtual memory is physically contiguous,
> and you can already do that from the PFN.
> 
> Bigger picture interface-wise, I think it is simpler and more understandable to
> the user to extend an existing concept (THP) rather than invent a new one
> (folios) that substantially overlaps with the existing (PMD-sized) THP concept.
> 
> That said, if you have plans in the folio roadmap that I'm not aware of, then
> perhaps those would change my mind. There is a thread here [1] where we are
> discussing the best way to expose "small-sized THP" (anon large folios) to user
> space - Metthew if you you stong feelings, please do reply!
> 
> [1]
> https://lore.kernel.org/linux-mm/6d89fdc9-ef55-d44e-bf12-fafff318aef8@xxxxxxxxxx/
> 
> Thanks,
> Ryan
> 
> 
> > 
> > 
> >>
> >>> Looking at other interfaces, we do expose:
> >>>
> >>> include/uapi/linux/kernel-page-flags.h:#define KPF_COMPOUND_HEAD        15
> >>> include/uapi/linux/kernel-page-flags.h:#define KPF_COMPOUND_TAIL        16
> >>>
> >>> So maybe we should just continue talking about compound pages or do we have
> >>> to use both terms here in this interface?
> >>
> >> I don;t know how easy it's going to be to distinguish between a head
> >> and tail page in the Glorious Future once pages and folios are separated.
> > 
> > Probably a page-based interface would be the wrong interface for that;
> > fortunately, this interface has a "debugging" smell to it, so we might be able
> > to replace it.

This interface exposes per-pfn (not per-page) data records, specifying pfn by
file offset. It does not care about distinction between head and tail.
So I don't think that we can avoid referring to tail pages even after page-to-folio
conversion is complete.

But I agree that this interface is for debugging or testing.  To clarify
this, we might consider relocating this interface to a more suitable location
within debugfs, making it effectively invisible to non-debugging processes.
And maybe this could be the case also for other similar interfaces /proc/kpage*.
So all these files can be handled together to address this problem.

Thanks,
Naoya Horiguchi