Re: [PATCH v4 04/21] mm/hugetlb: Introduce nr_free_vmemmap_pages in the struct hstate

[Mike Kravetz]

On 11/13/20 2:59 AM, Muchun Song wrote:
> diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c
> new file mode 100644
> index 000000000000..a6c9948302e2
> --- /dev/null
> +++ b/mm/hugetlb_vmemmap.c
> @@ -0,0 +1,108 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Free some vmemmap pages of HugeTLB
> + *
> + * Copyright (c) 2020, Bytedance. All rights reserved.
> + *
> + *     Author: Muchun Song <songmuchun@xxxxxxxxxxxxx>
> + *

Oscar has already made some suggestions to change comments.  I would suggest
changing the below text to something like the following.

> + * Nowadays we track the status of physical page frames using struct page
> + * structures arranged in one or more arrays. And here exists one-to-one
> + * mapping between the physical page frame and the corresponding struct page
> + * structure.
> + *
> + * The HugeTLB support is built on top of multiple page size support that
> + * is provided by most modern architectures. For example, x86 CPUs normally
> + * support 4K and 2M (1G if architecturally supported) page sizes. Every
> + * HugeTLB has more than one struct page structure. The 2M HugeTLB has 512
> + * struct page structure and 1G HugeTLB has 4096 struct page structures. But
> + * in the core of HugeTLB only uses the first 4 (Use of first 4 struct page
> + * structures comes from HUGETLB_CGROUP_MIN_ORDER.) struct page structures to
> + * store metadata associated with each HugeTLB. The rest of the struct page
> + * structures are usually read the compound_head field which are all the same
> + * value. If we can free some struct page memory to buddy system so that we
> + * can save a lot of memory.
> + *

struct page structures (page structs) are used to describe a physical page
frame.  By default, there is a one-to-one mapping from a page frame to
it's corresponding page struct.

HugeTLB pages consist of multiple base page size pages and is supported by
many architectures. See hugetlbpage.rst in the Documentation directory for
more details.  On the x86 architecture, HugeTLB pages of size 2MB and 1GB
are currently supported.  Since the base page size on x86 is 4KB, a 2MB
HugeTLB page consists of 512 base pages and a 1GB HugeTLB page consists of
4096 base pages.  For each base page, there is a corresponding page struct.

Within the HugeTLB subsystem, only the first 4 page structs are used to
contain unique information about a HugeTLB page.  HUGETLB_CGROUP_MIN_ORDER
provides this upper limit.  The only 'useful' information in the remaining
page structs is the compound_head field, and this field is the same for all
tail pages.

By removing redundant page structs for HugeTLB pages, memory can returned
to the buddy allocator for other uses.

> + * When the system boot up, every 2M HugeTLB has 512 struct page structures
> + * which size is 8 pages(sizeof(struct page) * 512 / PAGE_SIZE).
> + *
> + *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> + * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> + * |           |                     |     0     | -------------> |     0     |
> + * |           |                     |     1     | -------------> |     1     |
> + * |           |                     |     2     | -------------> |     2     |
> + * |           |                     |     3     | -------------> |     3     |
> + * |           |                     |     4     | -------------> |     4     |
> + * |     2M    |                     |     5     | -------------> |     5     |
> + * |           |                     |     6     | -------------> |     6     |
> + * |           |                     |     7     | -------------> |     7     |
> + * |           |                     +-----------+                +-----------+
> + * |           |
> + * |           |
> + * +-----------+
> + *
> + *

I think we want the description before the next diagram.

Reworded description here:

The value of compound_head is the same for all tail pages.  The first page of
page structs (page 0) associated with the HugeTLB page contains the 4 page
structs necessary to describe the HugeTLB.  The only use of the remaining pages
of page structs (page 1 to page 7) is to point to compound_head.  Therefore,
we can remap pages 2 to 7 to page 1.  Only 2 pages of page structs will be used
for each HugeTLB page.  This will allow us to free the remaining 6 pages to 
the buddy allocator.  

Here is how things look after remapping.

> + *
> + *    HugeTLB                  struct pages(8 pages)         page frame(8 pages)
> + * +-----------+ ---virt_to_page---> +-----------+   mapping to   +-----------+
> + * |           |                     |     0     | -------------> |     0     |
> + * |           |                     |     1     | -------------> |     1     |
> + * |           |                     |     2     | -------------> +-----------+
> + * |           |                     |     3     | -----------------^ ^ ^ ^ ^
> + * |           |                     |     4     | -------------------+ | | |
> + * |     2M    |                     |     5     | ---------------------+ | |
> + * |           |                     |     6     | -----------------------+ |
> + * |           |                     |     7     | -------------------------+
> + * |           |                     +-----------+
> + * |           |
> + * |           |
> + * +-----------+

-- 
Mike Kravetz