[PATCH 05/15] mm/hugetlb: keep surplus folios when the pool shrinks

[David Hildenbrand (Arm)]

From: Josef Bacik <josef@xxxxxxxxxxxxxx>

Surplus folios are accounted per node. When shrinking the pool,
set_max_huge_pages() removes free folios through
remove_pool_hugetlb_folio() and accounts them as persistent folios.

It can select a node whose folios are all surplus and leave its surplus
count larger than its total count. Free surplus folios can back
reservations created by gather_surplus_pages().

For example, with nr_overcommit_hugepages = 8 on two nodes:

  (node columns show total/free/surplus)

                                      total free surp   node0   node1
  mmap() of one page, bound to node 0     1    1    1   1 1 1   0 0 0
  node1/nr_hugepages = 7                  8    8    1   1 1 1   7 7 0
  nr_hugepages = 6                        7    7    1   0 0 1   7 7 0
  munmap()                                7    7    1   0 0 1   7 7 0
  nr_hugepages = 0                        1    1    1   0 0 1   1 1 0

After shrinking to six persistent folios, node 0 has no folios but its
surplus count is still one. Releasing the reservation cannot clear that
count, and shrinking the pool to zero leaves one free page on node 1.

A later write to node0/nr_hugepages underflows the unsigned per-node
persistent count: 0 - 1, resulting in allocation of many folios.

Use persistent_huge_pages_node() to skip nodes with no persistent folios,
as adjust_pool_surplus() does. This also excludes temporary folios.

Fixes: e4e574b767ba ("hugetlb: Try to grow hugetlb pool for MAP_SHARED mappings")
Cc: stable@xxxxxxxxxx
Signed-off-by: Josef Bacik <josef@xxxxxxxxxxxxxx>
[ david: cleanup code and description ]
Signed-off-by: David Hildenbrand (Arm) <david@xxxxxxxxxx>
---
 mm/hugetlb.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 110a5c2d9e50..55f7d692ed0d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1955,17 +1955,25 @@ static struct folio *remove_pool_hugetlb_folio(struct hstate *h,
 
 	lockdep_assert_held(&hugetlb_lock);
 	for_each_node_mask_to_free(h, nr_nodes, node, nodes_allowed) {
+		if (list_empty(&h->hugepage_freelists[node]))
+			continue;
+
 		/*
 		 * If we're returning unused surplus pages, only examine
-		 * nodes with surplus pages.
+		 * nodes with surplus pages. If we're shrinking the pool, only
+		 * examine nodes with persistent folios.
 		 */
-		if ((!acct_surplus || h->surplus_huge_pages_node[node]) &&
-		    !list_empty(&h->hugepage_freelists[node])) {
-			folio = list_entry(h->hugepage_freelists[node].next,
-					  struct folio, lru);
-			remove_hugetlb_folio(h, folio, acct_surplus);
-			break;
+		if (acct_surplus) {
+			if (!h->surplus_huge_pages_node[node])
+				continue;
+		} else if (!persistent_huge_pages_node(h, node)) {
+			continue;
 		}
+
+		folio = list_first_entry(&h->hugepage_freelists[node],
+					 struct folio, lru);
+		remove_hugetlb_folio(h, folio, acct_surplus);
+		break;
 	}
 
 	return folio;

-- 
2.43.0