[PATCH v4 0/8] mm: fix allocator races with memory hotplug

[Gregory Price (Meta)]

Memory hotplug rebuilds zonelists in place while allocations walk them,
and offlining an emptied zone frees pagesets the vmstat worker can still
hold.  These eight fixes cover the walkers that crash, fail, or pick the
wrong zone or node as a result.

   1. a restarted slowpath loses its retry with reserves
   2. the zonelist retry is compiled out without MEMORY_HOTREMOVE
   3. NULL preferred zone after a shrink rebuild
   4. __next_zones_zonelist() rereads a slot
   5. allocations land outside their zone or nodemask
   6. the hugetlb pool walk takes another node's page
   7. alloc_contig_pages() takes a range outside its filters
   8. mempolicy_slab_node() rereads a slot

All eight are for stable.

Patches 5-7 need patch 2, without which their retry is compiled
out on HOTREMOVE=n kernels,

Patch 5 needs patch 1.

Testing
=======

Each race was reproduced on unmodified source in a QEMU guest with three
NUMA nodes and a 512M pc-dimm on node 1, left offline at boot.

A test module widens each window with a kprobe that stalls one CPU inside
it, as an interrupt or a vCPU preemption would.

       base                            this series
patch  outcome                  runs   outcome                  runs
=====  =======================  =====  =======================  =====
 1     NULL returned            3/3    page from reserves       3/3
 2     allocation fails         3/3    waits for the rebuild    3/3
 3     NULL dereference         3/3    ZONE_NORMAL page         3/3
 4     NULL dereference         3/3    ZONE_NORMAL page         3/3
 5     wrong zone or node       18/18  right zone and node      18/18
 6     page from node 1         3/3    page from node 2         3/3
 7     1G page in ZONE_MOVABLE  3/3    1G page in ZONE_NORMAL   3/3
 8     NULL dereference         4/4    keeps allocating         3/3

Cost
====
Kernel instructions per order-0 allocation and free in the QEMU guest,
counted with a task-bound hardware instruction counter around 1M
iterations on a pinned kthread, non-debug kernel, mean of 15 rounds:

                           instructions per alloc+free
                             base   this series
  alloc_pages()             890.5         900.5
  bulk, nr=1, nodemask      865.5         875.5
  bulk, nr=16, per page     477.2         478.1

Bare metal, two-socket Xeon Gold 6138 (Skylake-SP, 80 CPUs),
12 kexec boots per kernel across two hosts.

will-it-scale page_fault1-3: each iteration touches one page, so the
unit is page faults per second, summed over all tasks.

each boot takes the mean of 8 one-second samples.

Base, millions of faults per second:

  tasks                         1     10     40     80
  page_fault1, processes     0.46   4.16  10.51   9.23
  page_fault1, threads       0.46   3.51   4.35   4.19
  page_fault2, processes     0.43   3.66   9.02   8.33
  page_fault2, threads       0.43   3.48   4.37   4.19
  page_fault3, processes     3.89  14.78   8.10   8.00
  page_fault3, threads       3.87  14.29   6.88   7.96

Change in faults per second with this series, + is faster:

  tasks                         1     10     40     80
  page_fault1, processes    -0.3%  +0.1%  -3.0%  -1.7%
  page_fault1, threads      -0.5%  -1.6%  -0.9%  -0.9%
  page_fault2, processes    +0.0%  +0.9%  -1.4%  -0.5%
  page_fault2, threads      +0.3%  +1.5%  -1.4%  -1.0%
  page_fault3, processes    -1.0%  -3.8%  -0.7%  +1.2%
  page_fault3, threads      -0.9%  -0.7%  +0.8%  -0.2%

The boot-to-boot stddev is 1.0-1.8% at 1 task and up to 15% above
1 task, so no column shows a change larger than its noise.

Changes in v4
=============
- dropped 2 patches already covered by other posted series
- Added patch 1. We found that a restart, also reachable through
  cpuset updates, lost the caller's retry with reserves.
- Patch 4's changelog now says that an eligible page is dropped too,
  and that an ALLOC_NOLOCK caller gets NULL.
- Patch 6 checks for a rebuild before escalating to the pass that may
  migrate hugetlb pages.
- Added patch 8 for mempolicy_slab_node()'s double read, the one
  remaining instance found by auditing every zonelist walker.

Link: https://lore.kernel.org/linux-mm/20261014222456.2318284-1-gourry@xxxxxxxxxx/
Link: https://lore.kernel.org/linux-mm/20261015210539.3243563-1-gourry@xxxxxxxxxx/
Link: https://lore.kernel.org/linux-mm/20261019172346.3344005-1-gourry@xxxxxxxxxx/

-- 
2.53.0-Meta


Gregory Price (Meta) (8):
  mm/page_alloc: reset can_retry_reserves when the slowpath restarts
  mm/page_alloc: detect zonelist rebuilds from memory hot-add
  mm/page_alloc: snapshot the preferred zone against shrink rebuilds
  mm/mmzone: read each zoneref slot once when filtering by nodemask
  mm/page_alloc: retry allocations that raced with a zonelist rebuild
  mm/hugetlb: retry the pool walk when the zonelist is rebuilt
  mm/page_alloc: retry alloc_contig_pages() when the zonelist is rebuilt
  mm/mempolicy: read the zoneref slot once in mempolicy_slab_node()

 mm/hugetlb.c    | 13 ++++++--
 mm/mempolicy.c  |  4 ++-
 mm/mmzone.c     | 11 ++++---
 mm/page_alloc.c | 85 +++++++++++++++++++++++++++++++++++++++++--------
 mm/page_alloc.h |  3 ++
 5 files changed, 93 insertions(+), 23 deletions(-)