[PATCH v5 0/7] sched/numa: stop VMA scan filters from gating promotion

[Gregory Price (Meta)]

NUMA balancing uses hinting faults for both task placement and
memory-tier promotion. Several filters designed to avoid unproductive
socket-placement faults also prevent promotion.

When testing testing fairness controls [1] we observed unexpected
bandwidth utilization between tiers. We found that numa balancing
was completely broken - starting with a recent shmem fix [2].

No other numa controls are used on these workloads.
  - no numactl / mbind / mempolicy
  - no cpuset.mems

These fixes impact the fault-injecting side (spraying PROT_NONE),
not the fault-handling side. So mempolicy and cpuset doesn't really
come into the picture.

The core issue is that the global balancing mode says which mechanisms
are enabled, but it does not describe the intent of an individual
VMA/PTE walk (placement vs promotion).

This series plumbs the scan reasoning into the prot_none injection
while retaining the placement-scan filtering.

Testing
===
We tested on an AMD Zen5 platform with 768GB of DRAM and 256GB of CXL.

We use kernel.numa_balancing=2 in production, but there are subtle
interactions with numa_balancing=3 that are also handled by this set.

Workload: two ~430 GB databases with large (separate) shmem VMAs.

Before patch series:
  - 150-200 GB/s DRAM bandwidth
  - 40-45 GB/s CXL bandwidth, saturating the device
  - request latency above 5 ms with longer tails
  - Large portions of hot memory permanently stuck on CXL tier.

After patch series:
  - 250+ GB/s sustained DRAM bandwidth
  - approximately 5-10 GB/s sustained CXL bandwidth
  - request latency between 800 us and 2 ms
  - Hot memory now gets correctly migrated to the local node.

Stable Backporting
===
Patches 1-5 form the backportable fix set:

  1. Add promotion-only NUMA protection walks and derive private VMA
     state in the folio eligibility check.
  2. Permit eligible shared folios to be promoted to a fast tier and
     rename the promotion predicate to folio_numab_promotable().
  3. Scan mappings covered by the legacy file placement filter using
     promotion-only scans.
  4. Separate VMA placement eligibility from partial-scan continuation.
  5. Scan PID-inactive VMAs for promotion and account for placement scans
     separately.

Patches 4 and 5 are technically part of the same fix, but are separated
to make it easier to review - they must be backported together.

Patches 6-7 are independently requested cleanups.

  6. Use BIT() for the change_protection() flags.
  7. Use the VMA flag API in the touched NUMA-balancing code.

Changes in v5:
 - Patch 4: small formatting change (David)
 - Patch 4: changelog comment for sashiko east-west false positive issue

[1] https://lore.kernel.org/all/20260807202059.2620949-1-joshua.hahnjy@xxxxxxxxx/
[2] https://lore.kernel.org/all/20260629163337.1264881-1-hannes@xxxxxxxxxxx/

Gregory Price (Meta) (7):
  mm: support promotion-only NUMA hinting scans
  mm: allow shared folios to be promoted to a fast tier
  sched/numa: scan read-only file mappings in tiering mode
  sched/numa: separate VMA placement from scan continuation
  sched/numa: scan PID-inactive VMAs for promotion
  mm: use BIT() for change_protection() flags
  mm: use VMA flag helpers in NUMA balancing

 include/linux/mm.h       |  23 +++++----
 include/linux/mm_types.h |  13 +++++
 kernel/sched/fair.c      | 100 ++++++++++++++++++++++++++++-----------
 mm/huge_memory.c         |   3 +-
 mm/internal.h            |   6 +--
 mm/memory-tiers.c        |   9 ++--
 mm/memory.c              |   2 +-
 mm/mempolicy.c           |  37 ++++++++-------
 mm/migrate.c             |  13 +++--
 mm/mprotect.c            |   7 +--
 10 files changed, 140 insertions(+), 73 deletions(-)

-- 
2.53.0-Meta