[PATCH v2 2/3] sched/numa: do not feed tiering promotions into task placement stats

[Li Zhe]

NUMA balancing has separate mode bits for normal task placement and
memory tiering. In memory-tiering-only mode, NUMA hinting faults on
slow-tier pages are used to detect hot pages and promote them to a top
tier. They are not normal task-placement faults.

For slow-tier folios, memory tiering reuses the folio last-cpupid field
to store access-time metadata. numa_migrate_check() therefore cannot
return a real cpupid for those faults. Before this change, it returned the
same all-ones last_cpupid value that normal AutoNUMA uses for a first or
unknown access.

That can make a memory-tiering-only promotion look like an ordinary first
private AutoNUMA fault. After a successful promotion, do_numa_page()
reports the target top-tier node to task_numa_fault(), so the existing
slow-node skip no longer applies. task_numa_fault() then accounts the
fault in p->numa_faults and p->numa_faults_locality, and
task_numa_placement() can consume that sample to update normal task
placement state even though normal NUMA balancing is disabled. In other
words, hot-page promotion feedback can leak into the CPU placement
heuristic.

Mark faults that carried access-time metadata with TNF_ACCESS_TIME. If
normal NUMA balancing is disabled, keep the migrated-page accounting but
skip normal task-placement accounting for those faults.

When normal NUMA balancing and memory tiering are both enabled, promoted
pages can still feed task placement. In that mode, the user has enabled
both CPU placement feedback and tiering promotion, so a promoted top-tier
fault is treated as an ordinary unset-cpupid AutoNUMA fault.

This does not change whether a page is promoted. It only prevents
memory-tiering-only promotion faults from being consumed by the normal
AutoNUMA placement heuristic.

Using a fault flag avoids reserving an in-band value from the cpupid
namespace, where any masked value may also be produced by a real CPU/PID
pair.

Signed-off-by: Li Zhe <lizhe.67@xxxxxxxxxxxxx>
---
 include/linux/sched/numa_balancing.h |  1 +
 kernel/sched/fair.c                  | 13 +++++++++++++
 mm/memory.c                          |  6 ++++--
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/include/linux/sched/numa_balancing.h b/include/linux/sched/numa_balancing.h
index 52b22c5c396d..200f50075d5c 100644
--- a/include/linux/sched/numa_balancing.h
+++ b/include/linux/sched/numa_balancing.h
@@ -14,6 +14,7 @@
 #define TNF_SHARED	0x04
 #define TNF_FAULT_LOCAL	0x08
 #define TNF_MIGRATE_FAIL 0x10
+#define TNF_ACCESS_TIME	0x20
 
 enum numa_vmaskip_reason {
 	NUMAB_SKIP_UNSUITABLE,
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 46a11f5ddafb..136207b62d87 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4149,6 +4149,19 @@ void task_numa_fault(int last_cpupid, int mem_node, int pages, int flags)
 	if (!p->mm)
 		return;
 
+	/*
+	 * Slow-tier pages use last_cpupid as access-time metadata instead of
+	 * a real cpupid. Do not feed these faults into normal task placement
+	 * when normal NUMA balancing is disabled.
+	 */
+	if (flags & TNF_ACCESS_TIME) {
+		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_NORMAL)) {
+			if (migrated)
+				p->numa_pages_migrated += pages;
+			return;
+		}
+	}
+
 	/*
 	 * NUMA faults statistics are unnecessary for the slow memory
 	 * node for memory tiering mode.
diff --git a/mm/memory.c b/mm/memory.c
index 0fb676dd2bd6..00b171e2fe23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -6241,10 +6241,12 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	 * For memory tiering mode, cpupid of slow memory page is used
 	 * to record page access time.  So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_use_access_time(folio)) {
+		*flags |= TNF_ACCESS_TIME;
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
-	else
+	} else {
 		*last_cpupid = folio_last_cpupid(folio);
+	}
 
 	/* Record the current PID accessing VMA */
 	vma_set_access_pid_bit(vma);
-- 
2.20.1