Re: [Patch v4 17/22] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
From: Jirka Hladky
Date: Mon Nov 16 2026 - 09:36:11 EST
Hi Chen Yu, Tim,
> Do you mean accumulate the pages into the group in task_numa_fault()?
> ...
> Would this decay cause a jitter of the footprint? Anyway, I thought
> what you suggested is the following for fix2 ...
I built and tested the sketch on the Sierra Forest box: a kernel built
from the same 7.4.0-0.rc2.19.eln159 source as my baselines, patched
with your sketch plus the fixes needed to compile it, and with one
change: the decay is applied once per group per sweep via a
last_decay_seq stamp, since decaying per task would halve the group
footprint once per thread per sweep.
The accounting mechanism works exactly as designed:
- The footprint accumulates during the sweep and crosses the
27,648-page LLC threshold at t=4.4 s; the aggregation is withdrawn
and stays withdrawn (grp->cpu = -1 in 96% of 27,796 samples). To
answer the jitter question: with once-per-sweep decay I saw no
oscillation in a run whose footprint reached ~3.95M pages.
- Removing the exit()/exec() subtraction held up fine as well.
But the regression is not fixed. NAS MG class D, 1x64 threads, n=5,
median time:
stock rc2.19, CAS off 26.45 s
stock rc2.19, CAS on 41.78 s (+58%)
sketch, CAS off 26.57 s
sketch, CAS on 39.19 s (+47.5%)
The regression drops from +58% to +47.5%: 2.6 s of the 15.3 s gap,
about one sixth, and the reason is placement hysteresis. The 2.1 s
window between the aggregation settling
and the gate tripping has already shaped the run: the threads re-spread
at only ~1 thread per second (4/60 across the two nodes at t=5 s,
20/44 at t=28 s), and the memory first-touched during the window is
59/41 toward the aggregated node, with NUMA balancing then pulling even
more memory there as the run continues. Withdrawing grp->cpu stops the
steering but does not undo the placement, the fault statistics, or the
page layout.
A few observations for the design:
1. Fix 1 (the RSS gate) looks critical for this workload class. MG's
RSS crosses the 108 MiB LLC size within ~0.1 s of the allocation
phase, so an RSS gate would refuse aggregation almost immediately,
before first-touch lays down the skewed layout.
2. Fix 2's trip time is bounded below by the scanner, not by the fault
count: the 4.4 s crossing decomposes into the 1 s scan delay plus
the 2-3 s the scanner needs to reach the large VMAs of a 26.4 GiB
address space. Even the theoretical ~1.8 s would arrive after ~60%
of the working set is first-touched.
3. One caveat on the withdrawal path, shared by both fixes: whenever
the gate trips after aggregation has been in effect, passive
withdrawal (clearing grp->cpu) recovers almost nothing on this
workload; the tasks re-spread at ~1 per second. An effective
withdrawal would need to actively re-spread the tasks or reset
their NUMA preferences. With Fix 1 in place this matters less for
the regression class I reported, since aggregation is refused
before it settles; it applies to workloads that grow past the LLC
mid-run.
Happy to re-test when the next revision is posted. The exact version I
tested, on top of your sketch with the changes above, is below. The diff
is against v7.4-rc2; the hunks are in code my downstream patchset does
not touch, so it should apply to your tree as well.
Thanks,
Jirka
On Mon, Nov 16, 2026 at 6:45 AM Chen Yu <chen.yu@xxxxxxxxx> wrote:
---
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1446,6 +1446,7 @@
int preferred_llc;
/* 1: task was enqueued to its preferred LLC, 0 otherwise */
int pref_llc_queued;
+ unsigned long total_footprint;
#endif
struct rseq_data rseq;
@@ -2396,6 +2397,7 @@
u64 nr_running_avg;
unsigned long next_scan;
unsigned long footprint;
+ unsigned long last_decay_seq;
int cpu;
refcount_t refcnt;
struct rcu_head rcu;
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -1621,6 +1621,7 @@
grp->next_scan = jiffies;
grp->nr_running_avg = 0;
grp->footprint = 0;
+ grp->last_decay_seq = 0;
refcount_set(&grp->refcnt, 1);
/*
* The update to grp->pcpu_sched should not be reordered
@@ -1730,19 +1731,6 @@
{
struct sched_cache_group *grp = sched_cache_replace_grp(p, NULL);
-#ifdef CONFIG_NUMA_BALANCING
- /*
- * Subtract this task's footprint from the group before dropping the
- * reference, so the group footprint converges as its threads exit.
- * Unlocked for performance; clamp to avoid underflow.
- */
- if (grp && p->total_numa_faults) {
- unsigned long fp = READ_ONCE(grp->footprint);
- unsigned long sub = min(fp, p->total_numa_faults);
-
- WRITE_ONCE(grp->footprint, fp - sub);
- }
-#endif
sched_cache_group_put(grp);
}
@@ -2125,6 +2113,7 @@
*/
p->preferred_llc = -1;
p->pref_llc_queued = 0;
+ p->total_footprint = 0;
}
#else /* CONFIG_SCHED_CACHE */
@@ -3826,6 +3815,62 @@
return nid;
}
+#if defined(CONFIG_SCHED_CACHE) && defined(CONFIG_NUMA_BALANCING)
+/*
+ * Fix 2 (Chen Yu sketch, test build): accumulate the pages reported by
+ * task_numa_fault() into the group footprint as they arrive within the
+ * current sweep, instead of waiting for task_numa_placement() at the sweep
+ * boundary. Batched per task to limit writes to the shared counter.
+ */
+#define SCHED_CACHE_FP_THRESHOLD 10
+static void sched_cache_footprint_accum(struct task_struct *p, int pages)
+{
+ struct sched_cache_group *grp;
+
+ p->total_footprint += pages;
+
+ if (p->total_footprint < SCHED_CACHE_FP_THRESHOLD)
+ return;
+
+ guard(rcu)();
+ grp = rcu_dereference(p->sched_cache_grp);
+ if (!grp)
+ return;
+
+ /* drain the per-task footprint into the group */
+ WRITE_ONCE(grp->footprint, READ_ONCE(grp->footprint) + p->total_footprint);
+ p->total_footprint = 0;
+}
+
+/*
+ * Decay once per group per sweep: the first task whose placement() observes
+ * a new scan sequence halves the accumulated window. (Decaying per task, as
+ * in the original sketch, would halve the group footprint once per thread
+ * per sweep.)
+ */
+static void sched_cache_footprint_decay(struct task_struct *p)
+{
+ struct sched_cache_group *grp;
+ unsigned long seq = p->numa_scan_seq;
+ unsigned long old;
+
+ guard(rcu)();
+ grp = rcu_dereference(p->sched_cache_grp);
+ if (!grp)
+ return;
+
+ old = READ_ONCE(grp->last_decay_seq);
+ if (old == seq)
+ return;
+ if (cmpxchg(&grp->last_decay_seq, old, seq) == old)
+ WRITE_ONCE(grp->footprint, READ_ONCE(grp->footprint) / 2);
+}
+
+#else
+static inline void sched_cache_footprint_accum(struct task_struct *p,
int pages) { }
+static inline void sched_cache_footprint_decay(struct task_struct *p) { }
+#endif
+
static void task_numa_placement(struct task_struct *p)
__context_unsafe(/* conditional locking */)
{
@@ -3911,36 +3956,6 @@
ng->total_faults += diff;
group_faults += ng->faults[mem_idx];
}
-#ifdef CONFIG_SCHED_CACHE
- /*
- * Per task p->numa_faults[mem_idx] converges,
- * so the accumulation of each task's faults
- * converges too - Given the number of threads,
- * it cannot overflow an unsigned long.
- * Racy with concurrent updates from other threads
- * sharing this mm. Acceptable since footprint is a
- * heuristic and occasional lost updates are tolerable.
- *
- * If a task exits, its corresponding footprint must
- * be subtracted from p->sched_cache_grp->footprint,
- * otherwise the footprint will not converge: the
- * exiting thread's footprint remains unchanged/undecayed.
- * See exit_mm().
- *
- * Lost updates and unsynchronized subtraction
- * in exit_mm() can cause footprint + diff to
- * go negative. Clamp to zero to prevent the
- * unsigned footprint from wrapping.
- */
- scoped_guard(rcu) {
- grp = rcu_dereference(p->sched_cache_grp);
-
- if (grp) {
- new_fp = (long)READ_ONCE(grp->footprint) + diff;
- WRITE_ONCE(grp->footprint, max(new_fp, 0L));
- }
- }
-#endif
}
if (!ng) {
@@ -3970,6 +3985,7 @@
}
update_task_scan_period(p, fault_types[0], fault_types[1]);
+ sched_cache_footprint_decay(p);
}
static inline int get_numa_group(struct numa_group *grp)
@@ -4210,6 +4226,7 @@
p->numa_faults[task_faults_idx(NUMA_MEMBUF, mem_node, priv)] += pages;
p->numa_faults[task_faults_idx(NUMA_CPUBUF, cpu_node, priv)] += pages;
p->numa_faults_locality[local] += pages;
+ sched_cache_footprint_accum(p, pages);
}
static void reset_ptenuma_scan(struct task_struct *p)