Re: [PATCH v8 1/2] sched/cache: Reduce the overhead of task_cache_work by only scan the visisted cpus
From: Luo Gengkun
Date: Tue Jul 28 2026 - 05:22:07 EST
On 2026/7/27 9:09, Chen, Yu C wrote:
On 7/23/2026 12:04 PM, Luo Gengkun wrote:I don't think we need it back. Here is why:
[ ... ]
+static unsigned long fraction_mm_sched(int cpu,
+ struct mm_struct *mm)
{
+ struct sched_cache_time *pcpu_sched =
+ per_cpu_ptr(mm->sc_stat.pcpu_sched, cpu);
+ struct rq *rq = cpu_rq(cpu);
+
guard(raw_spinlock_irqsave)(&rq->cpu_epoch_lock);
__update_mm_sched(rq, pcpu_sched);
+ /* Skip the rq that has not been hit for a long time */
+ if ((rq->cpu_epoch - pcpu_sched->epoch_last_visit) > llc_epoch_affinity_timeout) {
In v2 there is a check if the cpu has been set before writing:
cpumask_test_cpu(cpu_of(rq), &mm->sc_stat.visited_cpus)
https://lore.kernel.org/all/20260414150745.225416-1-luogengkun2@xxxxxxxxxx/
do we need to bring that back?
In v2, for_each_cpu was used instead of for_each_cpu_and in the inner loop,
meaning some CPUs being checked might not have been set. Therefore,
cpumask_test_cpu was necessary to filter out those cases.
Now, with for_each_cpu_and(i, sched_domain_span(sd), &mm->sc_stat.visited_cpus),
we can ensure each scanned CPU is set, so the issue no longer exists.
Furthermore, the only place where the visited_cpus bits are cleared is
task_cache_work(), which is only called once per scan period, there is no
risk of the bit being cleared concurrently mid-loop.
However, is there a possibility that the current task_cache_work() execution
hasn't finished yet when the next scan window arrives? For instance, if the
current task work is heavily delayed or preempted by unexpected interrupt,
jiffies could advance past next_scan before the loop completes.
If we move the `work->next = work;` to the very end of task_cache_work(),
would that resolve this issue? By doing so, the existing `work->next == work`
check in task_tick_cache() should fail and no new task work will be submitted.
Please let me know if I'm missing something.
Agree. Since we now solely use visited_cpus to filter CPUs and get_scan_cpumasks()+ cpumask_clear_cpu(cpu, mm->sc_stat.visited_cpus);
+ return 0;
+ }
+
[ ... ]
- for_each_cpu(i, sched_domain_span(sd)) {
- occ = fraction_mm_sched(cpu_rq(i),
- per_cpu_ptr(mm->sc_stat.pcpu_sched, i));
+ for_each_cpu_and(i, sched_domain_span(sd), mm->sc_stat.visited_cpus) {
Does using
for_each_cpu_and(i, sched_domain_span(sd), cpus)
reduce the race window to read mm->sc_stat.pcpu_sched?
has been removed, this inner loop will no longer incorrectly skip those CPUs
in cross-node LLC domains. Thus, the issue previously raised by Sashiko
no longer exists.
[ ... ]
I ran hackbench[1] with above changes on Intel Sapphire Rapids with memory interleave enabled,
and on AMD Milan. There is no much difference on Sapphire Rapids, while a slight regress on
Milan AMD EPYC 9554P 64-Core when NUMA balancing is enabled(but it looks to be within
run-to-run variance) Please double check if this is the case on your AMD server.
And you can add my tag in next version:
Tested-by: Chen Yu <yu.c.chen@xxxxxxxxx>
Data on Milan
./launch.sh compare hackbench base_numab visit_numab
=========================================
Hackbench Comparison: base_numab vs visit_numab
=========================================
MODE GROUPS FDS | base_numab(s) | visit_numab(s) | DIFF(%) | VERDICT
---------- ------ -----+--------------------+--------------------+------------+-----------
process 1 10 | 34.564 ±0.53% | 34.103 ±1.48% | 1.33% | IMPROVED
process 1 16 | 60.556 ±0.52% | 62.296 ±2.11% | -2.87% | REGRESSED
process 1 2 | 2.981 ±0.03% | 3.814 ±21.34% | -27.94% | REGRESSED <--run-to-run variance
process 1 20 | 80.683 ±1.51% | 81.392 ±0.28% | -0.88% | REGRESSED
process 1 4 | 8.446 ±12.50% | 7.185 ±6.86% | 14.93% | IMPROVED
process 1 6 | 11.355 ±1.09% | 11.775 ±0.98% | -3.70% | REGRESSED
process 1 8 | 19.282 ±6.61% | 19.976 ±7.10% | -3.60% | REGRESSED
process 2 10 | 36.729 ±2.08% | 36.899 ±1.25% | -0.46% | REGRESSED
process 2 16 | 63.758 ±0.47% | 63.373 ±0.74% | 0.60% | IMPROVED
process 2 2 | 3.240 ±4.01% | 3.289 ±6.17% | -1.51% | REGRESSED
process 2 20 | 85.519 ±1.97% | 83.396 ±0.26% | 2.48% | IMPROVED
process 2 4 | 8.551 ±3.08% | 9.132 ±9.56% | -6.79% | REGRESSED
process 2 6 | 15.256 ±10.90% | 11.869 ±1.21% | 22.20% | IMPROVED
process 2 8 | 26.729 ±2.24% | 25.681 ±11.10% | 3.92% | IMPROVED
process 4 10 | 39.921 ±1.01% | 39.274 ±0.37% | 1.62% | IMPROVED
process 4 16 | 74.032 ±2.08% | 83.334 ±5.79% | -12.56% | REGRESSED
process 4 2 | 4.339 ±11.59% | 4.367 ±10.33% | -0.65% | REGRESSED
process 4 20 | 120.967 ±0.67% | 124.640 ±3.31% | -3.04% | REGRESSED
process 4 4 | 11.088 ±4.61% | 11.250 ±6.90% | -1.46% | REGRESSED
process 4 6 | 21.023 ±0.77% | 20.639 ±1.40% | 1.83% | IMPROVED
process 4 8 | 31.368 ±0.36% | 30.306 ±0.77% | 3.39% | IMPROVED
process 8 10 | 56.356 ±1.47% | 55.669 ±0.98% | 1.22% | IMPROVED
process 8 16 | 144.752 ±0.84% | 139.409 ±0.61% | 3.69% | IMPROVED
process 8 2 | 5.697 ±4.14% | 5.947 ±5.28% | -4.39% | REGRESSED
process 8 20 | 220.395 ±0.32% | 215.589 ±0.66% | 2.18% | IMPROVED
process 8 4 | 14.589 ±7.09% | 14.148 ±2.56% | 3.02% | IMPROVED
process 8 6 | 23.872 ±1.39% | 24.558 ±4.92% | -2.87% | REGRESSED
process 8 8 | 34.434 ±0.60% | 36.422 ±4.85% | -5.77% | REGRESSED
threads 1 10 | 35.582 ±0.62% | 35.697 ±0.94% | -0.32% | REGRESSED
threads 1 16 | 62.996 ±1.36% | 63.255 ±0.65% | -0.41% | REGRESSED
threads 1 2 | 3.216 ±0.16% | 3.253 ±1.20% | -1.15% | REGRESSED
threads 1 20 | 83.611 ±1.53% | 82.771 ±0.23% | 1.00% | IMPROVED
threads 1 4 | 6.952 ±0.50% | 6.957 ±0.06% | -0.07% | REGRESSED
threads 1 6 | 14.028 ±3.49% | 12.997 ±0.41% | 7.35% | IMPROVED
threads 1 8 | 21.511 ±1.71% | 21.395 ±2.73% | 0.54% | IMPROVED
threads 2 10 | 37.655 ±1.42% | 38.516 ±0.54% | -2.29% | REGRESSED
threads 2 16 | 67.366 ±2.88% | 68.230 ±3.80% | -1.28% | REGRESSED
threads 2 2 | 3.301 ±1.21% | 3.344 ±1.50% | -1.30% | REGRESSED
threads 2 20 | 94.511 ±1.12% | 87.065 ±0.73% | 7.88% | IMPROVED
threads 2 4 | 8.793 ±7.80% | 10.709 ±3.75% | -21.79% | REGRESSED
threads 2 6 | 16.934 ±11.50% | 16.483 ±15.58% | 2.66% | IMPROVED
threads 2 8 | 26.700 ±7.11% | 25.430 ±3.37% | 4.76% | IMPROVED
threads 4 10 | 40.121 ±1.36% | 40.582 ±2.06% | -1.15% | REGRESSED
threads 4 16 | 83.463 ±2.77% | 79.598 ±0.41% | 4.63% | IMPROVED
threads 4 2 | 4.855 ±3.95% | 4.753 ±15.72% | 2.10% | IMPROVED
threads 4 20 | 134.076 ±0.44% | 131.447 ±1.24% | 1.96% | IMPROVED
threads 4 4 | 13.193 ±6.68% | 11.804 ±13.90% | 10.53% | IMPROVED
threads 4 6 | 21.001 ±3.50% | 18.046 ±0.96% | 14.07% | IMPROVED
threads 4 8 | 30.247 ±2.34% | 28.422 ±2.57% | 6.03% | IMPROVED
threads 8 10 | 58.323 ±0.46% | 60.638 ±1.94% | -3.97% | REGRESSED
threads 8 16 | 149.357 ±0.83% | 153.554 ±0.57% | -2.81% | REGRESSED
threads 8 2 | 5.157 ±3.26% | 5.542 ±6.69% | -7.47% | REGRESSED
threads 8 20 | 227.619 ±0.40% | 229.307 ±1.23% | -0.74% | REGRESSED
threads 8 4 | 15.586 ±3.65% | 14.631 ±0.11% | 6.13% | IMPROVED
threads 8 6 | 25.401 ±2.47% | 24.989 ±2.39% | 1.62% | IMPROVED
threads 8 8 | 36.215 ±3.22% | 37.484 ±3.35% | -3.50% | REGRESSED
Note: Values shown as mean ±%stddev (seconds). DIFF based on means.
DIFF(%) = (base_numab - visit_numab) / base_numab * 100
Positive = improvement (visit_numab faster), Negative = regression.
[1] https://github.com/chen-yu-surf/bench_tool.git
Thanks for the testing and the tag!
I believe that the results are likely due to run-to-run variance. Below are my test
results, and I noticed that some test cases regressed on my side while they improved
on yours. For example, the threads 2 2 case showed completely opposite trends between
our environments.
Data AMD EPYC 9654
=========================================
Hackbench comparison
BASE: base_numab_0728
TEST: visit_numab_0728
=========================================
MODE G FD | BASE(s) | TEST(s) | DIFF% | RESULT
------- -- ----+-----------------+-----------------+---------+----------
process 1 10 | 63.856/4.7% | 65.950/0.8% | -3.28% | REGRESSED
process 1 16 | 107.365/1.5% | 106.472/1.0% | 0.83% | IMPROVED
process 1 2 | 9.813/0.7% | 9.215/3.8% | 6.09% | IMPROVED
process 1 20 | 130.314/0.9% | 132.356/0.9% | -1.57% | REGRESSED
process 1 4 | 18.624/2.9% | 19.663/6.3% | -5.58% | REGRESSED
process 1 6 | 33.364/3.5% | 32.145/1.4% | 3.65% | IMPROVED
process 1 8 | 45.797/4.5% | 45.128/10.7% | 1.46% | IMPROVED
process 2 10 | 63.760/1.1% | 64.099/0.4% | -0.53% | REGRESSED
process 2 16 | 107.555/1.3% | 105.975/0.6% | 1.47% | IMPROVED
process 2 2 | 9.995/1.5% | 9.794/3.1% | 2.01% | IMPROVED
process 2 20 | 132.837/0.3% | 133.416/1.5% | -0.44% | REGRESSED
process 2 4 | 19.617/3.5% | 21.275/5.1% | -8.45% | REGRESSED
process 2 6 | 33.962/3.2% | 34.376/2.6% | -1.22% | REGRESSED
process 2 8 | 47.400/4.2% | 49.125/1.2% | -3.64% | REGRESSED
process 4 10 | 63.588/1.6% | 64.254/0.9% | -1.05% | REGRESSED
process 4 16 | 106.697/0.6% | 105.565/0.8% | 1.06% | IMPROVED
process 4 2 | 8.720/4.4% | 9.915/4.9% | -13.70% | REGRESSED
process 4 20 | 132.137/0.9% | 133.264/0.3% | -0.85% | REGRESSED
process 4 4 | 22.398/3.8% | 19.267/4.1% | 13.98% | IMPROVED
process 4 6 | 33.693/2.8% | 35.592/0.7% | -5.64% | REGRESSED
process 4 8 | 50.416/2.5% | 49.572/2.9% | 1.67% | IMPROVED
process 8 10 | 63.858/0.9% | 65.193/0.9% | -2.09% | REGRESSED
process 8 16 | 106.599/1.0% | 106.779/0.6% | -0.17% | REGRESSED
process 8 2 | 9.581/4.7% | 8.549/4.1% | 10.77% | IMPROVED
process 8 20 | 134.607/0.4% | 133.946/1.2% | 0.49% | IMPROVED
process 8 4 | 20.298/2.6% | 22.213/3.6% | -9.43% | REGRESSED
process 8 6 | 36.864/1.3% | 36.373/0.9% | 1.33% | IMPROVED
process 8 8 | 51.013/1.6% | 51.373/0.9% | -0.71% | REGRESSED
threads 1 10 | 64.432/1.4% | 65.835/2.6% | -2.18% | REGRESSED
threads 1 16 | 109.816/3.2% | 99.865/3.9% | 9.06% | IMPROVED
threads 1 2 | 7.246/30.6% | 5.060/0.7% | 30.17% | IMPROVED
threads 1 20 | 116.748/0.6% | 121.446/4.4% | -4.02% | REGRESSED
threads 1 4 | 18.362/20.5% | 11.685/1.5% | 36.36% | IMPROVED
threads 1 6 | 31.788/1.7% | 35.312/4.7% | -11.09% | REGRESSED
threads 1 8 | 44.421/3.6% | 51.299/6.4% | -15.48% | REGRESSED
threads 2 10 | 65.545/4.0% | 70.301/2.3% | -7.26% | REGRESSED
threads 2 16 | 96.461/5.7% | 107.936/1.9% | -11.90% | REGRESSED
threads 2 2 | 5.634/5.5% | 5.472/2.3% | 2.88% | IMPROVED
threads 2 20 | 115.252/2.2% | 134.723/0.5% | -16.89% | REGRESSED
threads 2 4 | 23.177/10.5% | 20.270/9.2% | 12.54% | IMPROVED
threads 2 6 | 40.291/4.0% | 38.460/3.2% | 4.54% | IMPROVED
threads 2 8 | 50.297/6.5% | 49.422/2.8% | 1.74% | IMPROVED
threads 4 10 | 60.081/6.4% | 65.470/2.2% | -8.97% | REGRESSED
threads 4 16 | 104.858/5.6% | 109.031/1.7% | -3.98% | REGRESSED
threads 4 2 | 10.508/4.1% | 10.232/2.7% | 2.63% | IMPROVED
threads 4 20 | 136.411/1.2% | 134.034/1.6% | 1.74% | IMPROVED
threads 4 4 | 23.156/4.8% | 24.758/3.9% | -6.92% | REGRESSED
threads 4 6 | 36.906/3.9% | 40.785/0.6% | -10.51% | REGRESSED
threads 4 8 | 48.687/6.0% | 53.476/4.8% | -9.84% | REGRESSED
threads 8 10 | 64.297/4.5% | 65.892/0.4% | -2.48% | REGRESSED
threads 8 16 | 110.432/0.4% | 109.087/1.6% | 1.22% | IMPROVED
threads 8 2 | 9.147/5.5% | 9.948/7.2% | -8.76% | REGRESSED
threads 8 20 | 138.656/1.1% | 138.635/0.8% | 0.02% | IMPROVED
threads 8 4 | 21.609/4.1% | 24.328/5.5% | -12.58% | REGRESSED
threads 8 6 | 42.429/4.2% | 38.484/2.4% | 9.30% | IMPROVED
threads 8 8 | 52.500/3.0% | 52.328/0.9% | 0.33% | IMPROVED
thanks,
Gengkun