[PATCH v2] sched/pelt: Skip redundant decay and divisions when sched_avg is zero

From: Chris Wiseman

Date: Tue Nov 17 2026 - 07:50:20 EST


Commit d040e0734fb3 ("schied/fair: Skip calculating @contrib without
load") skipped __accumulate_pelt_segments() inside accumulate_sum() when
load is zero, noting that ___update_load_sum() is called with load == 0
45x more often than with non-zero load.

However, when load is zero and sa->load_sum, sa->runnable_sum, and
sa->util_sum have already decayed to zero, accumulate_sum() still
executes decay_load(0, periods) three times whenever a 1024us PELT
period boundary is crossed, and ___update_load_avg() still executes
three divisions of zero (0 / divider).

In addition, update_irq_load_avg(rq, 0) unconditionally invokes
cap_scale() twice and a second ___update_load_sum(rq->clock,
&rq->avg_irq, 1, 1, 1) call even when running == 0. Because the first
___update_load_sum() call already advanced last_update_time to
rq->clock & ~1023ULL, the second call sees delta < 1024 and always
returns 0 after redundant clock and capacity scaling arithmetic.

In accumulate_sum(), when !load and !(sa->load_sum | sa->runnable_sum |
sa->util_sum), update sa->period_contrib and return periods before
Step 1. In ___update_load_avg(), set *_avg to 0 instead of dividing when
the corresponding *_sum is 0. In update_irq_load_avg(), skip the
capacity scaling and second ___update_load_sum() call when !running.

Evaluating update_other_load_avgs() over 1,000,000 randomized steps
confirms bit-for-bit identical sched_avg state and return values across
all fields, while 20,000,000-iteration microbenchmarks on x86_64 and
arm64 show:

Regime (1ms blocked-avg updates) Base (cyc) Patched (cyc) Delta
------------------------------------ ---------- ------------- ------
x86_64 Idle RT/DL/HW/IRQ 78.53 32.82 -58.2%
x86_64 100% active RT+DL+HW+IRQ 116.01 117.63 +1.4%
arm64 Idle RT/DL/HW/IRQ 77.78 37.91 -51.3%
arm64 100% active RT+DL+HW+IRQ 120.40 113.52 -5.7%

Suggested-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
Signed-off-by: Chris Wiseman <chriswiseman@xxxxxxxxxx>
---
Changes in v2:
- Move the zero-sum fast path into accumulate_sum() before Step 1 and
still return periods (Vincent Guittot).
- Avoid dividing in ___update_load_avg() when *_sum is 0, dropping the
sa->load_avg check and the kernel/sched/fair.c change (Vincent Guittot).
- Add arm64 cycle measurements alongside x86_64.
- v1: https://lore.kernel.org/r/18e837a02ddbadb6.3137322e32332e3235332e313736.1b3220.0000sendgmr@xxxxxxxxxx

kernel/sched/pelt.c | 25 ++++++++++++++++++-------
1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c
index 779eb58a4261..7308fb28b0fa 100644
--- a/kernel/sched/pelt.c
+++ b/kernel/sched/pelt.c
@@ -109,6 +109,11 @@ accumulate_sum(u64 delta, struct sched_avg *sa,
delta += sa->period_contrib;
periods = delta / 1024; /* A period is 1024us (~1ms) */

+ if (!load && !(sa->load_sum | sa->runnable_sum | sa->util_sum)) {
+ sa->period_contrib = delta % 1024;
+ return periods;
+ }
+
/*
* Step 1: decay old *_sum if we crossed period boundaries.
*/
@@ -262,9 +267,11 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)
/*
* Step 2: update *_avg.
*/
- sa->load_avg = div_u64(load * sa->load_sum, divider);
- sa->runnable_avg = div_u64(sa->runnable_sum, divider);
- WRITE_ONCE(sa->util_avg, sa->util_sum / divider);
+ sa->load_avg = sa->load_sum ? div_u64(load * sa->load_sum, divider) : 0;
+ sa->runnable_avg = sa->runnable_sum ?
+ div_u64(sa->runnable_sum, divider) : 0;
+ WRITE_ONCE(sa->util_avg,
+ sa->util_sum ? sa->util_sum / divider : 0);
}

/*
@@ -437,8 +444,10 @@ int update_irq_load_avg(struct rq *rq, u64 running)
* clock_task. Instead we directly scale the running time to
* reflect the real amount of computation
*/
- running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
- running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
+ if (running) {
+ running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
+ running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
+ }

/*
* We know the time that has been used by interrupt since last update
@@ -455,10 +464,12 @@ int update_irq_load_avg(struct rq *rq, u64 running)
0,
0,
0);
- ret += ___update_load_sum(rq->clock, &rq->avg_irq,
- 1,
- 1,
- 1);
+ if (running) {
+ ret += ___update_load_sum(rq->clock, &rq->avg_irq,
+ 1,
+ 1,
+ 1);
+ }

if (ret) {
___update_load_avg(&rq->avg_irq, 1);
--
2.47.0