Re: [PATCH] sched/pelt: Skip redundant decay and divisions when sched_avg is zero
From: Vincent Guittot
Date: Mon Nov 16 2026 - 06:30:20 EST
On Sun, 15 Nov 2026 at 13:26, Chris Wiseman <chriswiseman@xxxxxxxxxx> wrote:
>
> Commit d040e0734fb3 ("schied/fair: Skip calculating @contrib without
> load") skipped __accumulate_pelt_segments() inside accumulate_sum() when
> load is zero, noting that ___update_load_sum() is called with load == 0
> 45x more often than with non-zero load.
>
> However, when load is zero and sa->load_sum, sa->runnable_sum,
> sa->util_sum, and sa->load_avg have already decayed to zero,
> accumulate_sum() still executes decay_load(0, periods) three times and
> returns periods > 0 whenever a 1024us PELT period boundary is crossed.
> Consequently, ___update_load_sum() returns 1 and the caller invokes
> ___update_load_avg(), performing three divisions of zero (0 / divider)
> and reporting a false decay to callers such as
> __sched_balance_update_blocked_averages() -> cpufreq_update_util()
> (wired up in commit bef69dd87828 ("sched/cpufreq: Move the
> cfs_rq_util_change() call to cpufreq_update_util()")).
>
> In addition, update_irq_load_avg(rq, 0) unconditionally invokes a second
> ___update_load_sum(rq->clock, &rq->avg_irq, 1, 1, 1) call even when
> running == 0. Because the first ___update_load_sum() call already
> advanced last_update_time to rq->clock & ~1023ULL, the second call sees
> delta < 1024 and always returns 0 after redundant clock and capacity
> scaling arithmetic.
>
> When !load and unlikely(!(sa->load_sum | sa->runnable_sum |
> sa->util_sum | sa->load_avg)), update sa->period_contrib and return 0
> directly from ___update_load_sum(), while still flushing any pending
> rate-limited cfs_rq->tg_load_avg_contrib in __update_blocked_fair().
> In update_irq_load_avg(), skip the capacity scaling and second
> ___update_load_sum() call when !running.
>
> Evaluating update_other_load_avgs() over 1,000,000 randomized steps
> confirms bit-for-bit identical sched_avg state across all fields, while
> 10,000,000-iteration microbenchmarks on x86_64 show:
>
> Regime (1ms blocked-avg updates) Base (cyc) Patched (cyc) Delta
> ------------------------------------ ---------- ------------- ------
> Idle RT/DL/HW/IRQ 72.53 18.33 -74.7%
> 100% active RT+DL+HW+IRQ (worst) 99.21 99.68 +0.5%
>
> Spurious decay returns from update_other_load_avgs() when RT, DL, HW,
> and IRQ averages are already zero drop from 100% of 1ms updates to 0.
>
> Signed-off-by: Chris Wiseman <chriswiseman@xxxxxxxxxx>
> ---
> kernel/sched/fair.c | 2 ++
> kernel/sched/pelt.c | 28 +++++++++++++++++++++-------
> 2 files changed, 23 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index f77e722..83d3928 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -11538,6 +11538,8 @@ static bool __update_blocked_fair(struct rq *rq, bool *done)
>
> if (cfs_rq == &rq->cfs)
> decayed = true;
> + } else if (cfs_rq->tg_load_avg_contrib) {
> + update_tg_load_avg(cfs_rq);
Normally the cfs_rq is removed once it has been decayed which
includes tg_load_avg_contrib so they should not be updated
This is because of your change which prevents updating update_tg_load_avg
> }
>
> /* Propagate pending load changes to the parent, if any: */
> diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c
> index 779eb58..935b8a4 100644
> --- a/kernel/sched/pelt.c
> +++ b/kernel/sched/pelt.c
> @@ -214,8 +214,18 @@ ___update_load_sum(u64 now, struct sched_avg *sa,
> *
> * Also see the comment in accumulate_sum().
> */
> - if (!load)
> + if (!load) {
> + /*
> + * Skip decay and divisions when sums are already zero; check
> + * load_avg for newly forked tasks (init_entity_runnable_average).
> + */
> + if (unlikely(!(sa->load_sum | sa->runnable_sum | sa->util_sum |
> + sa->load_avg))) {
why load_avg ? when load_sum == 0, load_avg must be 0
> + sa->period_contrib = (delta + sa->period_contrib) % 1024;
> + return 0;
I would prefer that this to be included in accumulate_sum before step 1
Wouldn't it be enough to skip the decay_load and
__accumulate_pelt_segments in accumulate_sum but still return the
decay. And in ___update_load_avg() set the *_avg to 0 instead of doing
the div when the *_sum is 0
> + }
> runnable = running = 0;
> + }
>
> /*
> * Now we know we crossed measurement unit boundaries. The *_avg
> @@ -437,8 +447,10 @@ int update_irq_load_avg(struct rq *rq, u64 running)
> * clock_task. Instead we directly scale the running time to
> * reflect the real amount of computation
> */
> - running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
> - running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
> + if (running) {
> + running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
> + running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
> + }
fair enough
>
> /*
> * We know the time that has been used by interrupt since last update
> @@ -455,10 +467,12 @@ int update_irq_load_avg(struct rq *rq, u64 running)
> 0,
> 0,
> 0);
> - ret += ___update_load_sum(rq->clock, &rq->avg_irq,
> - 1,
> - 1,
> - 1);
> + if (running) {
> + ret += ___update_load_sum(rq->clock, &rq->avg_irq,
> + 1,
> + 1,
> + 1);
> + }
fair enough
>
> if (ret) {
> ___update_load_avg(&rq->avg_irq, 1);
> --
> 2.52.0.rc1