Re: [PATCH v2] sched/pelt: Skip redundant decay and divisions when sched_avg is zero
From: Vincent Guittot
Date: Tue Nov 17 2026 - 09:42:12 EST
On Tue, 17 Nov 2026 at 13:50, Chris Wiseman <chriswiseman@xxxxxxxxxx> wrote:
>
> Commit d040e0734fb3 ("schied/fair: Skip calculating @contrib without
> load") skipped __accumulate_pelt_segments() inside accumulate_sum() when
> load is zero, noting that ___update_load_sum() is called with load == 0
> 45x more often than with non-zero load.
>
> However, when load is zero and sa->load_sum, sa->runnable_sum, and
> sa->util_sum have already decayed to zero, accumulate_sum() still
> executes decay_load(0, periods) three times whenever a 1024us PELT
> period boundary is crossed, and ___update_load_avg() still executes
> three divisions of zero (0 / divider).
>
> In addition, update_irq_load_avg(rq, 0) unconditionally invokes
> cap_scale() twice and a second ___update_load_sum(rq->clock,
> &rq->avg_irq, 1, 1, 1) call even when running == 0. Because the first
> ___update_load_sum() call already advanced last_update_time to
> rq->clock & ~1023ULL, the second call sees delta < 1024 and always
> returns 0 after redundant clock and capacity scaling arithmetic.
>
> In accumulate_sum(), when !load and !(sa->load_sum | sa->runnable_sum |
> sa->util_sum), update sa->period_contrib and return periods before
> Step 1. In ___update_load_avg(), set *_avg to 0 instead of dividing when
> the corresponding *_sum is 0. In update_irq_load_avg(), skip the
> capacity scaling and second ___update_load_sum() call when !running.
>
> Evaluating update_other_load_avgs() over 1,000,000 randomized steps
> confirms bit-for-bit identical sched_avg state and return values across
> all fields, while 20,000,000-iteration microbenchmarks on x86_64 and
> arm64 show:
>
> Regime (1ms blocked-avg updates) Base (cyc) Patched (cyc) Delta
> ------------------------------------ ---------- ------------- ------
> x86_64 Idle RT/DL/HW/IRQ 78.53 32.82 -58.2%
> x86_64 100% active RT+DL+HW+IRQ 116.01 117.63 +1.4%
> arm64 Idle RT/DL/HW/IRQ 77.78 37.91 -51.3%
> arm64 100% active RT+DL+HW+IRQ 120.40 113.52 -5.7%
>
> Suggested-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
FWIW, you can remove the suggested-by. I only commented on your v1 but
it didn't deeply change what you proposed.
> Signed-off-by: Chris Wiseman <chriswiseman@xxxxxxxxxx>
Reviewed-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
> ---
> Changes in v2:
> - Move the zero-sum fast path into accumulate_sum() before Step 1 and
> still return periods (Vincent Guittot).
> - Avoid dividing in ___update_load_avg() when *_sum is 0, dropping the
> sa->load_avg check and the kernel/sched/fair.c change (Vincent Guittot).
> - Add arm64 cycle measurements alongside x86_64.
> - v1: https://lore.kernel.org/r/18e837a02ddbadb6.3137322e32332e3235332e313736.1b3220.0000sendgmr@xxxxxxxxxx
>
> kernel/sched/pelt.c | 25 ++++++++++++++++++-------
> 1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c
> index 779eb58a4261..7308fb28b0fa 100644
> --- a/kernel/sched/pelt.c
> +++ b/kernel/sched/pelt.c
> @@ -109,6 +109,11 @@ accumulate_sum(u64 delta, struct sched_avg *sa,
> delta += sa->period_contrib;
> periods = delta / 1024; /* A period is 1024us (~1ms) */
>
> + if (!load && !(sa->load_sum | sa->runnable_sum | sa->util_sum)) {
> + sa->period_contrib = delta % 1024;
> + return periods;
> + }
> +
> /*
> * Step 1: decay old *_sum if we crossed period boundaries.
> */
> @@ -262,9 +267,11 @@ ___update_load_avg(struct sched_avg *sa, unsigned long load)
> /*
> * Step 2: update *_avg.
> */
> - sa->load_avg = div_u64(load * sa->load_sum, divider);
> - sa->runnable_avg = div_u64(sa->runnable_sum, divider);
> - WRITE_ONCE(sa->util_avg, sa->util_sum / divider);
> + sa->load_avg = sa->load_sum ? div_u64(load * sa->load_sum, divider) : 0;
> + sa->runnable_avg = sa->runnable_sum ?
> + div_u64(sa->runnable_sum, divider) : 0;
> + WRITE_ONCE(sa->util_avg,
> + sa->util_sum ? sa->util_sum / divider : 0);
> }
>
> /*
> @@ -437,8 +444,10 @@ int update_irq_load_avg(struct rq *rq, u64 running)
> * clock_task. Instead we directly scale the running time to
> * reflect the real amount of computation
> */
> - running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
> - running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
> + if (running) {
> + running = cap_scale(running, arch_scale_freq_capacity(cpu_of(rq)));
> + running = cap_scale(running, arch_scale_cpu_capacity(cpu_of(rq)));
> + }
>
> /*
> * We know the time that has been used by interrupt since last update
> @@ -455,10 +464,12 @@ int update_irq_load_avg(struct rq *rq, u64 running)
> 0,
> 0,
> 0);
> - ret += ___update_load_sum(rq->clock, &rq->avg_irq,
> - 1,
> - 1,
> - 1);
> + if (running) {
> + ret += ___update_load_sum(rq->clock, &rq->avg_irq,
> + 1,
> + 1,
> + 1);
> + }
>
> if (ret) {
> ___update_load_avg(&rq->avg_irq, 1);
> --
> 2.47.0