Re: [RFC PATCH] sched/fair: Address avruntime drift caused by entity_lag()

[Vincent Guittot]

On Tue, 7 Jul 2026 at 14:51, chenjinghuang <chenjinghuang2@xxxxxxxxxx> wrote:
>
> On 7/7/2026 12:03 AM, Vincent Guittot wrote:
> > On Fri, 3 Jul 2026 at 11:50, Chen Jinghuang <chenjinghuang2@xxxxxxxxxx> wrote:
> >>
> >> While running EEVDF-related test cases, we observed an anomalous
> >> phenomenon where the vruntime of a continuously running scheduling
> >> entity (se) exhibits an overall downward trend.
> >>
> >> The test topology and observed behavior are as follows:
> >>
> >> CPU 0                              CPU 1 / Other CPUs
> >> +----------------------------+     +----------------------------+
> >> | cgroup A        cgroup B   |     | cgroup A        cgroup B   |
> >> |  |- A1 (Run)     |- B1     |     |  |- A2, A3...     |- B2, B3...|
> >> |                    (50us S)|     |    (Frequent sleep/wake)   |
> >> |                    (10us W)|     |                            |
> >> +----------------------------+     +----------------------------+
> >>        |
> >>        v
> >> A2/A3 frequent sleep/wake -> Frequent reweight of gse(A) on CPU 0
> >>   -> Triggers vlag clamping -> Unidirectional avruntime drift
> >>   -> gse->vruntime drops abnormally (while gse weight remains stable)
> >>
> >> According to function_graph traces, frequent sleep/wake cycles of
> >> sibling tasks within the same cgroup cause the running task to repeatedly
> >> trigger update_cfs_cgroup() -> reweight_entity().
> >>
> >> The root cause appears to point to the entity_lag() boundary limit
> >> introduced in `reweight_entity()` to prevent overflows. During repeated
> >> weight scaling, this limit introduces an asymmetric cumulative error.
> >> Connsider an example here where se->vruntime > avruntime:
> >>
> >> 1. Downscale:
> >>    se->vlag = entity_lag(cfs_rq, se, avruntime);
> >
> > The clamp applied in entity_lag() should "in theory never" happen.
> >
>
> However, I observed that vlag does indeed get clamped in entity_lag() on
> CPU 0 in the test case

Ok, we have to understand why the lag exceeds the slice duration

>
> > Do you have more details to share about your setup?
> >
>
> Here is the exact layout of the cgroups:
> - cgroup A: Besides task A1 (which is bound to CPU 0), there are 20 other
>   tasks running on the remaining CPUs (excluding CPU 0) that frequently
>   sleep and wake up. This setup is designed to continuously trigger weight
>   adjustments for A1.
> - cgroup B: There is only a single task B1 bound to CPU 0, which sleeps for
>   long periods and wakes up briefly to compete with A1.(Please note that my
>   previous diagram mistakenly showed other tasks running under cgroup B on
>   other CPUs; that was an error on my part).
>
> > Which kernel version do you use ? A number of changes have been queued
> > recently that should ensure that the vlag never goes above the
> > theoretical limit.
> >
>
> I am running the mainline kernel, specifically on the v7.2-rc2 tag.

Thanks, I will try to reproduce it.
There are a number of additional patches in tip/sched/core that target
shorter slice preemption but also ensure that the lag of other tasks
"never" goes above the limit.

>
> > How many CPUs are in the system ?
> > Is there other tasks running in the cgroups ?
> >
>
> The system has 32 CPUs in total.
>
> > Could you try to run without RUN_TO_PARITY ?
> > echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features
> >
>
> I tried disabling RUN_TO_PARITY, but the issue persists—vlag
> still exceeds the limit and triggers the clamp.

This is even weirder because the protection is only 700us compared
with a default slice of 2.8ms

>
> > Do you set the share of task groups to particular values?
> > Do you change the slice of the tasks ?
> >
>
> No, I haven't set specific shares for the task groups (they are all at the
> default value of 1024), nor have I modified the default task slices.

Thanks for the details

>
> > Note that even without clamping, part of the vlag is lost in div64_long
> >
> >>    se->vlag = div64_long(se->vlag * old_weight, weight);
> >>    se->vruntime = avruntime - se->vlag;
> >>
> >>    If the calculated vlag exceeds the limit boundary, it is clamped.
> >>    This causes the scaled vlag to be bigger than its theoretical value,
> >>    making the resulting se->vruntime smaller than it should be (forced
> >>    closer to avruntime).
> >>
> >> 2. Upscale:
> >>    Because se->vruntime already carries the clamping error from the
> >>    previous step, it implicitly shifts the subsequent global avruntime
> >>    computation trajectory. When the weight is restored, se->vruntime
> >>    deviates from its original true value, preventing the true vlag from
> >>    being reconstructed and leading to a unidirectional avruntime drift.
> >>
> >> Under frequent cgroup reweighting, this asymmetric truncation error
> >> accumulates, causing a unidirectional downward drift of vruntime.
> >>
> >> Regarding this vlag limit mechanism in reweight_entity(), we would like
> >> to raise a few questions for discussion:
> >>
> >> Question 1: Is this deviation of se->vruntime expected?
> >> For a currently running entity (curr se), due to RUN_TO_PARITY, it
> >> typically operates in a state where se->vruntime > avruntime. During
> >> reweighting, the limit forces its vruntime downward (closer to avruntime),
> >> while attributes like deadline remain unaffected by the limit.
> >>
> >> Question 2: Is the unidirectional drift of avruntime reasonable?
> >> Since avruntime = avg_vruntime(), the deviation of se->vruntime
> >> before and after reweighting (while its weight remains unchanged) shifts
> >> the global avruntime downward. According to EEVDF core principles,
> >> avruntime should remain invariant across weight changes.
> >> (https://lore.kernel.org/lkml/20260415211149.2658910-1-jstultz@xxxxxxxxxx/),
> >>
> >> Question 3: Is the reduction of avruntime fair to other entities?
> >> A non-functional reduction in global avruntime passively decreases the
> >> se->vlag of other queued entities on the runqueue, which practically
> >> worsens the starvation of those tasks.
> >>
> >> Question 4: Does this lead to complete preemption suppression?
> >> In entity_tick -> update_deadline, because curr->vruntime is abnormally
> >> pulled down by the clamping error while its deadline is unchanged, the
> >> condition vruntime < deadline can be continuously satisfied at every
> >> tick. This skips resched_curr(), suppressing preemption until the
> >> frequent reweighting stops or curr's vlag falls back within the limit.
> >>
> >> While entity_lag() was originally introduced to prevent overflow-induced
> >> NULL pointer dereferences, given its significant impact on EEVDF fairness,
> >> should the necessity and implementation of this limit be reconsidered?
> >>
> >> Analyzed-by: Hui Tang <tanghui20@xxxxxxxxxx>
> >> Analyzed-by: Zhang Qiao <zhangqiao22@xxxxxxxxxx>
> >> Analyzed-by: Zicheng Qu <quzicheng@xxxxxxxxxx>
> >> Analyzed-by: Chen Jinghuang <chenjinghuang2@xxxxxxxxxx>
> >
>