Re: [RFC PATCH] sched/fair: Address avruntime drift caused by entity_lag()

[Vincent Guittot]

On Tue, 7 Jul 2026 at 16:52, Vincent Guittot <vincent.guittot@xxxxxxxxxx> wrote:
>
> On Tue, 7 Jul 2026 at 14:51, chenjinghuang <chenjinghuang2@xxxxxxxxxx> wrote:
> >
> > On 7/7/2026 12:03 AM, Vincent Guittot wrote:
> > > On Fri, 3 Jul 2026 at 11:50, Chen Jinghuang <chenjinghuang2@xxxxxxxxxx> wrote:
> > >>
> > >> While running EEVDF-related test cases, we observed an anomalous
> > >> phenomenon where the vruntime of a continuously running scheduling
> > >> entity (se) exhibits an overall downward trend.
> > >>
> > >> The test topology and observed behavior are as follows:
> > >>
> > >> CPU 0                              CPU 1 / Other CPUs
> > >> +----------------------------+     +----------------------------+
> > >> | cgroup A        cgroup B   |     | cgroup A        cgroup B   |
> > >> |  |- A1 (Run)     |- B1     |     |  |- A2, A3...     |- B2, B3...|
> > >> |                    (50us S)|     |    (Frequent sleep/wake)   |
> > >> |                    (10us W)|     |                            |
> > >> +----------------------------+     +----------------------------+
> > >>        |
> > >>        v
> > >> A2/A3 frequent sleep/wake -> Frequent reweight of gse(A) on CPU 0
> > >>   -> Triggers vlag clamping -> Unidirectional avruntime drift
> > >>   -> gse->vruntime drops abnormally (while gse weight remains stable)
> > >>
> > >> According to function_graph traces, frequent sleep/wake cycles of
> > >> sibling tasks within the same cgroup cause the running task to repeatedly
> > >> trigger update_cfs_cgroup() -> reweight_entity().
> > >>
> > >> The root cause appears to point to the entity_lag() boundary limit
> > >> introduced in `reweight_entity()` to prevent overflows. During repeated
> > >> weight scaling, this limit introduces an asymmetric cumulative error.
> > >> Connsider an example here where se->vruntime > avruntime:
> > >>
> > >> 1. Downscale:
> > >>    se->vlag = entity_lag(cfs_rq, se, avruntime);
> > >
> > > The clamp applied in entity_lag() should "in theory never" happen.
> > >
> >
> > However, I observed that vlag does indeed get clamped in entity_lag() on
> > CPU 0 in the test case
>
> Ok, we have to understand why the lag exceeds the slice duration
>
> >
> > > Do you have more details to share about your setup?
> > >
> >
> > Here is the exact layout of the cgroups:
> > - cgroup A: Besides task A1 (which is bound to CPU 0), there are 20 other
> >   tasks running on the remaining CPUs (excluding CPU 0) that frequently
> >   sleep and wake up. This setup is designed to continuously trigger weight
> >   adjustments for A1.
> > - cgroup B: There is only a single task B1 bound to CPU 0, which sleeps for
> >   long periods and wakes up briefly to compete with A1.(Please note that my

If task B1 sleeps for long periods and wakes up briefly, the vlag of
gse(A) is null most of the time and will hardly ever exceed the clamp
value. I can see gse(A) reweight on CPU0 but vlag is 0 most of the
time.

If I increase the runtime of task B1, I see some negative vlag values
for gse(A) but never values outside the clamp. How do you check that
vlag is outside the clamp value ?

I will continue trying to reproduce your problem, but I'm probably
missing something in the setup.

> >   previous diagram mistakenly showed other tasks running under cgroup B on
> >   other CPUs; that was an error on my part).
> >
> > > Which kernel version do you use ? A number of changes have been queued
> > > recently that should ensure that the vlag never goes above the
> > > theoretical limit.
> > >
> >
> > I am running the mainline kernel, specifically on the v7.2-rc2 tag.
>
> Thanks, I will try to reproduce it.
> There are a number of additional patches in tip/sched/core that target
> shorter slice preemption but also ensure that the lag of other tasks
> "never" goes above the limit.
>
> >
> > > How many CPUs are in the system ?
> > > Is there other tasks running in the cgroups ?
> > >
> >
> > The system has 32 CPUs in total.
> >
> > > Could you try to run without RUN_TO_PARITY ?
> > > echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features
> > >
> >
> > I tried disabling RUN_TO_PARITY, but the issue persists—vlag
> > still exceeds the limit and triggers the clamp.
>
> This is even weirder because the protection is only 700us compared
> with a default slice of 2.8ms
>
> >
> > > Do you set the share of task groups to particular values?
> > > Do you change the slice of the tasks ?
> > >
> >
> > No, I haven't set specific shares for the task groups (they are all at the
> > default value of 1024), nor have I modified the default task slices.
>
> Thanks for the details
>
> >
> > > Note that even without clamping, part of the vlag is lost in div64_long
> > >
> > >>    se->vlag = div64_long(se->vlag * old_weight, weight);
> > >>    se->vruntime = avruntime - se->vlag;
> > >>
> > >>    If the calculated vlag exceeds the limit boundary, it is clamped.
> > >>    This causes the scaled vlag to be bigger than its theoretical value,
> > >>    making the resulting se->vruntime smaller than it should be (forced
> > >>    closer to avruntime).
> > >>
> > >> 2. Upscale:
> > >>    Because se->vruntime already carries the clamping error from the
> > >>    previous step, it implicitly shifts the subsequent global avruntime
> > >>    computation trajectory. When the weight is restored, se->vruntime
> > >>    deviates from its original true value, preventing the true vlag from
> > >>    being reconstructed and leading to a unidirectional avruntime drift.
> > >>
> > >> Under frequent cgroup reweighting, this asymmetric truncation error
> > >> accumulates, causing a unidirectional downward drift of vruntime.
> > >>
> > >> Regarding this vlag limit mechanism in reweight_entity(), we would like
> > >> to raise a few questions for discussion:
> > >>
> > >> Question 1: Is this deviation of se->vruntime expected?
> > >> For a currently running entity (curr se), due to RUN_TO_PARITY, it
> > >> typically operates in a state where se->vruntime > avruntime. During
> > >> reweighting, the limit forces its vruntime downward (closer to avruntime),
> > >> while attributes like deadline remain unaffected by the limit.
> > >>
> > >> Question 2: Is the unidirectional drift of avruntime reasonable?
> > >> Since avruntime = avg_vruntime(), the deviation of se->vruntime
> > >> before and after reweighting (while its weight remains unchanged) shifts
> > >> the global avruntime downward. According to EEVDF core principles,
> > >> avruntime should remain invariant across weight changes.
> > >> (https://lore.kernel.org/lkml/20260415211149.2658910-1-jstultz@xxxxxxxxxx/),
> > >>
> > >> Question 3: Is the reduction of avruntime fair to other entities?
> > >> A non-functional reduction in global avruntime passively decreases the
> > >> se->vlag of other queued entities on the runqueue, which practically
> > >> worsens the starvation of those tasks.
> > >>
> > >> Question 4: Does this lead to complete preemption suppression?
> > >> In entity_tick -> update_deadline, because curr->vruntime is abnormally
> > >> pulled down by the clamping error while its deadline is unchanged, the
> > >> condition vruntime < deadline can be continuously satisfied at every
> > >> tick. This skips resched_curr(), suppressing preemption until the
> > >> frequent reweighting stops or curr's vlag falls back within the limit.
> > >>
> > >> While entity_lag() was originally introduced to prevent overflow-induced
> > >> NULL pointer dereferences, given its significant impact on EEVDF fairness,
> > >> should the necessity and implementation of this limit be reconsidered?
> > >>
> > >> Analyzed-by: Hui Tang <tanghui20@xxxxxxxxxx>
> > >> Analyzed-by: Zhang Qiao <zhangqiao22@xxxxxxxxxx>
> > >> Analyzed-by: Zicheng Qu <quzicheng@xxxxxxxxxx>
> > >> Analyzed-by: Chen Jinghuang <chenjinghuang2@xxxxxxxxxx>
> > >
> >