Re: [Question] Sched: Severe scheduling latency (>10s) observed on kernel 6.12 with specific workload

[John Stultz]

On Thu, Apr 9, 2026 at 9:01 PM John Stultz <jstultz@xxxxxxxxxx> wrote:
>
> On Thu, Apr 9, 2026 at 8:31 PM Xuewen Yan <xuewen.yan94@xxxxxxxxx> wrote:
> > On Fri, Apr 10, 2026 at 5:39 AM John Stultz <jstultz@xxxxxxxxxx> wrote:
> >>
> > > So, with this config I think I may have reproduced it on a device
> > > (using android16-6.12).  I've not quite seen 10+ seconds, but I have
> > > seen >2second delays for kworker threads (though usually the max seems
> > > to be around 600ms).
> >
> > Thanks for the detailed update! It’s great to hear that you’ve managed
> > to reproduce the issue on a real device. Even though the latency is
> > around 2 seconds (instead of 10+), that still significantly confirms
> > the problem exists. The difference in magnitude might just be due to
> > specific background load conditions.
>
> So, I believe there is an issue in the android16-6.12 tree where some
> changes from upstream, not in 6.12-stable were backported, and then
> some fixes from 6.12-stable were reverted, and the mix of the two is
> causing problems. I've tried to align the tree more closely to
> 6.12-stable and I'm seeing the results improve, so now its just a
> matter of trying to de-tangle things in the android tree.
>
> So, for the community folks: I don't think upstream or the -stable
> tree is connected to what is being reported.

So unfortuantely, it looks like I'm going to have to eat my words here. :(

In the android16-6.12 tree, the behavior has been isolated down to a
backport of commit 6d71a9c61604 ("sched/fair: Fix EEVDF entity
placement bug causing scheduling lag").
  https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6d71a9c6160479899ee744d2c6d6602a191deb1f

Specifically the `place_entity(cfs_rq, se, 0);` addition in `reweight_entity()`

I was assuming android16-6.12 was missing other related changes
causing the bad behavior, but Xuewen pointed out similar problems
could be seen on android17-6.18. I aligned that tree to the
6.18-stable branch, and could also reproduce it.  Additionally, I
tested with our andorid-mainline branch (as of 6.19 the base) and it
showed the same issue.  So this does "look" to be upstream related.

Removing the place_entity() line added in commit 6d71a9c61604 from
reweight_entity() seems to prevent the behavior.

I've been able to trigger this "bad behavior" on devices using the
rt-app with the configuration[1] Xuewen first provided (putting 10
spinners per cpu on the bottom 4 cpus in a background v1 cpu cgroup).
Then I run `cyclictest -m -t -a --policy=SCHED_OTHER -b 1000000 -D
120s` in the root cgroup, and (usually well) within two minutes I'll
see > 1second delays in cyclictest.

If I remove the cgroup from the rt-app config, the issue doesn't reproduce.

Unfortuantely I've only been able to reproduce this on device, which
requires the android kernel tree. I've installed an old debian11 image
on x86 QEMU to be able to utilize cpu v1 cgroup support, but I haven't
reproduced the issue there, which is a bit confounding.

The issue doesn't immediatley trigger, but usually after a few seconds
of normal behavior I'll start to see cyclictest on one or two of the
cpus start to trip larger 100ms+ latencies until it hits the 1second
boundary.

Late last week I went digging into place_entity() to try to understand
why it was tripping, but wasn't very succesful in narrowing down what
might be going wrong. I did see that the place_entity() call seems to
always be on a non-task se, and its almost always exiting at the `if
(se->rel_deadline)` case. It does go through the lag calculation
conditional, but not always.

Anyway, I'm going to continue digging into this, but I just wanted to
give folks a heads up in case there were any ideas to explore.

thanks
-john


[1] Xuewen's rt-app config:
{
    "tasks" : {
            "t0" : {
                    "instance" : 40,
                    "priority" : 0,
                    "cpus" : [ 0, 1, 2, 3 ],
"taskgroup" : "/background",
                    "loop" : -1,
                    "run" : 200,
                    "sleep" : 50
            }
    }
}