Re: sched/core warning triggers on rcu torture test
From: Paul E. McKenney
Date: Tue Jun 26 2018 - 13:46:33 EST
On Tue, Jun 26, 2018 at 06:32:55PM +0200, Peter Zijlstra wrote:
> On Tue, Jun 26, 2018 at 06:16:04PM +0200, Anna-Maria Gleixner wrote:
> > Hi,
> >
> > during rcu torture tests (TREE04 and TREE07) I noticed, that a
> > WARN_ON_ONCE() in sched core triggers on a recent 4.18-rc2 based
> > kernel (6f0d349d922b ("Merge
> > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net")) as well as
> > on a 4.17.3.
First, I am very glad that I am not the only one running rcutorture! ;-)
> > I'm running the tests on a machine with 144 cores:
> >
> > tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 144 --duration 120 --configs "9*TREE07"
> > tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 144 --duration 120 --configs "18*TREE04"
> >
> >
> > The warning was introduced by commit d84b31313ef8 ("sched/isolation:
> > Offload residual 1Hz scheduler tick").
> >
> >
> > Output looks similar for all tests I did (this one is the output of
> > the 4.18-rc2 based kernel):
> >
> > WARNING: CPU: 11 PID: 906 at kernel/sched/core.c:3138 sched_tick_remote+0xb6/0xc0
>
> That's nohz_full stuff, is that a normal part of rcutorture? In any
> case, is the one housekeeping CPU getting seriously overloaded or
> something?
Yes, nohz_full is a normal part for rcutorture because RCU has to deal
differently with userspace execution in the nohz_full case.
I do see this splat (at least when I don't comment it out), but I
do share my system with others, so I could easily be overloading the
housekeeping vCPUs due to hypervisor preemption. I was intending to
dig into this one once I got done consolidating RCU-bh, RCU-preempt,
and RCU-sched at Linus's behest.
On overloading the housekeeping CPU without outside load, let's look at
TREE04 and TREE07 separately.
TREE04 uses eight CPUs, and seven of them ("nohz_full=1-7") are nohz_full
CPUs, and rcutorture doesn't generate all that large of a callback load.
It looks like all 144 CPUs are used in this case (18*8), though RCU
enforces idle periods in order to test idle/non-idle transitions.
But was there anything else running on the machine at the time?
TREE07 uses 16 CPUs, and eight of them ("nohz_full=2-9") are nohz_full
CPUs. Again, it looks like all 144 CPUs are used (9*8).
I sometimes see this on TASKS03 as well, which uses two CPUs, and one of
them ("nohz_full=1") is a nohz_full CPU.
If your system is otherwise idle, would it make sense to trace context
switches on CPU 0 to see what it is up to? And to do an ftrace_dump()
and turn tracing off when the warning triggers as well?
Thanx, Paul