Re: Intermittent early panic in try_to_wake_up

[Kevin Winchester]

Mike Galbraith wrote:
> On Thu, 2009-10-22 at 20:33 -0300, Kevin Winchester wrote:
> > Hi,
> >
> > A week or two ago I saw a panic on boot in try_to_wake_up, but it was not
> > reproducible and I had not written down any trace information.  This
> > evening I saw it twice more, but then on the third boot things worked fine.
> > This time I copied down the stack trace:
> >
> > try_to_wake_up+0x2e/0x102
> > wake_up_process+0x10/0x12
> > kthread_create+0x88/0x12c
> > ?ksoftirqd+0x00/0xb7
> > cpu_callback+0x42/0x8f
> > ?spawn_ksoftirqd+0x0/0x39
> > spawn_ksoftirqd+0x17/0x39
> > do_one_initcall+0x58/0x147
> >
> > The first time it happened, I remember checking the git logs and it was
> > shortly after:
> >
> > commit f5dc37530ba8a35aae0f7f4f13781d1904f71e94
> > Author: Mike Galbraith <efault@xxxxxx>
> > Date:   Fri Oct 9 08:35:03 2009 +0200
> >
> >     sched: Update the clock of runqueue select_task_rq() selected
> >     
> >     In try_to_wake_up(), we update the runqueue clock, but
> >     select_task_rq() may select a different runqueue than the one we
> >     updated, leaving the new runqueue's clock stale for a bit.
> >     
> >     This patch cures occasional huge latencies reported by latencytop
> >     when coming out of idle on a mostly idle NO_HZ box.
> >     
> >     Signed-off-by: Mike Galbraith <efault@xxxxxx>
> >     Signed-off-by: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx>
> >     LKML-Reference: <1255070103.7639.30.camel@xxxxxxxxxxxxxxxx>
> >     Signed-off-by: Ingo Molnar <mingo@xxxxxxx>
> >
> >
> > ...so perhaps that has something to do with it.
>
> I don't think that's very likely.  Box did explode near my grubby
> fingerprints though.
>
> > Config below.  Any help would be appreciated.
>
> Building with your config, try_to_wake_up+0x2e is around..
>
> (gdb) list *try_to_wake_up+0x2e
> 0xffffffff81029107 is in try_to_wake_up (kernel/sched.c:2324).
> 2319            this_cpu = get_cpu();
> 2320
> 2321            smp_wmb();
> 2322            rq = orig_rq = task_rq_lock(p, &flags);
> 2323            update_rq_clock(rq);
> 2324            if (!(p->state & state))
> 2325                    goto out;
> 2326
> 2327            if (p->se.on_rq)
> 2328                    goto out_running;
>
> I don't see how any of that can explode without something very bad
> having happened to ksoftirqd before we tried to wake it.

I thought this problem had solved itself, but I've hit it again three times
in the last few days.

I've expanded the CC list a little (based on get_maintainer for
kernel/softirq.c, since that seems to be involved somehow, and Rafael since
this definitely seems to be a regression), to see if anyone else has any
ideas.

--
Kevin Winchester

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/