Re: frequent lockups in 3.18rc4

From: Chris Mason
Date: Wed Dec 03 2014 - 15:00:45 EST

On Wed, Dec 3, 2014 at 2:00 PM, Dave Jones <davej@xxxxxxxxxx> wrote:
On Wed, Dec 03, 2014 at 10:45:57AM -0800, Linus Torvalds wrote:
> On Wed, Dec 3, 2014 at 10:41 AM, Dave Jones <davej@xxxxxxxxxx> wrote:
> >
> > I've been stuck on this kernel for a few days now trying to prove it
> > good/bad one way or the other, and I'm leaning towards good, given
> > that it recovers, even though the traces look similar.
> Ugh. But this does *not* happen with 3.16, right? Even the non-fatal case?

correct. at least not in any of the runs that I did to date.

> If so, I'd be inclined to call it "bad". But there might well be two
> bugs: one that makes that NMI watchdog trigger, and another one that
> then makes it be a hard lockup. I'd think it would be good to figure
> out the "NMI watchdog starts triggering" one first, though.

I think you're right.

So right after sending my last mail, I rebooted, and restarted the run
on the same kernel again.

As I was writing this mail, this happened.

[ 524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c178:20182]

and that's all that made it over the console. I couldn't log in via ssh,
and thought "ah-ha, so it IS bad". I walked over to reboot it, and
found I could actually log in on the console. check out this dmesg..

[ 503.683055] Clocksource tsc unstable (delta = -95946009388 ns)
[ 503.692038] Switched to clocksource hpet
[ 524.420897] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [trinity-c178:20182]

Neat. We often see switching to hpet on boxes as they are diving into softlockup pain, but it's not usually before the softlockups.

Are you configured for CONFIG_NOHZ_FULL?

I'd love to blame the only commit to kernel/smp.c between 3.16 and 3.17

commit 478850160636c4f0b2558451df0e42f8c5a10939
Author: Frederic Weisbecker <fweisbec@xxxxxxxxx>
Date: Thu May 8 01:37:48 2014 +0200

irq_work: Implement remote queueing

You've also mentioned a few times where messages stopped hitting the console?

commit 5874af2003b1aaaa053128d655710140e3187226
Author: Jan Kara <jack@xxxxxxx>
Date: Wed Aug 6 16:09:10 2014 -0700

printk: enable interrupts before calling console_trylock_for_printk()


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at