Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC

From: Fengguang Wu
Date: Sun Oct 06 2013 - 22:11:40 EST

Next message: Felipe Contreras: "Re: [PATCH] acpi: update win8 OSI blacklist"
Previous message: Felipe Contreras: "[PATCH] acpi: add missing win8 OSI comment to blacklist"
In reply to: Linus Torvalds: "Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC"
Next in thread: Fengguang Wu: "Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote:
> On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu <fengguang.wu@xxxxxxxxx> wrote:
> >
> > I got the below dmesg and the first bad commit is commit cf39c8e5352b:
> > Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
>
> Ugh. How reliable is the double fault? Because bisecting it to the
> merge that didn't even have any conflicts in it as far as I can
> remember means that there's something really subtle going on wrt some
> semantic conflict or other. Or, alternatively, it means that the
> bisect failed because the double fault isn't 100% reliable..

Oops, it's not a reliable bisect...

The "first" bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs
and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the
bad boots has "double fault:" in its dmesg.

Looking into all the 3530 bad dmesgs, I find all kinds of bug messages:

$ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort -nr

3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
3047 BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
3046 Kernel panic - not syncing: Fatal exception in interrupt
2969 BUG: kernel boot oops
374 BUG: kernel test oops
255 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
167 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011!
167 invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
148 Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
48 INFO: lockdep is turned off.
43 BUG: unable to handle kernel
33 BUG: kernel boot crashed
30 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20
27 general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
13 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
13 BUG: unable to handle kernel NULL pointer dereference at (null)
12 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
11 Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
9 INFO: trying to register non-static key.
9 BUG: scheduling while atomic: init/136/0x10000002
8 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
8 BUG: unable to handle kernel NULL pointer dereference
6 Oops: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
5 BUG: unable to handle kernel paging request at ffffffffffffffa8
5 BUG: Bad page map in process init pte:ffffffffffffffff pmd:06d9e067
5 BUG: Bad page map in process init pte:ffffffff pmd:06d9e067
4 Oops: 0002 [#1]
4 Kernel panic - not syncing: Attempted to kill the idle task!
4 BUG: unable to handle kernel paging request at ffff88000cd94000
3 invalid opcode: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 WARNING: CPU: 0 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 BUG: kernel boot hang
3 BUG: Bad page map in process init pte:ffffffff81f0fa00 pmd:06d9e067
3 BUG: Bad page map in process init pte:ffffffff81b52e93 pmd:06d9e067
3 BUG: Bad page map in process init pte:dead4ead00000000 pmd:06d9e067
2 kernel BUG at /c/wfg/linux-drm/include/linux/mm.h:286!
2 general protection fault: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
2 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 Oops: 0002 [#1] PREEMPT
2 Oops: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
2 BUG: unable to handle kernel paging request at 0000000000010012
2 BUG: unable to handle kernel NULL pointer dereference at 00000000000008e0
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000017
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
2 BUG: scheduling while atomic: rc.local/134/0x10000002
2 BUG: scheduling while atomic: S99-rc.local/210/0x10000002
2 BUG: Bad page map in process init pte:ffffffff81b6e50a pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e290 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e208 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e180 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e0f8 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e080 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e070 pmd:06d9e067
2 BUG: Bad page map in process init pte:00000001 pmd:06d9e067
1 tail/181 is trying to release lock (&sig->cred_guard_mutex) at:
1 kernel BUG at /c/wfg/linux-drm/mm/slab.c:463!
1 general protection fault: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 double fault: 0000 [#9] PREEMPT SMP DEBUG_PAGEALLOC
1 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 222 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/include/linux/kref.h:47 kref_get+0x2d/0x36()
1 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_io.c:1411 tty_open+0x215/0x42b()
1 WARNING: CPU: 1 PID: 129 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 117 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 114 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 129 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 Oops: 0010 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] e PorR dEirEecMtoPryT
1 Oops: 0002 [#1] dPonRe.E
1 Oops: 0002 [#1] PREEMPT SMP puDlaEteB-vUolGat_ilPe.AGEALLOC
1 Oops: 0002 [#1] PREEMPT SMP
1 Oops: 0002 [#1] PREEMPT /bSinM/sPh: /procD/sEBUG_PAGEALLOC
1 Oops: 0000 [#4] PREEMPT SMP DEBUG_PAGEALLOC
1 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
1 INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.349 msecs
1 BUG: unable to handle kernel paging request at ffff88000cd6c800
1 BUG: unable to handle kernel paging request at 00002939a1ff0088
1 BUG: unable to handle kernel paging request at 0000200000000000
1 BUG: unable to handle kernel paging request at 000000010000003f
1 BUG: unable to handle kernel paging request at 000000007fffffff
1 BUG: unable to handle kernel paging request at 0000000003ffffc0
1 BUG: unable to handle kernel ULL pointer dereference at 0000000000000008
1 BUG: unable to handle kernel ULL pointer dereference
1 BUG: unable to handle kernel PleaNse wULait: bLoo pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULL pointerereference at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000027
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000014
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
1 BUG: unable to handle kernel NULL poinr deference at 0000000000000008
1 BUG: unable to 9: No suchandle kernel NULL pointer dereference at 0000000000000008
1 BUG: spinlock lockup suspected on CPU#1, migration/1/11
1 BUG: spinlock lockup suspected on CPU#1, kworker/1:1/20
1 BUG: spinlock lockup suspected on CPU#0, sh/126
1 BUG: spinlock lockup suspected on CPU#0, sh/115
1 BUG: scheduling while atomic: sh/130/0x10000002
1 BUG: scheduling while atomic: reboot/222/0x10000002
1 BUG: scheduling while atomic: rc/212/0x10000002
1 BUG: scheduling while atomic: init/212/0x10000002
1 BUG: scheduling while atomic: init/138/0x10000002
1 BUG: Bad rss-counter state mm:ffff88000ca206c0 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9d4080 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff880006a106c0 idx:1 val:2
1 BUG: Bad page map in process init pte:ffffffff82863660 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632c0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632b8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632b0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632a0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82863280 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8275cf00 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82314d30 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82310d10 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff823102f0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8230e7f0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8ab2a pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aaf9 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aabc pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aaa9 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8191eba0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff8800076d53c0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e1c8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e140 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e0b8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d90000 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff88000692ec40 pmd:06d9e067
1 BUG: Bad page map in process init pte:dead4ead003c003c pmd:06d9e067
1 BUG: Bad page map in process init pte:dead4ead002c002c pmd:06d9e067
1 BUG: Bad page map in process init pte:200005401 pmd:06d9e067

It's not really the "first" bad commit because its parent commit 3398d252a4d
("Merge tag 'modules-next-for-linus' of git://git.kernel.org/pub/scm/linux/kerne l/git/rusty/linux")
has a similar 495 good dmesgs and 14587 bad dmesgs with error messages:

13847 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
13792 BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
13760 Kernel panic - not syncing: Fatal exception in interrupt
10011 BUG: kernel boot oops
4280 BUG: kernel test oops
382 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
316 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011!
301 invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
225 Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
178 INFO: lockdep is turned off.
84 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20
65 BUG: unable to handle kernel
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
42 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
40 general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
29 BUG: kernel boot crashed
27 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
22 BUG: scheduling while atomic: init/136/0x10000002
21 BUG: scheduling while atomic: S99-rc.local/210/0x10000002
17 invalid opcode: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
15 Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
14 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC
13 Oops: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
13 BUG: unable to handle kernel NULL pointer dereference at (null)
13 BUG: unable to handle kernel NULL pointer dereference
10 BUG: unable to handle kernel paging request at ffffffffffffffa8
8 Oops: 0002 [#1]
8 BUG: scheduling while atomic: init/214/0x10000002
8 BUG: scheduling while atomic: init/212/0x10000002
7 BUG: scheduling while atomic: init/213/0x10000002
6 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
6 BUG: unable to handle kernel paging request at 0000000000010012
6 BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
6 BUG: scheduling while atomic: rc.local/134/0x10000002
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
4 BUG: unable to handle kernel NULL pointer dereference at 00000000000008e0
4 BUG: scheduling while atomic: ifconfig/230/0x10000002
4 BUG: kernel boot hang
3 WARNING: CPU: 1 PID: 117 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 WARNING: CPU: 1 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 INFO: trying to register non-static key.
3 BUG: unable to handle kernel paging request at ffff88000cd9f000
3 BUG: unable to handle kernel N
3 BUG: scheduling while atomic: rc/212/0x10000002
2 kernel BUG at /c/wfg/linux-drm/mm/slab.c:463!
2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
2 WARNING: CPU: 0 PID: 124 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 123 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 Oops: 0002 [#2] PREEMPT SMP DEBUG_PAGEALLOC
2 Oops: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
2 BUG: unable to handle kernel ULL pointer dereference at 0000000000000008
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
2 BUG: scheduling while atomic: reboot/210/0x10000002
2 BUG: scheduling while atomic: rc/136/0x10000002
2 BUG: scheduling while atomic: S20sendsigs/141/0x10000002
1 t[ri ni ty: n ot f1o.668013] Kernel panic - not syncing: Fatal exception in interrupt
1 sh: /[pr oc /s 4.771591] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 proc o[n /proc fa 2.048043] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 n/modp[ro be : 2.087974] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 kernel BUG at /c/wfg/linux-drm/kernel/cred.c:758!
1 invalid opcode: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 gr[ep : /p ro c/2fi.le087655] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCsy
1 general protection fault: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 et c/rc5 .d1.667815] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC/S99-rc.loca
1 bi[n/ sh : / pr oc4/s.179326] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 221 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/lockdep.c:960 __bfs+0xf1/0x1b3()
1 WARNING: CPU: 1 PID: 138 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 115 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 113 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 221 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 0 PID: 200 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 199 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 196 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 191 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 187 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 147 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 144 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 132 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/include/linux/kref.h:47 kref_get+0x2d/0x36()
1 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_io.c:1411 tty_open+0x215/0x42b()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 126 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 Oops: 0010 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] iled: NPRo suchE deEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] grPRep: /Eproc/fileEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] bPooRtlEogEd.M
1 Oops: 0002 [#1] PRe EorE dMirPecTto ry
1 Oops: 0002 [#1] PREmount:E moMPTunt ing SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT nSe M15P: /triniDtyE:BUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT SmoMunPt: mountDinEg BUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCs/
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCm
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCgrep: /proc/file
1 Oops: 0002 [#1] PREEMPT SMP
1 Oops: 0002 [#1] PREEMPT
1 Oops: 0002 [#1] CPonRfiEguEriMngP nTet wSMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] /PbiRn/EshE: M/pProTc/ sSMP eDlfE/fBd/U9:G N_o PsuAcGEALLOC
1 Oops: 0000 [#4] PREEMPT SMP DEBUG_PAGEALLOC
1 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
1 BUG: unable to handle kernel paging request at ffffffffe7e9eb38
1 BUG: unable to handle kernel paging request at ffff880006db9808
1 BUG: unable to handle kernel paging request at ffff880006db9800
1 BUG: unable to handle kernel paging request at ffff880006db3008
1 BUG: unable to handle kernel paging request at ffff880006db3000
1 BUG: unable to handle kernel paging request at 00002939a1ff0088
1 BUG: unable to handle kernel paging request at 0000200000000080
1 BUG: unable to handle kernel paging request at 00000000c1dcacfe
1 BUG: unable to handle kernel paging request at 000000007fffffff
1 BUG: unable to handle kernel paging request at 0000000003ffffc0
1 BUG: unable to handle kernel gNULL pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULproc on /proc Lfa pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULgrep: /proc/cmdline: No such file or directory
1 BUG: unable to handle kernel NULRuLnni ngp poositinnstter /detec/rrpem-fpoestrinensctse/100.. .at 00
1 BUG: unable to handle kernel NULL pointer ereference
1 BUG: unable to handle kernel NULL pointer derefrence at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereferenceStarting Bootlog at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereference atmo 00unt: 0moun00ti0ng0 00pro0c on /proc fa00ile000d: No such de8
1 BUG: unable to handle kernel NULL pointer dereference at moun0000t: 0moun0ting0 000000p0roc 0on /8proc
1 BUG: unable to handle kernel NULL pointer dereference at gr0e0p:0 0/p0ro00000c0/cm0dl0008in
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000017
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
1 BUG: unable to handle kernel NULL pointer dereference a/bt in0000/sh: 00/p0ro0c/0000s000elf/fd/9:8
1 BUG: unable to handle kernel NULL pointer dereferce at 0000000000000008
1 BUG: unable to handle kernel NULL pointer derefence at 0000000000000008
1 BUG: unable to handle kernel NULL pointe dereference at 0000000000000008
1 BUG: unable to handle kernel NULL poiner dereference at 0000000000000008
1 BUG: unable to handle kernel NULL p
1 BUG: unable to handle kernel NUL/etc/rcS.d/S00fbLsetup : pointer lidne 3:e reference at 0000000000000008
1 BUG: unable to handle kernel NUL/biLn/s ph: /poiroc/nsterel f/fd/9: dNo sucereh file feror edirectnorcy
1 BUG: unable to handle kernel NU/bin/LLsh: / procp/soelf/fdi/9: No sucnteh frile or dire ctdoryeref
1 BUG: unable to handle kernel NU
1 BUG: unable to handle kernel /kNerUneLl-Lte stpointer dereference at 0000000000000008
1 BUG: unable to handle kernel -> 0:9
1 BUG: spinlock lockup suspected on CPU#0, swapper/0/0
1 BUG: scheduling while atomic: stty/188/0x10000002
1 BUG: scheduling while atomic: sleep/130/0x00000002
1 BUG: scheduling while atomic: sleep/129/0x10000002
1 BUG: scheduling while atomic: sleep/129/0x00000002
1 BUG: scheduling while atomic: reboot/221/0x10000002
1 BUG: scheduling while atomic: reboot/134/0x10000002
1 BUG: scheduling while atomic: rc/214/0x10000002
1 BUG: scheduling while atomic: rc/213/0x10000002
1 BUG: scheduling while atomic: rc.local/135/0x10000002
1 BUG: Bad rss-counter state mm:ffff88000cab4d00 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000caaf340 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca33340 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca1d980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca1b980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9e1980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9de080 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9d06c0 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9be6c0 idx:1 val:2

Sorry I should have did more analyzes before reporting it out. The Xen tree is
clean: 10000 good boots w/o a single bad dmesg. Which is also indicated by the
below bisect log:

git bisect bad cf39c8e5352b # 18:34 727- Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
...
git bisect good 23b7eaf82207 # 03:19 5000+ hvc_xen: Remove unnecessary __GFP_ZERO from kzalloc
git bisect good 3398d252a4da # 13:35 10000+ Merge tag 'modules-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good 23b7eaf82207 # 14:55 10000+ hvc_xen: Remove unnecessary __GFP_ZERO from kzalloc
git bisect bad 14951f22f1cc # 14:55 0- Revert "perf symbols: Demangle cloned functions"
git bisect bad e62063d69911 # 15:28 1444- Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect bad a0cf1abc25ac # 15:28 0- Add linux-next specific files for 20130927
git bisect bad b31801966611 # 15:56 728- Merge branch 'tools/kvm'

Where the 5000+/10000+ etc. are the number of boots before claiming the kernel GOOD;
the 727-/0-/728- are the number of boots (minus 1) before finding the bug message in dmesg.

In this bisect, the bug message is "double fault:" and the bisect works in this way:

loop {
if grep "double fault:" dmesg; then
report FAILURE
else if booted N times
report GOOD and break
}

I'll check and improve the bisect script to verify the bisect result
for working with this situation.

Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Felipe Contreras: "Re: [PATCH] acpi: update win8 OSI blacklist"
Previous message: Felipe Contreras: "[PATCH] acpi: add missing win8 OSI comment to blacklist"
In reply to: Linus Torvalds: "Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC"
Next in thread: Fengguang Wu: "Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]