Re: Linux 4.14-rc6: WARNING: CPU: 9 PID: 5377 at arch/x86/events/intel/core.c:2228 intel_pmu_handle_irq+0x4a8/0x4c0
From: Fengguang Wu
Date: Mon Oct 30 2017 - 18:50:08 EST
On Mon, Oct 30, 2017 at 11:02:58AM +0100, Peter Zijlstra wrote:
On Mon, Oct 30, 2017 at 07:27:36AM +0100, Fengguang Wu wrote:
[ 189.480568] perf: interrupt took too long (5132 > 4982), lowering kernel.perf_event_max_sample_rate to 38000
[ 189.690660] perf: interrupt took too long (6582 > 6415), lowering kernel.perf_event_max_sample_rate to 30000
[ 189.901706] perf: interrupt took too long (8268 > 8227), lowering kernel.perf_event_max_sample_rate to 24000
[ 272.841032] perfevents: irq loop stuck!
[ 272.841038] ------------[ cut here ]------------
[ 272.841046] WARNING: CPU: 9 PID: 5377 at arch/x86/events/intel/core.c:2228 intel_pmu_handle_irq+0x4a8/0x4c0
So I've not seen this in a fair while; is this new in 4.14?
It looks a pretty old error. Here is the dmesg for 4.12:
[ 229.514000] Test Case count_global_group_cpu/mem-loads/_cpu/cache-references/_cpu/stalled-cycles-backend/_u PASS!
[ 229.514002]
[ 229.519591] Test Case count_global_group_cpu/mem-loads/_cpu/cache-references/_cpu/stalled-cycles-backend/_k PASS!
[ 229.519594]
[ 229.521742] ROUND : perf hardware event sample group test
[ 229.521744]
[ 229.689807] perfevents: irq loop stuck!
[ 229.689807] ------------[ cut here ]------------
[ 229.689809] WARNING: CPU: 4 PID: 23149 at arch/x86/events/intel/core.c:2114 intel_pmu_handle_irq+0x4a8/0x4c0
[ 229.689809] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver netconsole sr_mod cdrom sd_mod sg intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_realtek ata_generic irqbypass dcdbas pata_acpi snd_hda_codec_generic dell_smm_hwmon uas serio_raw crc32c_intel snd_hda_codec_hdmi pcspkr i7core_edac ata_piix usb_storage libata snd_hda_intel snd_hda_codec firewire_ohci snd_hda_core firewire_core snd_hwdep crc_itu_t snd_pcm snd_timer snd soundcore shpchp acpi_cpufreq ip_tables broadcom bcm_phy_lib
[ 229.689828] CPU: 4 PID: 23149 Comm: perf Not tainted 4.12.0 #1
[ 229.689829] Hardware name: Dell Inc. Studio XPS 8000/0X231R, BIOS A01 08/11/2009
[ 229.689829] task: ffff8801b4308000 task.stack: ffffc900019e8000
[ 229.689830] RIP: 0010:intel_pmu_handle_irq+0x4a8/0x4c0
[ 229.689831] RSP: 0018:ffff8801bfd05c00 EFLAGS: 00010086
[ 229.689832] RAX: 000000000000001c RBX: 0000000000000064 RCX: 0000000000000000
[ 229.689833] RDX: 000000000000001c RSI: 00000000000000f6 RDI: 0000000000000300
[ 229.689833] RBP: ffff8801bfd05df0 R08: 0000000000000000 R09: 0000000400000000
[ 229.689834] R10: ffff8801bfd05b48 R11: 0000000000000005 R12: ffff8801bfd0a3c0
[ 229.689835] R13: ffff88011a4dd800 R14: 0000000000000040 R15: ffff8801bfd0a5c0
[ 229.689835] FS: 0000000000000000(0000) GS:ffff8801bfd00000(0000) knlGS:0000000000000000
[ 229.689836] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 229.689837] CR2: 00007fb532eae9f4 CR3: 000000011d150000 CR4: 00000000000006e0
[ 229.689837] Call Trace:
[ 229.689838] <NMI>
[ 229.689838] ? alloc_set_pte+0x3aa/0x580
[ 229.689839] perf_event_nmi_handler+0x2c/0x50
[ 229.689839] ? sched_clock+0x9/0x10
[ 229.689840] ? sched_clock+0x9/0x10
[ 229.689840] ? perf_event_nmi_handler+0x2c/0x50
[ 229.689841] nmi_handle+0x71/0x130
[ 229.689841] default_do_nmi+0x53/0x110
[ 229.689842] do_nmi+0xec/0x140
[ 229.689842] end_repeat_nmi+0x1a/0x1e
[ 229.689843] RIP: 0010:native_write_msr+0x6/0x30
[ 229.689843] RSP: 0018:ffffc900019ebc18 EFLAGS: 00000046
[ 229.689845] RAX: 0000000000000bb0 RBX: ffff88011a4dd800 RCX: 000000000000038d
[ 229.689845] RDX: 0000000000000000 RSI: 0000000000000bb0 RDI: 000000000000038d
[ 229.689846] RBP: ffffc900019ebc38 R08: fffffffffffffffe R09: 0000000000000000
[ 229.689847] R10: ffffc900019ebba8 R11: 0000000000000005 R12: 000000000000000b
[ 229.689847] R13: 0000000000000008 R14: 0000000000000001 R15: 00000000000000c1
[ 229.689848] ? native_write_msr+0x6/0x30
[ 229.689848] ? native_write_msr+0x6/0x30
[ 229.689849] </NMI>
[ 229.689849] ? intel_pmu_enable_event+0x19c/0x1d0
[ 229.689850] x86_pmu_start+0x7a/0xa0
[ 229.689850] x86_pmu_enable+0x272/0x2e0
[ 229.689851] perf_pmu_enable+0x7/0x10
[ 229.689851] ctx_resched+0x95/0xe0
[ 229.689852] perf_event_exec+0x176/0x1e0
[ 229.689852] setup_new_exec+0x74/0x180
[ 229.689853] load_elf_binary+0x3c6/0x10a0
[ 229.689854] ? selinux_inode_permission+0xc6/0x180
[ 229.689854] search_binary_handler+0xab/0x1e0
[ 229.689855] do_execveat_common+0x553/0x730
[ 229.689855] SyS_execve+0x31/0x40
[ 229.689856] do_syscall_64+0x67/0x150
[ 229.689856] entry_SYSCALL64_slow_path+0x25/0x25
[ 229.689857] RIP: 0033:0x7fee90ba4607
[ 229.689857] RSP: 002b:00007fff29bc2058 EFLAGS: 00000202 ORIG_RAX: 000000000000003b
[ 229.689859] RAX: ffffffffffffffda RBX: 00007fff29bc7033 RCX: 00007fee90ba4607
[ 229.689859] RDX: 000056135240fdd0 RSI: 00007fff29bc6890 RDI: 00007fff29bc7033
[ 229.689860] RBP: 00007fff29bc20d0 R08: 00007fff29bc2030 R09: 0000000000000000
[ 229.689861] R10: 00000000000005b5 R11: 0000000000000202 R12: 00007fff29bc6890
[ 229.689862] R13: 000056135240fdd0 R14: 0000000000000000 R15: 00007fff29bc6890
[ 229.689862] Code: ff ff 48 89 c2 e8 09 71 05 00 66 90 48 8b bd 40 fe ff ff 57 9d 66 66 90 66 90 e9 b2 fd ff ff 48 c7 c7 95 af c9 81 e8 f5 d4 18 00 <0f> ff e8 71 a9 ff ff c6 05 ba a8 30 01 01 e9 50 fe ff ff 0f 1f
[ 229.689877] ---[ end trace 3a9007d5f365d9db ]---
[ 229.689878]
[ 229.689878] CPU#4: ctrl: 0000000000000000
[ 229.689879] CPU#4: status: 0000000400000000
[ 229.689879] CPU#4: overflow: 0000000000000000
[ 229.689880] CPU#4: fixed: 0000000000000bb0
[ 229.689880] CPU#4: pebs: 0000000000000000
[ 229.689881] CPU#4: debugctl: 0000000000000000
[ 229.689881] CPU#4: active: 0000000600000002
[ 229.689882] CPU#4: gen-PMC0 ctrl: 000000000012100b
[ 229.689883] CPU#4: gen-PMC0 count: 0000ffff80000001
[ 229.689883] CPU#4: gen-PMC0 left: 000000007fffffff
[ 229.689884] CPU#4: gen-PMC1 ctrl: 000000000053003c
[ 229.689884] CPU#4: gen-PMC1 count: 0000fffffffffffe
[ 229.689885] CPU#4: gen-PMC1 left: 0000000000000002
[ 229.689885] CPU#4: gen-PMC2 ctrl: 0000000001923fb1
[ 229.689886] CPU#4: gen-PMC2 count: 0000ffff805bc121
[ 229.689886] CPU#4: gen-PMC2 left: 000000007fffffff
[ 229.689887] CPU#4: gen-PMC3 ctrl: 0000000000000000
[ 229.689887] CPU#4: gen-PMC3 count: 0000000000000000
[ 229.689888] CPU#4: gen-PMC3 left: 000000007fe1c525
[ 229.689889] CPU#4: fixed-PMC0 count: 0000ffff80034ecd
[ 229.689889] CPU#4: fixed-PMC1 count: 0000ffffc1e6e594
[ 229.689890] CPU#4: fixed-PMC2 count: 0000fffffffffffe
[ 229.689890] core: clearing PMU state on CPU#4
[ 230.172428] core: clearing PMU state on CPU#1
[ 230.627603] core: clearing PMU state on CPU#2
[ 231.426767] core: clearing PMU state on CPU#1
[ 231.427940] core: clearing PMU state on CPU#1
[ 231.867912] core: clearing PMU state on CPU#4
[ 232.328991] core: clearing PMU state on CPU#6
[ 232.335914] core: clearing PMU state on CPU#6
[ 232.778826] core: clearing PMU state on CPU#2
[ 232.780834] core: clearing PMU state on CPU#2
[ 258.662710] core: clearing PMU state on CPU#5
[ 259.105382] core: clearing PMU state on CPU#3
[ 259.551511] core: clearing PMU state on CPU#6
[ 260.345369] core: clearing PMU state on CPU#0
[ 260.346034] core: clearing PMU state on CPU#0
[ 260.792645] core: clearing PMU state on CPU#0
[ 261.242718] core: clearing PMU state on CPU#6
[ 261.248905] core: clearing PMU state on CPU#6
[ 261.713365] core: clearing PMU state on CPU#7
[ 261.720890] core: clearing PMU state on CPU#7
[ 262.083667] Thu Jul 6 09:22:01 CST 2017
[ 262.083672]
[ 262.091138] 61 test cases pass for perf_hw_event_sample_group test. 14 test cases fail for perf_hw_event_sample_group test.
Here is the dmesg for 4.10, however the call trace looks a bit different,
so it might be a different problem in 4.10.
[ 49.569619] 2017-03-31 00:36:43 ./case-anon-r-seq-mt
[ 49.569620]
[ 49.579122] 2017-03-31 00:36:43 ./usemem --runtime 300 -t 192 --readonly 366503875925
[ 49.579123]
[ 191.384432] perfevents: irq loop stuck!
[ 191.390313] perf: interrupt took too long (46642 > 5722), lowering kernel.perf_event_max_sample_rate to 4000
[ 191.402656] ------------[ cut here ]------------
[ 191.409134] WARNING: CPU: 77 PID: 3008 at arch/x86/events/intel/core.c:2093 intel_pmu_handle_irq+0x2cf/0x460
[ 191.425093] Modules linked in: loop sr_mod cdrom sd_mod sg rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl sb_edac edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm irqbypass mgag200 crct10dif_pclmul ttm crc32_pclmul drm_kms_helper ahci syscopyarea crc32c_intel sysfillrect sysimgblt snd_pcm ghash_clmulni_intel fb_sys_fops libahci pcbc snd_timer mpt3sas ipmi_si raid_class aesni_intel snd crypto_simdipmi_devintf scsi_transport_sas drm glue_helper soundcore pcspkr libata cryptd shpchp wmi ipmi_msghandler acpi_pad ip_tables
[ 191.486735] CPU: 77 PID: 3008 Comm: usemem Not tainted 4.10.0 #1
[ 191.495127] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRBDXSD1.86B.0335.R00.1601291644 01/29/2016
[ 191.508376] Call Trace:
[ 191.512769] <NMI>
[ 191.516427] dump_stack+0x63/0x8a
[ 191.521747] __warn+0xcb/0xf0
[ 191.526707] warn_slowpath_fmt+0x4f/0x60
[ 191.532648] intel_pmu_handle_irq+0x2cf/0x460
[ 191.539064] perf_event_nmi_handler+0x2c/0x50
[ 191.545523] nmi_handle+0x71/0x130
[ 191.550909] default_do_nmi+0x53/0x110
[ 191.556665] do_nmi+0xec/0x140
[ 191.561624] end_repeat_nmi+0x1a/0x1e
[ 191.567280] RIP: 0010:native_write_msr+0x6/0x30
[ 191.573924] RSP: 0000:ffff8877e3943e50 EFLAGS: 00000046
[ 191.581253] RAX: 00000000000000b0 RBX: ffff887871df1000 RCX: 000000000000038d
[ 191.590716] RDX: 0000000000000000 RSI: 00000000000000b0 RDI: 000000000000038d
[ 191.600185] RBP: ffff8877e3943e70 R08: 0000000000000007 R09: 0000000000000000
[ 191.609635] R10: 0000000000000001 R11: 00007effeff2fd80 R12: 000000000000000b
[ 191.619122] R13: 0000000000000004 R14: 0000000000000000 R15: ffff8877e3952728
[ 191.628555] ? native_write_msr+0x6/0x30
[ 191.634418] ? native_write_msr+0x6/0x30
[ 191.640253] </NMI>
[ 191.644015] <IRQ>
[ 191.647685] ? intel_pmu_enable_event+0x19c/0x1d0
[ 191.654356] x86_pmu_start+0x7e/0x100
[ 191.659790] x86_pmu_enable+0x272/0x2e0
[ 191.665435] ? __perf_install_in_context+0x150/0x150
[ 191.672315] perf_pmu_enable+0x7/0x10
[ 191.678448] perf_mux_hrtimer_handler+0x1bc/0x200
[ 191.684987] __hrtimer_run_queues+0xeb/0x280
[ 191.691009] hrtimer_interrupt+0xa3/0x1f0
[ 191.696727] local_apic_timer_interrupt+0x38/0x60
[ 191.703292] smp_apic_timer_interrupt+0x38/0x50
[ 191.709588] apic_timer_interrupt+0x93/0xa0
[ 191.715570] RIP: 0033:0x55b1a2e5a6cb
[ 191.720834] RSP: 002b:00007effeff2fd98 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
[ 191.730547] RAX: 0000000000000001 RBX: 00000003cb4aeea9 RCX: 000000001ed6206e
[ 191.739788] RDX: 0000000000000001 RSI: 00000003cb4aeea9 RDI: 000072ffd3ee1000
[ 191.748980] RBP: 00000003cb4aeea9 R08: 00007effeff2fe3c R09: 0000000000000001
[ 191.758174] R10: 001ec3d1af80ed56 R11: 00007effeff2fd80 R12: 0000000000000001
[ 191.767343] R13: 00007effeff2fe3c R14: 000072ffd3ee1000 R15: 0000001e5a577548
[ 191.776436] </IRQ>
[ 191.779857] ---[ end trace 7186982383071abb ]---
[ 191.786094]
[ 191.788781] CPU#77: ctrl: 0000000000000000
[ 191.794913] CPU#77: status: 0000000000000000
[ 191.801047] CPU#77: overflow: 0000000000000000
[ 191.807145] CPU#77: fixed: 00000000000000b0
[ 191.813165] CPU#77: pebs: 0000000000000000
[ 191.819140] CPU#77: debugctl: 0000000000000000
[ 191.825112] CPU#77: active: 0000000200000000
[ 191.831075] CPU#77: gen-PMC0 ctrl: 00000000001301b7
[ 191.837634] CPU#77: gen-PMC0 count: 0000ffff000096bf
[ 191.844224] CPU#77: gen-PMC0 left: 00000000ffff6943
[ 191.850804] CPU#77: gen-PMC1 ctrl: 00000000001301bb
[ 191.857396] CPU#77: gen-PMC1 count: 0000ffff0000300b
[ 191.863931] CPU#77: gen-PMC1 left: 00000000ffffcff6
[ 191.870459] CPU#77: gen-PMC2 ctrl: 00000000001301b7
[ 191.877035] CPU#77: gen-PMC2 count: 0000ffff00004e73
U#77: gen-PMC3 count: 0000ffff00014be6
[ 191.903129] CPU#77: gen-PMC3 left: 00000000fffeb41a
[ 191.909613] CPU#77: fixed-PMC0 count: 0000ffffaa34e4e1
[ 191.916125] CPU#77: fixed-PMC1 count: 0000ffffffffffff
[ 191.922596] CPU#77: fixed-PMC2 count: 0000000000000000
[ 191.929061] core: clearing PMU state on CPU#77
[ 349.874968] 324393413298 bytes / 300292916 usecs = 1054938 KB/s
[ 349.874973]
Here is the dmesg in 4.9:
[ 196.766194] Test Case cpu_PMU_event_count_test_cpu/stalled-cycles-frontend/ PASS!
[ 196.766593]
[ 196.768006] ROUND 1: perf hardware event sample test
[ 196.768223]
[ 199.427316] perfevents: irq loop stuck!
[ 199.427691] ------------[ cut here ]------------
[ 199.428011] WARNING: CPU: 7 PID: 18808 at arch/x86/events/intel/core.c:2093 intel_pmu_handle_irq+0x2ca/0x460
[ 199.431222] CPU: 7 PID: 18808 Comm: perf Not tainted 4.9.0 #1
[ 199.431584] Hardware name: Dell Inc. Studio XPS 8000/0X231R, BIOS A01 08/11/2009
[ 199.433408] Call Trace:
[ 199.433647] <NMI> [<ffffffff814771b9>] dump_stack+0x63/0x8a
[ 199.434009] [<ffffffff810803db>] __warn+0xcb/0xf0
[ 199.434334] [<ffffffff8108044f>] warn_slowpath_fmt+0x4f/0x60
[ 199.434695] [<ffffffff8100c8ba>] intel_pmu_handle_irq+0x2ca/0x460
[ 199.435107] [<ffffffff81240cb1>] ? exit_files+0x41/0x60
[ 199.435424] [<ffffffff8100599c>] perf_event_nmi_handler+0x2c/0x50
[ 199.435772] [<ffffffff81031fb1>] nmi_handle+0x71/0x130
[ 199.436086] [<ffffffff81032523>] default_do_nmi+0x53/0x110
[ 199.436413] [<ffffffff810326cc>] do_nmi+0xec/0x140
[ 199.436716] [<ffffffff8196bab1>] end_repeat_nmi+0x1a/0x1e
[ 199.437040] [<ffffffff81062396>] ? native_write_msr+0x6/0x30
[ 199.437376] [<ffffffff81062396>] ? native_write_msr+0x6/0x30
[ 199.437710] [<ffffffff81062396>] ? native_write_msr+0x6/0x30
[ 199.438048] <EOE> [<ffffffff8100c31c>] ? intel_pmu_enable_event+0x19c/0x1d0
[ 199.438435] [<ffffffff81006e3e>] x86_pmu_start+0x7e/0x100
[ 199.438760] [<ffffffff81007132>] x86_pmu_enable+0x272/0x2e0
[ 199.439121] [<ffffffff8117a6f7>] perf_pmu_enable+0x7/0x10
[ 199.439473] [<ffffffff8117ec8a>] ctx_resched+0x7a/0x90
[ 199.439790] [<ffffffff81185540>] perf_event_exec+0x140/0x190
[ 199.440126] [<ffffffff8122920f>] setup_new_exec+0x6f/0x1c0
[ 199.440452] [<ffffffff8127f8ad>] load_elf_binary+0x38d/0x1060
[ 199.440793] [<ffffffff810a81da>] ? __might_sleep+0x4a/0x80
[ 199.441125] [<ffffffff8122774b>] search_binary_handler+0xab/0x1e0
[ 199.441474] [<ffffffff81228f2e>] do_execveat_common+0x54e/0x730
[ 199.441846] [<ffffffff8122939a>] SyS_execve+0x3a/0x50
[ 199.442160] [<ffffffff81003b67>] do_syscall_64+0x67/0x180
[ 199.442487] [<ffffffff8196a32b>] entry_SYSCALL64_slow_path+0x25/0x25
[ 199.442849] ---[ end trace a3a6749931683848 ]---
[ 199.443159]
[ 272.841083] CPU: 9 PID: 5377 Comm: usemem Not tainted 4.14.0-rc6 #1
At least you've got the right kernel this time... :-)
Heh.
[ 272.841084] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRRFSDP1.86B.0271.R00.1510301446 10/30/2015
That's a dual socket board for Xeon E5v3, which translates to HSW-EP if
google is any good today. That's not a machine I have. Can you see this
on other hardware too, like say IVB-EP ?
This warning has showed up in 4 machines:
- lkp-bdw-ep2
- lkp-bdw-ex2
- nhm-white
- nhm-white2
wfg@inn /pkg/linux/x86_64-rhel-7.2/gcc-6/v4.13% g -l 'WARNING: .*intel_pmu_handle_irq' dmesg-*
dmesg-lkp-bdw-ep2:20171011050615:x86_64-rhel-7.2:gcc-6:4.14.0-rc4-02928-g06e7b50:1
dmesg-lkp-bdw-ex2:20171024032542:x86_64-rhel-7.2-kaslr:gcc-6:4.14.0-rc3-00232-gc495dc6e:1
They happens once in unixbench and another time in pigz testing.
The warning cannot be reliably reproduced.
wfg@inn /pkg/linux/x86_64-rhel-7.2/gcc-6/v4.10% g -l 'WARNING: .*intel_pmu_handle_irq' dmesg-*
dmesg-lkp-bdw-ex2:20170501234718:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502011236:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502012136:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502014040:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502032242:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502034027:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502043934:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502045839:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502052859:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502064733:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502080004:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502081243:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502083315:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170502091456:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-lkp-bdw-ex2:20170509182610:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170627153156:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170627154205:x86_64-rhel-7.2:gcc-6:4.12.0-rc6-next-20170622:1
dmesg-nhm-white:20170628103305:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170628104830:x86_64-rhel-7.2:gcc-6:4.12.0-rc7-00192-gc0bc126:1
dmesg-nhm-white:20170629120909:x86_64-rhel-7.2:gcc-6:4.12.0-rc7-wt-ath-10874-gb813bcd:1
dmesg-nhm-white:20170717190503:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170717191227:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170717191930:x86_64-rhel-7.2:gcc-6:4.10.0:1
dmesg-nhm-white:20170717192647:x86_64-rhel-7.2:gcc-6:4.12.0:1
dmesg-nhm-white2:20170701152813:x86_64-rhel-7.2:gcc-6:4.12.0-rc7-00192-gc0bc126:1
dmesg-nhm-white2:20170727173107:x86_64-rhel-7.2:gcc-6:4.10.0:1
Those mainly happen in tests: vm-scalability, unixbench, perf_test.
Thanks,
Fengguang