WARNING: kernel stack regs at 0000000068363d2a in telegraf:23122 has bad 'bp' value 000000007b842ec4

From: Jake M
Date: Mon Jun 24 2019 - 00:50:15 EST

Hi,

The box has crashed twice this week both times related to telegraf, we
are running ZFS and attached is the kernel config.

Any ideas?

Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
Kernel: 4.18.0-1016-aws

ZFS
filename: /lib/modules/4.18.0-1016-aws/updates/dkms/zfs.ko
version: 0.7.13-1
license: CDDL
author: OpenZFS on Linux
description: ZFS
alias: devname:zfs
alias: char-major-10-249
srcversion: CD1489EAB5842683DC3584F
depends: spl,znvpair,zcommon,zunicode,zavl,icp
retpoline: Y
name: zfs
vermagic: 4.18.0-1016-aws SMP mod_unload


[2311488.107255] PANIC: double fault, error_code: 0x0
[2311488.109565] CPU: 36 PID: 23122 Comm: telegraf Kdump: loaded
Tainted: P OE 4.18.0-1016-aws #18~18.04.1-Ubuntu
[2311488.109566] Hardware name: Amazon EC2 r5d.12xlarge/, BIOS 1.0 10/16/2017
[2311488.109573] RIP: 0010:do_async_page_fault+0x5/0x80
[2311488.109574] Code: 8d b5 78 ff ff ff 48 8d 78 10 e8 96 e6 06 00 eb
cb e8 1f 16 02 00 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
44 00 00 <55> 48 89 e5 53 65 8b 05 2f 70 1b 57 85 c0 74 37 65 8b 05 e4
6f 1b
[2311488.109602] RSP: 0018:fffffe0000637000 EFLAGS: 00010087
[2311488.109604] RAX: 00000000a9800a27 RBX: 0000000000000000 RCX:
ffffffffa9800a27
[2311488.109605] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
fffffe0000637008
[2311488.109606] RBP: fffffe0000637009 R08: 0000000000000000 R09:
0000000000000000
[2311488.109607] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[2311488.109608] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[2311488.109610] FS: 00007fa0097fa700(0000) GS:ffff9ef8fb500000(0000)
knlGS:0000000000000000
[2311488.109611] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2311488.109612] CR2: fffffe0000636ff8 CR3: 00000056c3de4003 CR4:
00000000007606e0
[2311488.109616] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[2311488.109617] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[2311488.109617] PKRU: 55555554
[2311488.109618] Call Trace:
[2311488.109620] Kernel panic - not syncing: Machine halted.
[2311488.112028] CPU: 36 PID: 23122 Comm: telegraf Kdump: loaded
Tainted: P OE 4.18.0-1016-aws #18~18.04.1-Ubuntu
[2311488.116715] Hardware name: Amazon EC2 r5d.12xlarge/, BIOS 1.0 10/16/2017
[2311488.119698] Call Trace:
[2311488.120932] <#DF>
[2311488.122003] dump_stack+0x63/0x85
[2311488.123641] panic+0xe4/0x244
[2311488.125077] ? async_page_fault+0x1e/0x30
[2311488.126957] df_debug+0x2d/0x30
[2311488.128487] do_double_fault+0xa1/0x140
[2311488.130258] double_fault+0x1e/0x30
[2311488.131911] RIP: 0010:do_async_page_fault+0x5/0x80
[2311488.134226] Code: 8d b5 78 ff ff ff 48 8d 78 10 e8 96 e6 06 00 eb
cb e8 1f 16 02 00 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
44 00 00 <55> 48 89 e5 53 65 8b 05 2f 70 1b 57 85 c0 74 37 65 8b 05 e4
6f 1b
[2311488.142318] RSP: 0018:fffffe0000637000 EFLAGS: 00010087
[2311488.144786] RAX: 00000000a9800a27 RBX: 0000000000000000 RCX:
ffffffffa9800a27
[2311488.148112] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
fffffe0000637008
[2311488.152820] RBP: fffffe0000637009 R08: 0000000000000000 R09:
0000000000000000
[2311488.158990] R10: 0000000000000000 R11: 0000000000000000 R12:
0000000000000000
[2311488.164822] R13: 0000000000000000 R14: 0000000000000000 R15:
0000000000000000
[2311488.170684] ? native_iret+0x7/0x7
[2311488.173652] WARNING: kernel stack regs at 0000000068363d2a in
telegraf:23122 has bad 'bp' value 000000007b842ec4
[2311488.173653] unwind stack type:5 next_sp:000000008e5f46ef
mask:0x20 graph_idx:0
[2311488.173655] 000000007876cc6b: fffffe0000637e50 (0xfffffe0000637e50)
[2311488.173657] 0000000076b771fd: ffffffffa8e32793
(show_trace_log_lvl+0x223/0x3d0)
[2311488.173660] 000000007707741a: ffffffffa9e97f70 (.LC0+0x530/0x710)
[2311488.173661] 00000000116bf189: ffffffffa9e980a2 (.LC0+0x662/0x710)
[2311488.173662] 00000000369b663b: ffff9ef718653c00 (0xffff9ef718653c00)
[2311488.173663] 00000000f2b9e90d: ffffffffa9800a27 (native_iret+0x7/0x7)
[2311488.173664] 0000000080f01fd0: 0000000000000000 ...
[2311488.173664] 0000000006e868c6: 0000000000000020 (0x20)
[2311488.173665] 00000000467d8117: 0000000000000005 (0x5)
[2311488.173666] 00000000614fb3b0: fffffe0000637000 (0xfffffe0000637000)
[2311488.173666] 00000000ce075175: fffffe0000638000 (0xfffffe0000638000)
[2311488.173667] 00000000f6499015: fffffe0000637000 (0xfffffe0000637000)
[2311488.173668] 00000000049718e3: 0000000000000005 (0x5)
[2311488.173668] 00000000381d5adf: fffffe0000637000 (0xfffffe0000637000)
[2311488.173669] 00000000b719e857: fffffe0000638000 (0xfffffe0000638000)
[2311488.173669] 00000000d98e04b3: fffffe0000637000 (0xfffffe0000637000)
[2311488.173670] 00000000b73751f2: 0000000000000020 (0x20)
[2311488.173671] 00000000c1803216: ffff9ef718653c00 (0xffff9ef718653c00)
[2311488.173671] 000000005a3cf788: 0000010100000000 (0x10100000000)
[2311488.173672] 0000000007b64ed7: 0000000000000000 ...
[2311488.173672] 0000000009d1d5db: fffffe0000637d68 (0xfffffe0000637d68)
[2311488.173675] 00000000e406678a: ffffffffa8e6d085
(do_async_page_fault+0x5/0x80)
[2311488.173675] 000000005c5c1c56: fffffe0000637f58 (0xfffffe0000637f58)
[2311488.173676] 0000000085febed7: 7d74493d0e5c7e00 (0x7d74493d0e5c7e00)
[2311488.173676] 000000005e33596d: 0000000000000086 (0x86)
[2311488.173677] 000000008cba48f1: 0000000000000000 ...
[2311488.173677] 000000005d67e7dc: fffffe0000637f18 (0xfffffe0000637f18)
[2311488.173678] 0000000018e5f25d: 00000056c3de4003 (0x56c3de4003)
[2311488.173678] 000000001d614f09: 0000000000000000 ...
[2311488.173679] 000000002fdcb7e7: fffffe0000637e60 (0xfffffe0000637e60)
[2311488.173680] 0000000039750c75: ffffffffa8e32974 (show_stack+0x34/0x50)
[2311488.173681] 000000006007bd6d: fffffe0000637e80 (0xfffffe0000637e80)
[2311488.173683] 00000000cc918138: ffffffffa977329e (dump_stack+0x63/0x85)
[2311488.173684] 0000000085744678: fffffe0000637f00 (0xfffffe0000637f00)
[2311488.173685] 00000000fe18536c: ffffffffa9ea5422 (.LC0+0x1b2/0x460)
[2311488.173686] 000000006771ae0a: fffffe0000637f08 (0xfffffe0000637f08)
[2311488.173688] 000000006c9cf5ba: ffffffffa8e8eb00 (panic+0xe4/0x244)
[2311488.173689] 00000000e638e3e1: ffff9ef700000008 (0xffff9ef700000008)
[2311488.173690] 00000000c98ad3ca: fffffe0000637f18 (0xfffffe0000637f18)
[2311488.173690] 00000000b1340d7a: fffffe0000637eb0 (0xfffffe0000637eb0)
[2311488.173691] 00000000a4519e0c: 7d74493d0e5c7e00 (0x7d74493d0e5c7e00)
[2311488.173692] 0000000089fc5351: ffffffffa98010be (async_page_fault+0x1e/0x30)
[2311488.173693] 000000002e312d25: fffffe0000637e48 (0xfffffe0000637e48)
[2311488.173693] 00000000e196c719: fffffe0000632000 (0xfffffe0000632000)
[2311488.173694] 00000000a305c4b8: 0000000000000004 (0x4)
[2311488.173695] 000000000185ec0d: 0000000000015ba0 (0x15ba0)
[2311488.173695] 00000000ce241026: 0000000000000004 (0x4)
[2311488.173696] 00000000c80f9f42: fffffe0000637f58 (0xfffffe0000637f58)
[2311488.173696] 000000009abeada7: 0000000000000000 ...
[2311488.173698] 0000000007fdbbd8: ffffffffa9c04f58 (ds.4711+0x2/0x2)
[2311488.173699] 0000000093e9aced: 00000056c3de4003 (0x56c3de4003)
[2311488.173699] 000000000e8ee0e7: fffffe0000637f20 (0xfffffe0000637f20)
[2311488.173700] 00000000a3e660e0: ffffffffa8e6987d (df_debug+0x2d/0x30)
[2311488.173701] 000000006f6b547c: fffffe0000637f58 (0xfffffe0000637f58)
[2311488.173702] 000000007768b6d3: fffffe0000637f48 (0xfffffe0000637f48)
[2311488.173703] 000000001a031e41: ffffffffa8e2f2e1 (do_double_fault+0xa1/0x140)
[2311488.173704] 000000008ae2ed69: 0000000000000001 (0x1)
[2311488.173704] 00000000e1c42ebd: 0000000000000000 ...
[2311488.173705] 00000000771c60dc: fffffe0000637f59 (0xfffffe0000637f59)
[2311488.173706] 000000001c388f18: ffffffffa9800c4e (double_fault+0x1e/0x30)
[2311488.173706] 0000000068363d2a: 0000000000000000 ...
[2311488.173707] 000000006bb2e642: fffffe0000637009 (0xfffffe0000637009)
[2311488.173707] 00000000ed5081db: 0000000000000000 ...
[2311488.173708] 0000000079766182: 00000000a9800a27 (0xa9800a27)
[2311488.173709] 00000000d7e8925c: ffffffffa9800a27 (native_iret+0x7/0x7)
[2311488.173709] 00000000758c3901: 0000000000000000 ...
[2311488.173710] 00000000c8cacbb5: fffffe0000637008 (0xfffffe0000637008)
[2311488.173711] 000000001b02ff67: ffffffffffffffff (0xffffffffffffffff)
[2311488.173712] 000000003f3101ae: ffffffffa8e6d085
(do_async_page_fault+0x5/0x80)
[2311488.173713] 0000000021ccdc1a: 0000000000000010 (0x10)
[2311488.173713] 000000000f682887: 0000000000010087 (0x10087)
[2311488.173714] 00000000d7b19537: fffffe0000637000 (0xfffffe0000637000)
[2311488.173714] 000000001b695c3e: 0000000000000018 (0x18)
[2311488.173715] </#DF>

Attachment: config-4.18.0-1016-aws
Description: Binary data