Re: kernel OOPS in MM(?)

[Evgenii Lepikhin]

Hello,

On 2016-03-10 12:31, Evgenii Lepikhin wrote:

> We need help to understand the source of the problem and may be to create a bugreport. Here is crash report:
>
> Mar 10 04:03:51 l28 kernel: [2075560.434445] BUG: unable to handle kernel paging request at 0000000040008021
> Mar 10 04:03:51 l28 kernel: [2075560.434669] IP: [<ffffffff810ee519>] __kmalloc+0x69/0x100
> Mar 10 04:03:51 l28 kernel: [2075560.434800] PGD b7e462067 PUD 0 
> Mar 10 04:03:51 l28 kernel: [2075560.434913] Oops: 0000 [#1] SMP 
> Mar 10 04:03:51 l28 kernel: [2075560.435044] Modules linked in:
> tcm_loop iscsi_target_mod target_core_pscsi target_core_file
> target_core_iblock target_core_mod ipt_NETFLOW(O) configfs iscsi_tcp
> libis
> csi_tcp libiscsi scsi_transport_iscsi fuse [last unloaded: ipfw_mod]
> Mar 10 04:03:51 l28 kernel: [2075560.435539] CPU: 4 PID: 27141 Comm: rm Tainted: G           O 3.12.51-jl-2015-12-25 #1
> Mar 10 04:03:51 l28 kernel: [2075560.435734] Hardware name: Intel Corporation S2600IP ........../S2600IP, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
> Mar 10 04:03:51 l28 kernel: [2075560.435939] task: ffff880e622ccba0 ti: ffff880eeb008000 task.ti: ffff880eeb008000
> Mar 10 04:03:51 l28 kernel: [2075560.436131] RIP: 0010:[<ffffffff810ee519>]  [<ffffffff810ee519>] __kmalloc+0x69/0x100
> Mar 10 04:03:51 l28 kernel: [2075560.436333] RSP: 0018:ffff880eeb009b38  EFLAGS: 00010282
> Mar 10 04:03:51 l28 kernel: [2075560.436439] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00000000a8a73dc2
> Mar 10 04:03:51 l28 kernel: [2075560.436632] RDX: 00000000a8a73dc1 RSI: 0000000000000000 RDI: 0000000000013500
> Mar 10 04:03:51 l28 kernel: [2075560.438248] RBP: ffff880eeb009b58 R08: ffff88103fc13500 R09: ffffffff811a0267
> Mar 10 04:03:51 l28 kernel: [2075560.438446] R10: ffff880eeb009d84 R11: 0000000000000000 R12: ffff88081f803a00
> Mar 10 04:03:51 l28 kernel: [2075560.438656] R13: 0000000040008021 R14: 0000000000000250 R15: ffff880250e833b0
> Mar 10 04:03:51 l28 kernel: [2075560.438851] FS:  00007fe2316dd700(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000
> Mar 10 04:03:51 l28 kernel: [2075560.439045] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Mar 10 04:03:51 l28 kernel: [2075560.439152] CR2: 0000000040008021 CR3: 0000000a20736000 CR4: 00000000000407e0
> Mar 10 04:03:51 l28 kernel: [2075560.439343] Stack:
> Mar 10 04:03:51 l28 kernel: [2075560.439439]  0000000000000000 0000000000000250 0000000000000060 0000000000000000
> Mar 10 04:03:51 l28 kernel: [2075560.439663]  ffff880eeb009b88 ffffffff811a0267 ffff881015fb7fe0 0000000000000060
> Mar 10 04:03:51 l28 kernel: [2075560.439898]  ffff880250e83490 0000000000000000 ffff880eeb009ba8 ffffffff811a02f8
> Mar 10 04:03:51 l28 kernel: [2075560.440153] Call Trace:
> Mar 10 04:03:51 l28 kernel: [2075560.440257]  [<ffffffff811a0267>] kmem_alloc+0x67/0xe0
> Mar 10 04:03:51 l28 kernel: [2075560.440365]  [<ffffffff811a02f8>] kmem_zalloc+0x18/0x40
> Mar 10 04:03:51 l28 kernel: [2075560.440473]  [<ffffffff811e0523>] xfs_log_commit_cil+0x373/0x4c0
> Mar 10 04:03:51 l28 kernel: [2075560.440585]  [<ffffffff811aab00>] ? xfs_bmap_search_multi_extents+0xe0/0x110
> Mar 10 04:03:51 l28 kernel: [2075560.440783]  [<ffffffff8119fb6c>] xfs_trans_commit+0x6c/0x250
> Mar 10 04:03:51 l28 kernel: [2075560.440899]  [<ffffffff811875d7>] xfs_bmap_finish+0xb7/0x1a0

Another issue on the same server, same instruction pointer:

Mar 16 04:53:54 l28 kernel: [521052.387878] BUG: unable to handle kernel paging request at 0000000040008021
Mar 16 04:53:54 l28 kernel: [521052.388022] IP: [<ffffffff810ee519>] __kmalloc+0x69/0x100
Mar 16 04:53:54 l28 kernel: [521052.388171] PGD 0 
Mar 16 04:53:54 l28 kernel: [521052.388289] Oops: 0000 [#1] SMP 
Mar 16 04:53:54 l28 kernel: [521052.388410] Modules linked in: tcm_loop iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod ipt_NETFLOW(O) configfs iscsi_tcp libis
csi_tcp libiscsi scsi_transport_iscsi fuse
Mar 16 04:53:54 l28 kernel: [521052.388913] CPU: 6 PID: 5947 Comm: iscsi_trx Tainted: G           O 3.12.51-jl-2015-12-25 #1
Mar 16 04:53:54 l28 kernel: [521052.389125] Hardware name: Intel Corporation S2600IP ........../S2600IP, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013
Mar 16 04:53:54 l28 kernel: [521052.389351] task: ffff88081a3a6720 ti: ffff8808162de000 task.ti: ffff8808162de000
Mar 16 04:53:54 l28 kernel: [521052.389566] RIP: 0010:[<ffffffff810ee519>]  [<ffffffff810ee519>] __kmalloc+0x69/0x100
Mar 16 04:53:54 l28 kernel: [521052.389782] RSP: 0018:ffff8808162dfd18  EFLAGS: 00010286
Mar 16 04:53:54 l28 kernel: [521052.389899] RAX: 0000000000000000 RBX: ffff880819a51800 RCX: 0000000003b305d3
Mar 16 04:53:54 l28 kernel: [521052.390112] RDX: 0000000003b305d2 RSI: 0000000000000000 RDI: 0000000000013500
Mar 16 04:53:54 l28 kernel: [521052.390309] RBP: ffff8808162dfd38 R08: ffff88103fd13500 R09: ffffffffa00e7072
Mar 16 04:53:54 l28 kernel: [521052.390503] R10: 0000000000100000 R11: 0000000000000030 R12: ffff88081f803a00
Mar 16 04:53:54 l28 kernel: [521052.390694] R13: 0000000040008021 R14: 00000000000080d0 R15: ffff8808162dfdd0
Mar 16 04:53:54 l28 kernel: [521052.390888] FS:  0000000000000000(0000) GS:ffff88103fd00000(0000) knlGS:0000000000000000
Mar 16 04:53:54 l28 kernel: [521052.391082] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 16 04:53:54 l28 kernel: [521052.391190] CR2: 0000000040008021 CR3: 000000000180b000 CR4: 00000000000407e0
Mar 16 04:53:54 l28 kernel: [521052.391382] Stack:
Mar 16 04:53:54 l28 kernel: [521052.391477]  ffff880819a51800 ffff8808162dfe50 ffff880816319cc0 0000000000000006
Mar 16 04:53:54 l28 kernel: [521052.393132]  ffff8808162dfeb8 ffffffffa00e7072 ffff8808162dfd78 ffff88103fd10880
Mar 16 04:53:54 l28 kernel: [521052.393345]  0000000000000000 ffff880800000000 0000000000001000 ffff880816318124
Mar 16 04:53:54 l28 kernel: [521052.393585] Call Trace:
Mar 16 04:53:54 l28 kernel: [521052.393703]  [<ffffffffa00e7072>] iscsi_target_rx_thread+0x592/0xd80 [iscsi_target_mod]
Mar 16 04:53:54 l28 kernel: [521052.393909]  [<ffffffff8148b200>] ? __schedule+0x2e0/0x8e0
Mar 16 04:53:54 l28 kernel: [521052.394052]  [<ffffffffa00e6ae0>] ? iscsi_target_tx_thread+0x280/0x280 [iscsi_target_mod]
Mar 16 04:53:54 l28 kernel: [521052.394274]  [<ffffffff8105c5cb>] kthread+0xbb/0xc0
Mar 16 04:53:54 l28 kernel: [521052.394392]  [<ffffffff8105c510>] ? __kthread_parkme+0x80/0x80
Mar 16 04:53:54 l28 kernel: [521052.394508]  [<ffffffff8148d818>] ret_from_fork+0x58/0x90
Mar 16 04:53:54 l28 kernel: [521052.394619]  [<ffffffff8105c510>] ? __kthread_parkme+0x80/0x80
Mar 16 04:53:54 l28 kernel: [521052.394740] Code: 65 4c 03 04 25 48 bc 00 00 49 8b 50 08 4d 8b 28 49 8b 40 10 4d 85 ed 74 6b 48 85 c0 74 66 49 63 44 24 20 48 8d 4a 01 49 8b 3c 24 <49> 8b 5c 05 00 4c 89 e8 65 48 0f c7 0f 0f 94 c0 84 c0 74 bd 49 
Mar 16 04:53:54 l28 kernel: [521052.395596] RIP  [<ffffffff810ee519>] __kmalloc+0x69/0x100
Mar 16 04:53:54 l28 kernel: [521052.395721]  RSP <ffff8808162dfd18>
Mar 16 04:53:54 l28 kernel: [521052.395841] CR2: 0000000040008021
Mar 16 04:53:54 l28 kernel: [521052.396883] ---[ end trace 20dde4e477f78d24 ]---

>
> Kernel 3.12.51. Gdb listing:
>
> (gdb) list *(__kmalloc+0x69)
> 0xffffffff810ee519 is in __kmalloc (mm/slub.c:260).
> [...]
> 258     static inline void *get_freepointer(struct kmem_cache *s, void *object)
> 259     {
> 260             return *(void **)(object + s->offset);
> 261     }
>
> What whould be the next step? Thank you.

-- 
UNIX/Ocaml engineer at 1Gb.ru. Telegram: johnlepikhin