Re: [Ocfs2-users] [OCFS2] Crash at o2net_shutdown_sc()

From: richard -rw- weinberger
Date: Sat Mar 02 2013 - 03:42:18 EST

Next message: Wang YanQing: "Re: [PATCH]serial: 8250: Fix detect XScale port wrong"
Previous message: Emmanuel Benisty: "Re: [RFC PATCH 0/2] ipc: do not hold ipc lock more than necessary"
In reply to: Srinivas Eeda: "Re: [Ocfs2-users] [OCFS2] Crash at o2net_shutdown_sc()"
Next in thread: Tejun Heo: "Re: [Ocfs2-users] [OCFS2] Crash at o2net_shutdown_sc()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Fri, Mar 1, 2013 at 10:42 PM, Srinivas Eeda <srinivas.eeda@xxxxxxxxxx> wrote:
> Yes that was the crash I was referring to which stopped me from testing my
> other patch on mainline. I think the crashes started since some workqueue
> patches introduced by commit 57b30ae77bf00d2318df711ef9a4d2a9be0a3a2a
> Earlier kernels should be fine.
>
> Patch https://lkml.org/lkml/2012/10/18/592 tried to address one fix which
> helped ramster that uses same ocfs2/o2net code. There still seems to be
> another problem that crashes ocfs2.

If commit 57b30ae (workqueue: reimplement cancel_delayed_work() using
try_to_grab_pending())
introduced that regression, it is time to CC Tejun.

>
> On 03/01/2013 01:25 PM, richard -rw- weinberger wrote:
>>
>> Hi!
>>
>> Using 3.8.1 OCFS2 crashes while joining nodes to the cluster.
>> The cluster consists of 10 nodes, while node3 joins the kernel on node3
>> crashes.
>> (Somtimes later...)
>> See dmesg below.
>> Is this a known issue? I didn't test older kernels so far.
>>
>> node1:
>> [ 1471.881922] o2dlm: Joining domain 1352E2692E704EEB8040E5B8FF560997
>> ( 0 ) 1 nodes
>> [ 1471.919522] JBD2: Ignoring recovery information on journal
>> [ 1471.947027] ocfs2: Mounting device (253,16) on (node 0, slot 0)
>> with ordered data mode.
>> [ 1475.802497] o2net: Accepted connection from node node2 (num 1) at
>> 192.168.66.2:7777
>> [ 1481.814048] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 8
>> [ 1481.814955] o2net: No longer connected to node node2 (num 1) at
>> 192.168.66.2:7777
>> [ 1482.468827] o2net: Accepted connection from node node3 (num 2) at
>> 192.168.66.3:7777
>> [ 1511.904100] o2net: No connection established with node 1 after 30.0
>> seconds, giving up.
>> [ 1514.472995] o2net: Connection to node node3 (num 2) at
>> 192.168.66.3:7777 shutdown, state 8
>> [ 1514.473960] o2net: No longer connected to node node3 (num 2) at
>> 192.168.66.3:7777
>> [ 1516.076044] o2net: Accepted connection from node node2 (num 1) at
>> 192.168.66.2:7777
>> [ 1520.181430] o2dlm: Node 1 joins domain
>> 1352E2692E704EEB8040E5B8FF560997 ( 0 1 ) 2 nodes
>> [ 1544.544030] o2net: No connection established with node 2 after 30.0
>> seconds, giving up.
>> [ 1574.624029] o2net: No connection established with node 2 after 30.0
>> seconds, giving up.
>>
>> node2:
>> [ 1475.613170] o2net: Connected to node node1 (num 0) at 192.168.66.1:7777
>> [ 1481.620253] o2hb: Unable to stabilize heartbeart on region
>> 1352E2692E704EEB8040E5B8FF560997 (vdb)
>> [ 1481.622489] o2net: No longer connected to node node1 (num 0) at
>> 192.168.66.1:7777
>> [ 1515.886605] o2net: Connected to node node1 (num 0) at 192.168.66.1:7777
>> [ 1519.992766] o2dlm: Joining domain 1352E2692E704EEB8040E5B8FF560997
>> ( 0 1 ) 2 nodes
>> [ 1520.017054] JBD2: Ignoring recovery information on journal
>> [ 1520.078888] ocfs2: Mounting device (253,16) on (node 1, slot 1)
>> with ordered data mode.
>> [ 1520.159590] mount.ocfs2 (2186) used greatest stack depth: 2568 bytes
>> left
>>
>> node3:
>> [ 1482.836865] o2net: Connected to node node1 (num 0) at 192.168.66.1:7777
>> [ 1482.837542] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1484.840952] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1486.844994] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1488.848952] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1490.853052] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1492.857046] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1494.861042] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1496.865024] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1498.869021] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1500.873016] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1502.877056] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1504.881042] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1506.885040] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1508.888991] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1510.893077] o2net: Connection to node node2 (num 1) at
>> 192.168.66.2:7777 shutdown, state 7
>> [ 1512.843172] (mount.ocfs2,2179,0):dlm_request_join:1477 ERROR: Error
>> -107 when sending message 510 (key 0x666c6172) to node 1
>> [ 1512.845580] (mount.ocfs2,2179,0):dlm_try_to_join_domain:1653 ERROR:
>> status = -107
>> [ 1512.847778] (mount.ocfs2,2179,0):dlm_join_domain:1955 ERROR: status =
>> -107
>> [ 1512.849334] (mount.ocfs2,2179,0):dlm_register_domain:2214 ERROR:
>> status = -107
>> [ 1512.850921] (mount.ocfs2,2179,0):o2cb_cluster_connect:368 ERROR:
>> status = -107
>> [ 1512.852511] (mount.ocfs2,2179,0):ocfs2_dlm_init:3004 ERROR: status =
>> -107
>> [ 1512.854090] (mount.ocfs2,2179,0):ocfs2_mount_volume:1881 ERROR: status
>> = -107
>> [ 1512.855476] ocfs2: Unmounting device (253,16) on (node 0)
>> [ 1512.855915] (mount.ocfs2,2179,0):ocfs2_fill_super:1230 ERROR: status =
>> -107
>> [ 1514.839138] o2net: No longer connected to node node1 (num 0) at
>> 192.168.66.1:7777
>> [ 1514.840690] BUG: unable to handle kernel NULL pointer dereference
>> at 0000000000000028
>> [ 1514.841627] IP: [<ffffffff816ce840>] kernel_sock_ioctl+0x50/0x50
>> [ 1514.841627] PGD 1d980067 PUD 1db0c067 PMD 0
>> [ 1514.841627] Oops: 0000 [#1] PREEMPT SMP
>> [ 1514.841627] Modules linked in:
>> [ 1514.841627] CPU 0
>> [ 1514.841627] Pid: 6, comm: kworker/u:0 Not tainted 3.8.1+ #8 Bochs Bochs
>> [ 1514.841627] RIP: 0010:[<ffffffff816ce840>] [<ffffffff816ce840>]
>> kernel_sock_ioctl+0x50/0x50
>> [ 1514.841627] RSP: 0018:ffff88001f949d40 EFLAGS: 00010292
>> [ 1514.841627] RAX: ffff88001e6e9c01 RBX: ffff88001e6e9000 RCX:
>> 0000000180080006
>> [ 1514.841627] RDX: 0000000180080007 RSI: 0000000000000002 RDI:
>> 0000000000000000
>> [ 1514.841627] RBP: ffff88001f949dc8 R08: 0000000000016090 R09:
>> 0000000000000001
>> [ 1514.841627] R10: ffffea000079ba40 R11: ffffffff81321207 R12:
>> ffff88001db69440
>> [ 1514.841627] R13: ffffffff8204c760 R14: ffff88001db695a8 R15:
>> ffff88001e6e9058
>> [ 1514.841627] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000)
>> knlGS:0000000000000000
>> [ 1514.841627] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 1514.841627] CR2: 0000000000000028 CR3: 000000001c009000 CR4:
>> 00000000000006f0
>> [ 1514.841627] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [ 1514.841627] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [ 1514.841627] Process kworker/u:0 (pid: 6, threadinfo
>> ffff88001f948000, task ffff88001f90a670)
>> [ 1514.841627] Stack:
>> [ 1514.841627] ffffffff81323776 ffff88001fad2e20 0000000000010500
>> ffff88001f949d98
>> [ 1514.841627] ffffffff810013fa ffff88001f949d78 ffffffff818a5c22
>> ffff88001f949da8
>> [ 1514.841627] ffffffff81069d26 0000000000012900 0000000000012900
>> 0000000000000000
>> [ 1514.841627] Call Trace:
>> [ 1514.841627] [<ffffffff81323776>] ? o2net_shutdown_sc+0x106/0x1e0
>> [ 1514.841627] [<ffffffff810013fa>] ? __switch_to+0x2a/0x4a0
>> [ 1514.841627] [<ffffffff818a5c22>] ? _raw_spin_unlock_irq+0x12/0x40
>> [ 1514.841627] [<ffffffff81069d26>] ? finish_task_switch+0x56/0xc0
>> [ 1514.841627] [<ffffffff81056eb3>] process_one_work+0x133/0x510
>> [ 1514.841627] [<ffffffff81323670>] ?
>> o2net_sc_connect_completed+0xf0/0xf0
>> [ 1514.841627] [<ffffffff810585ed>] worker_thread+0x15d/0x450
>> [ 1514.841627] [<ffffffff81058490>] ? busy_worker_rebind_fn+0x100/0x100
>> [ 1514.841627] [<ffffffff8105e10b>] kthread+0xbb/0xc0
>> [ 1514.841627] [<ffffffff818a0000>] ? e1000_regdump+0x262/0x3be
>> [ 1514.841627] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130
>> [ 1514.841627] [<ffffffff818accac>] ret_from_fork+0x7c/0xb0
>> [ 1514.841627] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130
>> [ 1514.841627] Code: ff ff ff ff ff ff 48 8b 47 28 ff 50 48 4c 89 a3
>> 48 e0 ff ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 66 66 2e 0f 1f 84 00
>> 00 00 00 00 <48> 8b 47 28 55 48 89 e5 ff 50 60 5d c3 0f 1f 00 55 41 b8
>> 4b 43
>> [ 1514.841627] RIP [<ffffffff816ce840>] kernel_sock_ioctl+0x50/0x50
>> [ 1514.841627] RSP <ffff88001f949d40>
>> [ 1514.841627] CR2: 0000000000000028
>> [ 1514.867242] ---[ end trace 36ffe9378168cdc2 ]---
>> [ 1514.867619] BUG: unable to handle kernel paging request at
>> ffffffffffffffd8
>> [ 1514.868007] IP: [<ffffffff8105e3cb>] kthread_data+0xb/0x20
>> [ 1514.868007] PGD 1e0d067 PUD 1e0e067 PMD 0
>> [ 1514.868007] Oops: 0000 [#2] PREEMPT SMP
>> [ 1514.868007] Modules linked in:
>> [ 1514.868007] CPU 0
>> [ 1514.868007] Pid: 6, comm: kworker/u:0 Tainted: G D 3.8.1+
>> #8 Bochs Bochs
>> [ 1514.868007] RIP: 0010:[<ffffffff8105e3cb>] [<ffffffff8105e3cb>]
>> kthread_data+0xb/0x20
>> [ 1514.868007] RSP: 0018:ffff88001f949928 EFLAGS: 00010092
>> [ 1514.868007] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
>> 0000000000000006
>> [ 1514.868007] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
>> ffff88001f90a670
>> [ 1514.868007] RBP: ffff88001f949928 R08: 0000000000000001 R09:
>> 0000000000000196
>> [ 1514.868007] R10: ffff88001f9116a0 R11: 0000000000000004 R12:
>> ffff88001f90aa40
>> [ 1514.868007] R13: 0000000000000000 R14: ffff88001f90a660 R15:
>> ffff88001f90a670
>> [ 1514.868007] FS: 0000000000000000(0000) GS:ffff88001fc00000(0000)
>> knlGS:0000000000000000
>> [ 1514.868007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>> [ 1514.868007] CR2: ffffffffffffffd8 CR3: 000000001d83d000 CR4:
>> 00000000000006f0
>> [ 1514.868007] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [ 1514.868007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [ 1514.868007] Process kworker/u:0 (pid: 6, threadinfo
>> ffff88001f948000, task ffff88001f90a670)
>> [ 1514.868007] Stack:
>> [ 1514.868007] ffff88001f949948 ffffffff81059980 ffff88001f949948
>> ffff88001fc12900
>> [ 1514.868007] ffff88001f9499c8 ffffffff818a4c48 ffff88001f949978
>> 0000000000000000
>> [ 1514.868007] ffff88001f949fd8 ffff88001f90a670 ffff88001f949fd8
>> ffff88001f949fd8
>> [ 1514.868007] Call Trace:
>> [ 1514.868007] [<ffffffff81059980>] wq_worker_sleeping+0x10/0xc0
>> [ 1514.868007] [<ffffffff818a4c48>] __schedule+0x528/0x770
>> [ 1514.868007] [<ffffffff818a4fa4>] schedule+0x24/0x70
>> [ 1514.868007] [<ffffffff81040c72>] do_exit+0x702/0xad0
>> [ 1514.868007] [<ffffffff818a7071>] oops_end+0x91/0xe0
>> [ 1514.868007] [<ffffffff8189a00d>] no_context+0x24e/0x279
>> [ 1514.868007] [<ffffffff817436f8>] ? tcp_rearm_rto+0xa8/0xd0
>> [ 1514.868007] [<ffffffff8189a1ed>] __bad_area_nosemaphore+0x1b5/0x1d4
>> [ 1514.868007] [<ffffffff8189a21a>] bad_area_nosemaphore+0xe/0x10
>> [ 1514.868007] [<ffffffff818a90b6>] __do_page_fault+0x296/0x490
>> [ 1514.868007] [<ffffffff816ce8ce>] ? sock_destroy_inode+0x2e/0x40
>> [ 1514.868007] [<ffffffff8114e787>] ? destroy_inode+0x37/0x60
>> [ 1514.868007] [<ffffffff8114e8ca>] ? evict+0x11a/0x1b0
>> [ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160
>> [ 1514.868007] [<ffffffff8112d421>] ? kfree+0x121/0x160
>> [ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160
>> [ 1514.868007] [<ffffffff818a92b9>] do_page_fault+0x9/0x10
>> [ 1514.868007] [<ffffffff818a65e2>] page_fault+0x22/0x30
>> [ 1514.868007] [<ffffffff81321207>] ? sc_kref_release+0x77/0x160
>> [ 1514.868007] [<ffffffff816ce840>] ? kernel_sock_ioctl+0x50/0x50
>> [ 1514.868007] [<ffffffff81323776>] ? o2net_shutdown_sc+0x106/0x1e0
>> [ 1514.868007] [<ffffffff810013fa>] ? __switch_to+0x2a/0x4a0
>> [ 1514.868007] [<ffffffff818a5c22>] ? _raw_spin_unlock_irq+0x12/0x40
>> [ 1514.868007] [<ffffffff81069d26>] ? finish_task_switch+0x56/0xc0
>> [ 1514.868007] [<ffffffff81056eb3>] process_one_work+0x133/0x510
>> [ 1514.868007] [<ffffffff81323670>] ?
>> o2net_sc_connect_completed+0xf0/0xf0
>> [ 1514.868007] [<ffffffff810585ed>] worker_thread+0x15d/0x450
>> [ 1514.868007] [<ffffffff81058490>] ? busy_worker_rebind_fn+0x100/0x100
>> [ 1514.868007] [<ffffffff8105e10b>] kthread+0xbb/0xc0
>> [ 1514.868007] [<ffffffff818a0000>] ? e1000_regdump+0x262/0x3be
>> [ 1514.868007] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130
>> [ 1514.868007] [<ffffffff818accac>] ret_from_fork+0x7c/0xb0
>> [ 1514.868007] [<ffffffff8105e050>] ? kthread_create_on_node+0x130/0x130
>> [ 1514.868007] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01
>> c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 78 03 00 00
>> 55 48 89 e5 <48> 8b 40 d8 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
>> 00 00
>> [ 1514.868007] RIP [<ffffffff8105e3cb>] kthread_data+0xb/0x20
>> [ 1514.868007] RSP <ffff88001f949928>
>> [ 1514.868007] CR2: ffffffffffffffd8
>> [ 1514.868007] ---[ end trace 36ffe9378168cdc3 ]---
>> [ 1514.868007] Fixing recursive fault but reboot is needed!
>>
>

--
Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Wang YanQing: "Re: [PATCH]serial: 8250: Fix detect XScale port wrong"
Previous message: Emmanuel Benisty: "Re: [RFC PATCH 0/2] ipc: do not hold ipc lock more than necessary"
In reply to: Srinivas Eeda: "Re: [Ocfs2-users] [OCFS2] Crash at o2net_shutdown_sc()"
Next in thread: Tejun Heo: "Re: [Ocfs2-users] [OCFS2] Crash at o2net_shutdown_sc()"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]