SLOB lockup (was: Re: [tip:core/locking] lockdep: annotate reclaimcontext (__GFP_NOFS), fix SLOB)

From: Ingo Molnar
Date: Sun Mar 15 2009 - 02:49:05 EST

Next message: Frans Pop: "Re: DRM oops while starting second X.org session in chroot"
Previous message: Uwe Kleine-König: "Re: some ARM fixes for 2.6.29-rc7-rt1"
Next in thread: Ingo Molnar: "Re: SLOB lockup (was: Re: [tip:core/locking] lockdep: annotatereclaim context (__GFP_NOFS), fix SLOB)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

* Ingo Molnar <mingo@xxxxxxx> wrote:

> Commit-ID: bf722c9d324864b4256edaa330751b77f2a19861
> Gitweb: http://git.kernel.org/tip/bf722c9d324864b4256edaa330751b77f2a19861
> Author: Ingo Molnar <mingo@xxxxxxx>
> AuthorDate: Sun, 15 Mar 2009 06:03:11 +0100
> Commit: Ingo Molnar <mingo@xxxxxxx>
> CommitDate: Sun, 15 Mar 2009 06:03:11 +0100
>
> lockdep: annotate reclaim context (__GFP_NOFS), fix SLOB
>
> Impact: build fix
>
> fix typo in mm/slob.c:
>
> mm/slob.c:469: error: âflagsâ undeclared (first use in this function)
> mm/slob.c:469: error: (Each undeclared identifier is reported only once
> mm/slob.c:469: error: for each function it appears in.)
>
> Cc: Nick Piggin <npiggin@xxxxxxx>
> Cc: Peter Zijlstra <a.p.zijlstra@xxxxxxxxx>
> LKML-Reference: <20090128135457.350751756@xxxxxxxxx>
> Signed-off-by: Ingo Molnar <mingo@xxxxxxx>

and with this fixed, and with SLOB now being tested in -tip, the
new lockdep assert attached below (followed by a real lockup)
pops up.

Seems like a genuine SLOB bug, probably present upstream as
well.

Ingo

--------------------------->

Remounting root filesystem in read-write mode: [ 137.292031] EXT3 FS on sda6, internal journal
[ OK ]
[ 137.324031]
[ 137.324031] =============================================
[ 137.324031] [ INFO: possible recursive locking detected ]
[ 137.324031] 2.6.29-rc8-tip #35362
[ 137.324031] ---------------------------------------------
[ 137.324031] rc.sysinit/5461 is trying to acquire lock:
[ 137.324031] (slob_lock){-.-.-.}, at: [<ffffffff8031d446>] slob_free+0x66/0x290
[ 137.324031]
[ 137.324031] but task is already holding lock:
[ 137.324031] (slob_lock){-.-.-.}, at: [<ffffffff8031d446>] slob_free+0x66/0x290
[ 137.324031]
[ 137.324031] other info that might help us debug this:
[ 137.324031] 3 locks held by rc.sysinit/5461:
[ 137.324031] #0: (&sb->s_type->i_mutex_key#6){+.+.+.}, at: [<ffffffff80320f0e>] do_truncate+0x5e/0x90
[ 137.324031] #1: (&sb->s_type->i_alloc_sem_key#3){+.+...}, at: [<ffffffff80338815>] notify_change+0x245/0x320
[ 137.324031] #2: (slob_lock){-.-.-.}, at: [<ffffffff8031d446>] slob_free+0x66/0x290
[ 137.324031]
[ 137.324031] stack backtrace:
[ 137.324031] Pid: 5461, comm: rc.sysinit Not tainted 2.6.29-rc8-tip #35362
[ 137.324031] Call Trace:
[ 137.324031] <IRQ> [<ffffffff802a24c0>] validate_chain+0xbd0/0x12c0
[ 137.324031] [<ffffffff802a1dc9>] ? validate_chain+0x4d9/0x12c0
[ 137.324031] [<ffffffff802a2f36>] __lock_acquire+0x386/0xb40
[ 137.324031] [<ffffffff802a378f>] lock_acquire+0x9f/0x140
[ 137.324031] [<ffffffff8031d446>] ? slob_free+0x66/0x290
[ 137.324031] [<ffffffff80f05553>] _spin_lock_irqsave+0x53/0x90
[ 137.324031] [<ffffffff8031d446>] ? slob_free+0x66/0x290
[ 137.324031] [<ffffffff8031d446>] slob_free+0x66/0x290
[ 137.324031] [<ffffffff8031d6a5>] __kmem_cache_free+0x35/0x40
[ 137.324031] [<ffffffff8031d6d4>] kmem_cache_free+0x24/0x70
[ 137.324031] [<ffffffff806b97ec>] free_object+0x6c/0xd0
[ 137.324031] [<ffffffff806b99e3>] __debug_check_no_obj_freed+0x193/0x1d0
[ 137.324031] [<ffffffff8029dcad>] ? trace_hardirqs_off+0xd/0x10
[ 137.324031] [<ffffffff806b9a37>] debug_check_no_obj_freed+0x17/0x20
[ 137.324031] [<ffffffff802f3619>] free_hot_cold_page+0x109/0x2b0
[ 137.324031] [<ffffffff802f3830>] free_hot_page+0x10/0x20
[ 137.324031] [<ffffffff802f3865>] __free_pages+0x25/0x40
[ 137.324031] [<ffffffff802f38cf>] free_pages+0x4f/0x60
[ 137.324031] [<ffffffff8031d530>] slob_free+0x150/0x290
[ 137.324031] [<ffffffff8031d6a5>] __kmem_cache_free+0x35/0x40
[ 137.324031] [<ffffffff8031d6d4>] kmem_cache_free+0x24/0x70
[ 137.324031] [<ffffffff80333765>] __d_free+0x45/0x70
[ 137.324031] [<ffffffff80333da5>] d_callback+0x15/0x20
[ 137.324031] [<ffffffff802cdd92>] rcu_process_callbacks+0x82/0xd0
[ 137.324031] [<ffffffff8027cfd2>] __do_softirq+0xa2/0x220
[ 137.324031] [<ffffffff8022f21c>] call_softirq+0x1c/0x30
[ 137.324031] [<ffffffff8023138a>] do_softirq+0x6a/0xb0
[ 137.324031] [<ffffffff8027ceb7>] irq_exit+0x97/0xa0
[ 137.324031] [<ffffffff80248104>] smp_apic_timer_interrupt+0x74/0xb0
[ 137.324031] [<ffffffff8022ec13>] apic_timer_interrupt+0x13/0x20
[ 137.324031] <EOI> [<ffffffff802a37a4>] ? lock_acquire+0xb4/0x140
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff80f039b8>] ? down_write+0x48/0x80
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff806610e9>] ? cap_inode_setattr+0x9/0x10
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff80320f1a>] ? do_truncate+0x6a/0x90
[ 137.324031] [<ffffffff8026bcbe>] ? sub_preempt_count+0xae/0xf0
[ 137.324031] [<ffffffff80661209>] ? cap_path_truncate+0x9/0x10
[ 137.324031] [<ffffffff8032dc34>] ? may_open+0x224/0x2b0
[ 137.324031] [<ffffffff8032e18c>] ? do_filp_open+0x17c/0x980
[ 137.324031] [<ffffffff8029dd2e>] ? put_lock_stats+0xe/0x30
[ 137.324031] [<ffffffff8026bcbe>] ? sub_preempt_count+0xae/0xf0
[ 137.324031] [<ffffffff80339236>] ? alloc_fd+0x116/0x140
[ 137.324031] [<ffffffff803203a3>] ? do_sys_open+0x63/0xf0
[ 137.324031] [<ffffffff80320470>] ? sys_open+0x20/0x30
[ 137.324031] [<ffffffff8022dfbb>] ? system_call_fastpath+0x16/0x1b
[ 137.324031] BUG: spinlock lockup on CPU#1, rcu_torture_rea/697, ffffffff813bf440
[ 137.324031] Pid: 697, comm: rcu_torture_rea Not tainted 2.6.29-rc8-tip #35362
[ 137.324031] Call Trace:
[ 137.324031] <IRQ> [<ffffffff806a8ed9>] ? delay_loop+0x9/0x40
[ 137.324031] [<ffffffff806b91be>] _raw_spin_lock+0x17e/0x190
[ 137.324031] [<ffffffff80f05573>] _spin_lock_irqsave+0x73/0x90
[ 137.324031] [<ffffffff8031d878>] ? slob_alloc+0x58/0x200
[ 137.324031] [<ffffffff8031d878>] slob_alloc+0x58/0x200
[ 137.324031] [<ffffffff8031dcde>] kmem_cache_alloc_node+0xbe/0xf0
[ 137.324031] [<ffffffff80d0f74e>] __alloc_skb+0x4e/0x150
[ 137.324031] [<ffffffff80d0a677>] sock_alloc_send_skb+0x1a7/0x200
[ 137.324031] [<ffffffff802cdc70>] ? __rcu_read_unlock+0x20/0xc0
[ 137.324031] [<ffffffff80d71a57>] ip_append_data+0x6f7/0xa60
[ 137.324031] [<ffffffff80d95d20>] ? icmp_glue_bits+0x0/0x70
[ 137.324031] [<ffffffff80d95c68>] icmp_push_reply+0x58/0x110
[ 137.324031] [<ffffffff80d95f0f>] icmp_reply+0x17f/0x1e0
[ 137.324031] [<ffffffff80e8f74a>] ? csum_partial+0xa/0x180
[ 137.324031] [<ffffffff80d9642d>] icmp_echo+0x5d/0x60
[ 137.324031] BUG: spinlock lockup on CPU#0, rc.sysinit/5461, ffffffff813bf440
[ 137.324031] Pid: 5461, comm: rc.sysinit Not tainted 2.6.29-rc8-tip #35362
[ 137.324031] Call Trace:
[ 137.324031] <IRQ> [<ffffffff806a8ed9>] ? delay_loop+0x9/0x40
[ 137.324031] [<ffffffff806b91be>] _raw_spin_lock+0x17e/0x190
[ 137.324031] [<ffffffff80f05573>] _spin_lock_irqsave+0x73/0x90
[ 137.324031] [<ffffffff8031d446>] ? slob_free+0x66/0x290
[ 137.324031] [<ffffffff8031d446>] slob_free+0x66/0x290
[ 137.324031] [<ffffffff8031d6a5>] __kmem_cache_free+0x35/0x40
[ 137.324031] [<ffffffff8031d6d4>] kmem_cache_free+0x24/0x70
[ 137.324031] [<ffffffff806b97ec>] free_object+0x6c/0xd0
[ 137.324031] [<ffffffff806b99e3>] __debug_check_no_obj_freed+0x193/0x1d0
[ 137.324031] [<ffffffff8029dcad>] ? trace_hardirqs_off+0xd/0x10
[ 137.324031] [<ffffffff806b9a37>] debug_check_no_obj_freed+0x17/0x20
[ 137.324031] [<ffffffff802f3619>] free_hot_cold_page+0x109/0x2b0
[ 137.324031] [<ffffffff802f3830>] free_hot_page+0x10/0x20
[ 137.324031] [<ffffffff802f3865>] __free_pages+0x25/0x40
[ 137.324031] [<ffffffff802f38cf>] free_pages+0x4f/0x60
[ 137.324031] [<ffffffff8031d530>] slob_free+0x150/0x290
[ 137.324031] [<ffffffff8031d6a5>] __kmem_cache_free+0x35/0x40
[ 137.324031] [<ffffffff8031d6d4>] kmem_cache_free+0x24/0x70
[ 137.324031] [<ffffffff80333765>] __d_free+0x45/0x70
[ 137.324031] [<ffffffff80333da5>] d_callback+0x15/0x20
[ 137.324031] [<ffffffff802cdd92>] rcu_process_callbacks+0x82/0xd0
[ 137.324031] [<ffffffff8027cfd2>] __do_softirq+0xa2/0x220
[ 137.324031] [<ffffffff8022f21c>] call_softirq+0x1c/0x30
[ 137.324031] [<ffffffff8023138a>] do_softirq+0x6a/0xb0
[ 137.324031] [<ffffffff8027ceb7>] irq_exit+0x97/0xa0
[ 137.324031] [<ffffffff80248104>] smp_apic_timer_interrupt+0x74/0xb0
[ 137.324031] [<ffffffff8022ec13>] apic_timer_interrupt+0x13/0x20
[ 137.324031] <EOI> [<ffffffff802a37a4>] ? lock_acquire+0xb4/0x140
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff80f039b8>] ? down_write+0x48/0x80
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff806610e9>] ? cap_inode_setattr+0x9/0x10
[ 137.324031] [<ffffffff80338815>] ? notify_change+0x245/0x320
[ 137.324031] [<ffffffff80320f1a>] ? do_truncate+0x6a/0x90
[ 137.324031] [<ffffffff8026bcbe>] ? sub_preempt_count+0xae/0xf0
[ 137.324031] [<ffffffff80661209>] ? cap_path_truncate+0x9/0x10
[ 137.324031] [<ffffffff8032dc34>] ? may_open+0x224/0x2b0
[ 137.324031] [<ffffffff8032e18c>] ? do_filp_open+0x17c/0x980
[ 137.324031] [<ffffffff8029dd2e>] ? put_lock_stats+0xe/0x30
[ 137.324031] [<ffffffff8026bcbe>] ? sub_preempt_count+0xae/0xf0
[ 137.324031] [<ffffffff80339236>] ? alloc_fd+0x116/0x140
[ 137.324031] [<ffffffff803203a3>] ? do_sys_open+0x63/0xf0
[ 137.324031] [<ffffffff80320470>] ? sys_open+0x20/0x30
[ 137.324031] [<ffffffff8022dfbb>] ? system_call_fastpath+0x16/0x1b
[ 137.324031] [<ffffffff80d11c9b>] ? __skb_checksum_complete_head+0x1b/0x70
[ 137.324031] [<ffffffff80d11d01>] ? __skb_checksum_complete+0x11/0x20
[ 137.324031] [<ffffffff80d960a2>] icmp_rcv+0x132/0x2e0
[ 137.324031] [<ffffffff80d6e526>] ip_local_deliver_finish+0x76/0x1e0
[ 137.324031] [<ffffffff80d6eb30>] ip_local_deliver+0x40/0xa0
[ 137.324031] [<ffffffff80d6e219>] ip_rcv_finish+0x129/0x3c0
[ 137.324031] [<ffffffff80d6ea0c>] ip_rcv+0x23c/0x320
[ 137.324031] [<ffffffff80d16c9c>] netif_receive_skb+0x2dc/0x540
[ 137.324031] [<ffffffff808f3efd>] nv_napi_poll+0x3cd/0x6c0
[ 137.324031] [<ffffffff80d19d5e>] net_rx_action+0x13e/0x210
[ 137.324031] [<ffffffff8024a071>] ? irq_complete_move+0x21/0x240
[ 137.324031] [<ffffffff8027cfd2>] __do_softirq+0xa2/0x220
[ 137.324031] [<ffffffff8022f21c>] call_softirq+0x1c/0x30
[ 137.324031] [<ffffffff8023138a>] do_softirq+0x6a/0xb0
[ 137.324031] [<ffffffff8027ceb7>] irq_exit+0x97/0xa0
[ 137.324031] [<ffffffff802306a5>] do_IRQ+0x95/0x110
[ 137.324031] [<ffffffff8022ea13>] ret_from_intr+0x0/0xf
[ 137.324031] <EOI> [<ffffffff80f05a26>] ? _spin_unlock_irq+0x36/0x60
[ 137.324031] [<ffffffff80f0186d>] ? thread_return+0x1e3/0x916
[ 137.324031] [<ffffffff8022eac0>] ? restore_args+0x0/0x30
[ 137.324031] [<ffffffff802cbff8>] ? rcu_torture_reader+0x188/0x2e0
[ 137.324031] [<ffffffff8029fc9d>] ? trace_hardirqs_on+0xd/0x10
[ 137.324031] [<ffffffff802cbff8>] ? rcu_torture_reader+0x188/0x2e0
[ 137.324031] [<ffffffff802cc9a0>] ? rcu_torture_timer+0x0/0x150
[ 137.324031] [<ffffffff80f059b7>] ? _spin_unlock_irqrestore+0x47/0x80
[ 137.324031] [<ffffffff802cbe70>] ? rcu_torture_reader+0x0/0x2e0
[ 137.324031] [<ffffffff8028e213>] ? kthread+0x53/0x80
[ 137.324031] [<ffffffff8022f11a>] ? child_rip+0xa/0x20
[ 137.324031] [<ffffffff8026bb68>] ? finish_task_switch+0x98/0x140
[ 137.324031] [<ffffffff80f05a2b>] ? _spin_unlock_irq+0x3b/0x60
[ 137.324031] [<ffffffff8022eac0>] ? restore_args+0x0/0x30
[ 137.324031] [<ffffffff8028e1c0>] ? kthread+0x0/0x80
[ 137.324031] [<ffffffff8022f110>] ? child_rip+0x0/0x20

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Frans Pop: "Re: DRM oops while starting second X.org session in chroot"
Previous message: Uwe Kleine-König: "Re: some ARM fixes for 2.6.29-rc7-rt1"
Next in thread: Ingo Molnar: "Re: SLOB lockup (was: Re: [tip:core/locking] lockdep: annotatereclaim context (__GFP_NOFS), fix SLOB)"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]