Re: [2.6.33-rc5] Weird deadlock when shutting down

[AmÃrico Wang]

On Thu, Mar 25, 2010 at 7:28 PM, David Howells <dhowells@xxxxxxxxxx> wrote:
> Benjamin Herrenschmidt <benh@xxxxxxxxxxxxxxxxxxx> wrote:
>
>> Johannes and I see this on our quad G5s... it -could- be similar to
>> one reported a short while ago by Xiaotian Feng <xtfeng@xxxxxxxxx>
>> under the subject [2.6.33-rc4] sysfs lockdep warnings on cpu hotplug.
>>
>> Basically, the machine deadlocks right after printing the following
>> when doing a shutdown:
>>
>> halt/4071 is trying to acquire lock:
>> Â(s_active){++++.+}, at: [<c0000000001ef868>] .sysfs_addrm_finish+0x58/0xc0
>>
>> but task is already holding lock:
>> Â(&per_cpu(cpu_policy_rwsem, cpu)){+.+.+.}, at: [<c0000000004cd6ac>] .lock_policy_rwsem_write+0x84/0xf4
>>
>> which lock already depends on the new lock.
>>
>> the existing dependency chain (in reverse order) is:
>>
>> <nothing else ... machine deadlocked here>
>
> I see this now, with full backtrace:
>
> =======================================================
> [ INFO: possible circular locking dependency detected ]
> 2.6.34-rc2-cachefs #115
> -------------------------------------------------------
> halt/2291 is trying to acquire lock:
> Â(s_active#31){++++.+}, at: [<ffffffff81104950>] sysfs_addrm_finish+0x31/0x5a
>
> but task is already holding lock:
> Â(&per_cpu(cpu_policy_rwsem, cpu)){+++++.}, at: [<ffffffff812a3a92>] lock_policy_rwsem_write+0x4a/0x7b
>
> which lock already depends on the new lock.
>


I already send a patch for this:

https://patchwork.kernel.org/patch/83558/

It is still in -mm tree, I think.

Thanks!

>
> the existing dependency chain (in reverse order) is:
>
> -> #1 (&per_cpu(cpu_policy_rwsem, cpu)){+++++.}:
> Â Â Â [<ffffffff81053da2>] __lock_acquire+0x1343/0x16cd
> Â Â Â [<ffffffff81054183>] lock_acquire+0x57/0x6d
> Â Â Â [<ffffffff813637b6>] down_write+0x3f/0x62
> Â Â Â [<ffffffff812a3a92>] lock_policy_rwsem_write+0x4a/0x7b
> Â Â Â [<ffffffff812a3b36>] store+0x39/0x79
> Â Â Â [<ffffffff81103589>] sysfs_write_file+0x103/0x13f
> Â Â Â [<ffffffff810afddc>] vfs_write+0xad/0x172
> Â Â Â [<ffffffff810aff5a>] sys_write+0x45/0x6c
> Â Â Â [<ffffffff81001eeb>] system_call_fastpath+0x16/0x1b
>
> -> #0 (s_active#31){++++.+}:
> Â Â Â [<ffffffff81053a59>] __lock_acquire+0xffa/0x16cd
> Â Â Â [<ffffffff81054183>] lock_acquire+0x57/0x6d
> Â Â Â [<ffffffff81104086>] sysfs_deactivate+0x8c/0xc9
> Â Â Â [<ffffffff81104950>] sysfs_addrm_finish+0x31/0x5a
> Â Â Â [<ffffffff81104a33>] sysfs_remove_dir+0x75/0x88
> Â Â Â [<ffffffff811bafae>] kobject_del+0x16/0x37
> Â Â Â [<ffffffff811bb00d>] kobject_release+0x3e/0x66
> Â Â Â [<ffffffff811bbd71>] kref_put+0x43/0x4d
> Â Â Â [<ffffffff811baf29>] kobject_put+0x47/0x4b
> Â Â Â [<ffffffff812a39b2>] __cpufreq_remove_dev+0x1da/0x236
> Â Â Â [<ffffffff8136178e>] cpufreq_cpu_callback+0x62/0x7a
> Â Â Â [<ffffffff81048362>] notifier_call_chain+0x32/0x5e
> Â Â Â [<ffffffff810483ed>] __raw_notifier_call_chain+0x9/0xb
> Â Â Â [<ffffffff81351736>] _cpu_down+0x90/0x29e
> Â Â Â [<ffffffff810311d3>] disable_nonboot_cpus+0x6f/0x105
> Â Â Â [<ffffffff8103f845>] kernel_power_off+0x21/0x3b
> Â Â Â [<ffffffff8103facd>] sys_reboot+0x103/0x16a
> Â Â Â [<ffffffff81001eeb>] system_call_fastpath+0x16/0x1b
>
> other info that might help us debug this:
>
> 4 locks held by halt/2291:
> Â#0: Â(reboot_mutex){+.+.+.}, at: [<ffffffff8103fa5b>] sys_reboot+0x91/0x16a
> Â#1: Â(cpu_add_remove_lock){+.+.+.}, at: [<ffffffff81031102>] cpu_maps_update_begin+0x12/0x14
> Â#2: Â(cpu_hotplug.lock){+.+.+.}, at: [<ffffffff8103113d>] cpu_hotplug_begin+0x27/0x4e
> Â#3: Â(&per_cpu(cpu_policy_rwsem, cpu)){+++++.}, at: [<ffffffff812a3a92>] lock_policy_rwsem_write+0x4a/0x7b
>
> stack backtrace:
> Pid: 2291, comm: halt Not tainted 2.6.34-rc2-cachefs #115
> Call Trace:
> Â[<ffffffff81052522>] print_circular_bug+0xae/0xbd
> Â[<ffffffff81053a59>] __lock_acquire+0xffa/0x16cd
> Â[<ffffffff81054183>] lock_acquire+0x57/0x6d
> Â[<ffffffff81104950>] ? sysfs_addrm_finish+0x31/0x5a
> Â[<ffffffff81044acf>] ? __init_waitqueue_head+0x35/0x46
> Â[<ffffffff81104086>] sysfs_deactivate+0x8c/0xc9
> Â[<ffffffff81104950>] ? sysfs_addrm_finish+0x31/0x5a
> Â[<ffffffff811044a3>] ? release_sysfs_dirent+0x9e/0xbe
> Â[<ffffffff81104950>] sysfs_addrm_finish+0x31/0x5a
> Â[<ffffffff81104a33>] sysfs_remove_dir+0x75/0x88
> Â[<ffffffff811bafae>] kobject_del+0x16/0x37
> Â[<ffffffff811bb00d>] kobject_release+0x3e/0x66
> Â[<ffffffff811bafcf>] ? kobject_release+0x0/0x66
> Â[<ffffffff811bbd71>] kref_put+0x43/0x4d
> Â[<ffffffff811baf29>] kobject_put+0x47/0x4b
> Â[<ffffffff812a39b2>] __cpufreq_remove_dev+0x1da/0x236
> Â[<ffffffff8136178e>] cpufreq_cpu_callback+0x62/0x7a
> Â[<ffffffff81048362>] notifier_call_chain+0x32/0x5e
> Â[<ffffffff810483ed>] __raw_notifier_call_chain+0x9/0xb
> Â[<ffffffff81351736>] _cpu_down+0x90/0x29e
> Â[<ffffffff810311d3>] disable_nonboot_cpus+0x6f/0x105
> Â[<ffffffff8103f845>] kernel_power_off+0x21/0x3b
> Â[<ffffffff8103facd>] sys_reboot+0x103/0x16a
> Â[<ffffffff813636d9>] ? do_nanosleep+0x78/0xb2
> Â[<ffffffff8104797d>] ? hrtimer_nanosleep+0xab/0x118
> Â[<ffffffff810473a6>] ? hrtimer_wakeup+0x0/0x21
> Â[<ffffffff81364e29>] ? retint_swapgs+0xe/0x13
> Â[<ffffffff81051e4e>] ? trace_hardirqs_on_caller+0x10c/0x130
> Â[<ffffffff8107496a>] ? audit_syscall_entry+0x17d/0x1b0
> Â[<ffffffff81364354>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> Â[<ffffffff81001eeb>] system_call_fastpath+0x16/0x1b
> Broke affinity for irq 4
> lockdep: fixing up alternatives.
> SMP alternatives: switching to UP code
> Power down.
> acpi_power_off called
>
>
> David
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html
> Please read the FAQ at Âhttp://www.tux.org/lkml/
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/