Re: mmotm 2010-03-09-19-15: Lot of scheduling while atomicwarnings related to RCU

From: Mel Gorman
Date: Wed Mar 10 2010 - 15:34:23 EST

Next message: Jack Steiner: "[PATCH] x86, UV: Cleanup of UV header for MMR definitions"
Previous message: Frederic Weisbecker: "Re: [RFC PATCH] perf: Store relevant events in a hlist"
In reply to: Andrew Morton: "Re: mmotm 2010-03-09-19-15: Lot of scheduling while atomic warningsrelated to RCU"
Next in thread: Neil Horman: "Re: mmotm 2010-03-09-19-15: Lot of scheduling while atomic warningsrelated to RCU"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, Mar 10, 2010 at 12:21:38PM -0800, Andrew Morton wrote:
> On Wed, 10 Mar 2010 19:57:46 +0000
> Mel Gorman <mel@xxxxxxxxx> wrote:
>
> > On Tue, Mar 09, 2010 at 07:16:03PM -0800, akpm@xxxxxxxxxxxxxxxxxxxx wrote:
> > > The mm-of-the-moment snapshot 2010-03-09-19-15 has been uploaded to
> > >
> > > http://userweb.kernel.org/~akpm/mmotm/
> > >
> > > and will soon be available at
> > >
> > > git://zen-kernel.org/kernel/mmotm.git
> > >
> > > It contains the following patches against 2.6.34-rc1:
> > >
> >
> > With preempt and sleeping spinlock debugging on, I'm seeing a lot of warnings
> > about scheduling while atomic. Starts off with messages like
> >
> > [ 0.572278] BUG: scheduling while atomic: swapper/1/0x00000002
> > [ 0.576004] 1 lock held by swapper/1:
> > [ 0.580002] #0: (rcu_read_lock){.+.+.+}, at: [<ffffffff811b6950>] kobject_uevent_env+0x363/0x4d0
> > [ 0.600001] Modules linked in:
> > [ 0.608003] Pid: 1, comm: swapper Not tainted 2.6.34-rc1-mm1-vanilla #1
> > [ 0.612001] Call Trace:
> > [ 0.616005] [<ffffffff810727fe>] ? __debug_show_held_locks+0x22/0x24
> > [ 0.620005] [<ffffffff810395ed>] __schedule_bug+0x77/0x7c
> > [ 0.624004] [<ffffffff812f4cc5>] schedule+0xcc/0x723
> > [ 0.628003] [<ffffffff812f55b6>] schedule_timeout+0x2d/0x28d
> > [ 0.632003] [<ffffffff8107315d>] ? mark_held_locks+0x4d/0x6b
> > [ 0.636004] [<ffffffff812f81e9>] ? _raw_spin_unlock_irq+0x30/0x5d
> > [ 0.640003] [<ffffffff810733de>] ? trace_hardirqs_on_caller+0x110/0x134
> > [ 0.644003] [<ffffffff812f4a0f>] wait_for_common+0xdc/0x152
> > [ 0.648003] [<ffffffff8103a43b>] ? default_wake_function+0x0/0x14
> > [ 0.652004] [<ffffffff812f4b28>] wait_for_completion+0x1d/0x1f
> > [ 0.656004] [<ffffffff8105cdf0>] call_usermodehelper_exec+0xad/0xfa
> > [ 0.660004] [<ffffffff81068429>] ? prepare_usermodehelper_creds+0x136/0x168
> > [ 0.664003] [<ffffffff811b6a37>] kobject_uevent_env+0x44a/0x4d0
> > [ 0.668003] [<ffffffff811b6950>] ? kobject_uevent_env+0x363/0x4d0
> > [ 0.672003] [<ffffffff811b6ac8>] kobject_uevent+0xb/0xd
> > [ 0.676002] [<ffffffff811b5e65>] kset_register+0x37/0x3f
> > [ 0.680004] [<ffffffff8123239a>] bus_register+0x11f/0x2b1
> > [ 0.684005] [<ffffffff8175973e>] platform_bus_init+0x2c/0x44
> > [ 0.688003] [<ffffffff817597b9>] driver_init+0x1d/0x29
> > [ 0.692004] [<ffffffff8172c69b>] kernel_init+0x12e/0x1e3
> > [ 0.696003] [<ffffffff81003be4>] kernel_thread_helper+0x4/0x10
> > [ 0.700004] [<ffffffff812f8794>] ? restore_args+0x0/0x30
> > [ 0.704003] [<ffffffff8172c56d>] ? kernel_init+0x0/0x1e3
> > [ 0.708002] [<ffffffff81003be0>] ? kernel_thread_helper+0x0/0x10
> >
> > There are a lot of warnings along these lines during boot. This system did
> > boot *eventually* but it was in obvious trouble. The warnings appear to
> > be all around kobject_uevent_env acquiring the RCU lock and then going to
> > sleep. Most likely candidate problem patches in mmotm are
> >
> > sysctl-fix-up-remaining-references-to-uevent_helper
> > sysctl-fix-up-remaining-references-to-uevent_helper-fix
> >
> > But there are also a whole host of other sysctl-related changes around RCU in
> > there. The changelog implies that these patches are part of some long-lived
> > review process so I didn't delve too deeply. The people most likely to be
> > involved based on comments in the changelog are cc'd.
> >
>
> Thanks. It's Neil's
> sysctl-fix-up-remaining-references-to-uevent_helper-fix.patch which does
>
> --- a/lib/kobject_uevent.c~sysctl-fix-up-remaining-references-to-uevent_helper-fix
> +++ a/lib/kobject_uevent.c
> @@ -273,10 +273,11 @@ int kobject_uevent_env(struct kobject *k
> #endif
>
> /* call uevent_helper, usually only enabled during early boot */
> + rcu_read_lock();
> helper = rcu_dereference(uevent_helper);
> if (helper[0])
> retval = uevent_call_helper(subsystem, env);
> -
> + rcu_read_unlock();
> exit:
> kfree(devpath);
> kfree(env);
>
>

Indeed, I spotted that but the large number of dependencies around it made
me wonder was it really that patch or a problem with the whole thing. I
confess I didn't actually look to see what they were doing. Shortly after
I sent the first mail, I removed all these patches from the series just so
it would boot without crapping all over the console.

> Ho hum. I guess all of
>
> #rcu-add-rcustring-adt-for-rcu-protected-strings.patch: TBU
> rcu-add-rcustring-adt-for-rcu-protected-strings.patch
> #add-a-kernel_address-that-works-for-data-too.patch: TBU
> add-a-kernel_address-that-works-for-data-too.patch
> sysctl-add-proc_rcu_string-to-manage-sysctls-using-rcu-strings.patch
> sysctl-use-rcu-strings-for-core_pattern-sysctl.patch
> sysctl-add-call_usermodehelper_cleanup.patch
> sysctl-convert-modprobe_path-to-proc_rcu_string.patch
> sysctl-convert-poweroff_command-to-proc_rcu_string.patch
> #sysctl-convert-hotplug-helper-string-to-proc_rcu_string.patch: jirislaby crash
> sysctl-convert-hotplug-helper-string-to-proc_rcu_string.patch
> #sysctl-use-rcu-protected-sysctl-for-ocfs-group-add-helper.patch: Joel nack
> sysctl-use-rcu-protected-sysctl-for-ocfs-group-add-helper.patch
> #
> sysctl-fix-up-remaining-references-to-uevent_helper.patch
> sysctl-fix-up-remaining-references-to-uevent_helper-fix.patch
> #
> sysctl-clean-up-vm-related-variable-declarations.patch
> sysctl-clean-up-vm-related-variable-declarations-fix.patch
> #
> sysctl-remove-references-to-ctl_unnumbered-which-has-been-removed.patch
> #
> #
> sysctl-extern-cleanup-c_a_d.patch
> sysctl-extern-cleanup-signal.patch
> sysctl-extern-cleanup-binfmts.patch
> sysctl-extern-cleanup-pid.patch
> sysctl-extern-cleanup-mm.patch
> sysctl-extern-cleanup-compat.patch
> sysctl-extern-cleanup-latencytop.patch
> sysctl-extern-cleanup-file-nr.patch
> sysctl-extern-cleanup-rcu.patch
> sysctl-extern-cleanup-block-iopoll.patch
> sysctl-extern-cleanup-module.patch
> sysctl-extern-cleanup-sg.patch
> sysctl-extern-cleanup-acct.patch
> sysctl-extern-cleanup-rtmutex.patch
> sysctl-extern-cleanup-lockdep.patch
> sysctl-extern-cleanup-poll.patch
> #
> kmod-add-init-function-to-usermodehelper.patch
> kmod-add-init-function-to-usermodehelper-fix.patch
> kmod-replace-call_usermodehelper_pipe-with-use-of-umh-init-function-and-resolve-limit.patch
> kmod-replace-call_usermodehelper_pipe-with-use-of-umh-init-function-and-resolve-limitcleanup.patch
>
> are not a prospect for 2.6.34. There are issues in the rcustring
> patches and the others have dependencies. I suppose I can rebase
> sysctl-extern-cleanup-* though.
>

--
Mel Gorman
Part-time Phd Student Linux Technology Center
University of Limerick IBM Dublin Software Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Jack Steiner: "[PATCH] x86, UV: Cleanup of UV header for MMR definitions"
Previous message: Frederic Weisbecker: "Re: [RFC PATCH] perf: Store relevant events in a hlist"
In reply to: Andrew Morton: "Re: mmotm 2010-03-09-19-15: Lot of scheduling while atomic warningsrelated to RCU"
Next in thread: Neil Horman: "Re: mmotm 2010-03-09-19-15: Lot of scheduling while atomic warningsrelated to RCU"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]