Re: linux-next: Tree for March 25 (Call trace: RCU|workqueues|block|VFS|ext4related?)

[Sedat Dilek]

On Sun, Mar 27, 2011 at 1:09 AM, Paul E. McKenney
<paulmck@xxxxxxxxxxxxxxxxxx> wrote:
> On Sat, Mar 26, 2011 at 11:15:22PM +0100, Sedat Dilek wrote:
>> On Sat, Mar 26, 2011 at 5:02 PM, Paul E. McKenney
>> <paulmck@xxxxxxxxxxxxxxxxxx> wrote:
>> > On Sat, Mar 26, 2011 at 01:34:53PM +0100, Sedat Dilek wrote:
>> [...]
>> >> Just FYI: Changed to the following settings:
>> >>
>> >> - Enable Preemptible Kernel (Low-Latency Desktop)
>> >> - Enable Preemptible tree-based hierarchical RCU
>> >> - Enable RCU priority boosting
>> >> - Reset RCU CPU stall timeout to default (60 seconds)
>> >>
>> >> So far I see no RCU stalls in the logs and my system runs as expected.
>> >> ( I have noticed here some "stalling" in the webbrowser, but I can do
>> >> my daily business. )
>> >
>> > OK, good to see some progress!
>> >
>>
>> On 1st impression thing went fine, but after a while jobs like opening
>> several tabs in firefox or doing a simple df command stalled the
>> machine. No, my system even got frozen and required a brutal reset.
>>
>> >From the logs (more see file-attachment):
>>
>> Mar 26 19:58:40 tbox kernel: [ 1440.640060] INFO: task systemd:1
>> blocked for more than 120 seconds.
>> Mar 26 19:58:40 tbox kernel: [ 1440.640074] "echo 0 >
>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>
>> Following it -> NOPE
>> $ echo 0 > /proc/sys/kernel/hung_task_timeout_secs
>
> These are tasks that are blocked, not tasks that are consuming CPU.
>
>> > Is there a runaway process consuming CPU? ÂThe reason that I ask is that
>> > an infinite loop in the kernel can result in a stall when PREEMPT=n
>> > but is less likely to if PREEMPT=y. ÂCould you please check with "top",
>> > "ps", or whatever?
>>
>> Unsure what you mean by this, as you can see from the logs, it's not
>> only one special task "stalling".
>> BTW, I have systemd here running.
>
> Right. ÂBut I need to know if there are tasks consuming lots of CPU,
> which is different than tasks that are stalled for a long time. ÂLook
> at the message: it says "blocked for more than 120 seconds", not
> "running for more than 120 seconds".
>
> For example, if one of the RCU kthreads is consuming lots of CPU, that
> would tell me that I should look for an RCU bug (and yank the patches
> from -next in the meantime). ÂOn the other hand, if some other task is
> consuming lots of CPU, then that would be a hint as to where to find
> the bug.
>
>> >> I am not sure what the change to PREEMPT exactly mean in the end.
>> >> ( Let's work with this new kernel and carefully check for possible
>> >> side-effects. )
>> >> For example CONFIG_RCU_FAST_NO_HZ=y is now dropped, where the Kconfig
>> >> descriptive text says some words on better energy saving. For a
>> >> notebook this is no good.
>> >
>> > CONFIG_RCU_FAST_NO_HZ is of no use on a uniprocessor system, so OK
>> > to disable it. ÂBut are you saying that CONFIG_RCU_FAST_NO_HZ=y
>> > results in problems that are removed by CONFIG_RCU_FAST_NO_HZ=n?
>> > That would be a surprise, and I need to know if this is the case.
>>
>> In my current setup (PREEMPT and TREE_PREEMPT_RCU)
>> CONFIG_RCU_FAST_NO_HZ is not considered and set via Kconfig-system
>> (see excerpt below).
>> But when you say for UP it is of no use, I need no more info.
>
> OK, "of no use" is overstating things a bit. ÂBut CONFIG_RCU_FAST_NO_HZ's
> main purpose is to get the last CPU into dyntick-idle state
> (CONFIG_NO_HZ), which is most useful if the system has several CPUs.
>
>> Might be good to add some recommended (and/or useless) kernel-config
>> settings to RCU/UP.txt?
>>
>> [ init/Kconfig ]
>> config RCU_FAST_NO_HZ
>> Â Â Â Â bool "Accelerate last non-dyntick-idle CPU's grace periods"
>> Â Â Â Â depends on TREE_RCU && NO_HZ && SMP
>> Â Â Â Â default n
>
> The "depends on TREE_RCU && NO_HZ && SMP" already excludes it from
> UP kernel builds, so no need to document.
>
>> >> I have also questions to some Kconfig dependencies, for example why I
>> >> can't select TREE_PREEMPT_RCU if CONFIG_PREEMPT_VOLUNTARY=y, etc.
>> >> Intended?
>> >
>> > Yes. ÂThere is no point in TREE_PREEMPT_RCU unless PREEMPT=y.
>> >
>>
>> OK.
>>
>> >> Maybe I collect all my askings in a separate email to RCU folks and ML
>> >> and do not disturb further people from other sub-trees.
>> >>
>> >> I enjoyed to read the numerous docs in Documentation/RCU/ (and noticed
>> >> some typos as well).
>> >> The RCU folk gave the word "FAQ" a new meaning: Frequenty Asked
>> >> Questions & Q*uiz* :-).
>> >>
>> >> Thanks for the helpful hints and explanations from the RCU folks!
>> >
>> > Glad you liked them! Â;-)
>> >
>>
>> Other sub-trees lack of no good or up2date docs.
>>
>> >> - Sedat -
>> >>
>> >> P.S.: Current RCU and HZ kernel-config settings
>> >>
>> >> # grep RCU /boot/config-$(uname -r)
>> >> # RCU Subsystem
>> >> CONFIG_TREE_PREEMPT_RCU=y
>> >> CONFIG_PREEMPT_RCU=y
>> >> CONFIG_RCU_TRACE=y
>> >> CONFIG_RCU_FANOUT=32
>> >> # CONFIG_RCU_FANOUT_EXACT is not set
>> >> CONFIG_TREE_RCU_TRACE=y
>> >> CONFIG_RCU_BOOST=y
>> >> CONFIG_RCU_BOOST_PRIO=1
>> >> CONFIG_RCU_BOOST_DELAY=500
>> >> # CONFIG_SPARSE_RCU_POINTER is not set
>> >> # CONFIG_RCU_TORTURE_TEST is not set
>> >> CONFIG_RCU_CPU_STALL_TIMEOUT=60
>> >> CONFIG_RCU_CPU_STALL_VERBOSE=y
>> >>
>> >> # grep _HZ /boot/config-$(uname -r)
>> >> CONFIG_NO_HZ=y
>> >> # CONFIG_HZ_100 is not set
>> >> CONFIG_HZ_250=y
>> >> # CONFIG_HZ_300 is not set
>> >> # CONFIG_HZ_1000 is not set
>> >> CONFIG_HZ=250
>> >
>> > OK, thank you for the info!
>> >
>>
>> N.P.
>>
>> > Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂThanx, Paul
>> >
>>
>> I guees I will revert step-by-step the RCU commits in linux-next and report.
>> This weekend I wanted to enhance Debian's live-cd framework with
>> overlayfs support and a customized kernel.
>> But then came RCU :-(.
>
> Well, if it turns out to be a problem in RCU I will certainly apologize.
>

No, that's not so dramatic.
Dealing with this RCU issue has nice side-effects: I remembered (and
finally did) to use a reduced kernel-config set.
The base for it I created with 'make localmodconfig' and did some
manual fine-tuning afterwards (throw out media, rc, dvd, unneeded FSs,
etc.).
Also, I can use fresh gcc-4.6 (4.6.0-1) from the official Debian repos.

So, I started building with
"revert-rcu-patches/0001-Revert-rcu-introduce-kfree_rcu.patch".
I will let you know.

- Sedat -

>> Can you say some words to kfree_rcu.2011.03.25b (rcu/kfree_rcu) GIT branch(es)?
>
> These are just applying Lia Jiangshan's kfree_rcu() to a number of places
> in the kernel. ÂYou can safely ignore them.
>
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂThanx, Paul
>
>> - Sedat -
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/