Re: sched_wakeup_new and sched_kthread_stop events cause greatoverload

[Steven Rostedt]

On Thu, 2010-03-25 at 17:36 +0800, Xiao Guangrong wrote:
> We have done sysbench test for ftrace's performance and it looks sched_wakeup_new
> and sched_kthread_stop events can cause great overload.
> 
> When we only enable sched_wakeup_new and sched_kthread_stop events, sysbench.threads
> shows the overload is 10%, sysbench.mutex shows the overload is 7.5%.
> 
> The more weird thing is that we found the sched_kthread_stop event is never called
> in this test, the test steps as follow:
> 
> echo > debugfs/tracing/set_event
> echo 1 > debugfs/tracing/events/sched/sched_wakeup_new/enable
> echo 1 > debugfs/tracing/events/sched/sched_kthread_stop/enable
> 
> com_opt="--num-threads=5000 --max-requests=50000"
> echo > debugfs/tracing/trace
> sysbench $com_opt --test=threads --thread-yields=1000 --thread-locks=10000 run >& log
> [or sysbench $com_opt --test=mutex --mutex-num=100 --mutex-locks=50000 --mutex-loops=10000 run for mutex]
> echo > debugfs/tracing/set_event
> 
> For sysbench.threads:
> cat debugfs/tracing/trace | grep "sched_wakeup_new" | wc -l
> 5001
> cat debugfs/tracing/trace | grep "sched_kthread_stop" | wc -l
> 0

Strange? So if you did:

cat debugfs/tracing/trace | wc -l

you should get 5005?

> 
> For sysbench.mutex:
> cat debugfs/tracing/trace | grep "sched_wakeup_new" | wc -l
> 5001
> cat debugfs/tracing/trace | grep "sched_kthread_stop" | wc -l
> 0
> 
> And, if only enable sched_kthread_stop event, the sysbench.threads's
> overload is 5.90%, the sysbench.mutex's overload is 3.36%.

Could be something that messes with the cache lines?

> 
> It hardly explain why sched_kthread_stop is never called but cause great overload.

Would be worth running perf against this test, to see where the issues
are occurring.

-- Steve


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/