Re: [rcu] 5057f55e543: dmesg.BUG:soft_lockup-CPU_stuck_for_s

From: Fengguang Wu
Date: Mon Oct 06 2014 - 05:17:25 EST

On Mon, Oct 06, 2014 at 01:54:56PM +0800, Fengguang Wu wrote:
> On Mon, Oct 06, 2014 at 01:50:24PM +0800, Fengguang Wu wrote:
> > Hi Paul,
> >
> > FYI, we noticed a number of ups and downs for commit
> >
> > 5057f55e543b7859cfd26bc281291795eac93f8a ("rcu: Bind RCU grace-period kthreads if NO_HZ_FULL")
>
> Here is an overview of the performance/power/latency/kernel size
> index. The baseline (71a9b26963f8c2d) number is 100, the larger, the better.
>
> 96 perf-index 5057f55e543b7859cfd26bc281291795eac93f8a
> 99 power-index 5057f55e543b7859cfd26bc281291795eac93f8a
> 101 latency-index 5057f55e543b7859cfd26bc281291795eac93f8a
> 102 size-index 5057f55e543b7859cfd26bc281291795eac93f8a

The performance changes seem to have a strong correlation with the
time.involuntary_context_switches changes.

71a9b26963f8c2d 5057f55e543b7859cfd26bc28 time.involuntary_context_switches
--------------- ------------------------- ------------------------------------
1209498 Â 1% -6.5% 1131376 Â 1% lkp-a04/netperf/900s-200%-TCP_STREAM
31677 Â 0% +42.5% 45147 Â 1% lkp-a05/iperf/300s-tcp
116081 Â 0% +12.5% 130610 Â 1% lkp-a06/qperf/600s
546 Â24% +2329.3% 13280 Â 8% lkp-sb03/nepim/300s-100%-tcp
863 Â45% +1260.1% 11737 Â12% lkp-sb03/nepim/300s-100%-tcp6
343 Â18% +12507.6% 43294 Â 2% lkp-sb03/nepim/300s-100%-udp6
633 Â23% +1292.7% 8816 Â15% lkp-sb03/nepim/300s-25%-tcp
417 Â 5% +1714.5% 7572 Â11% lkp-sb03/nepim/300s-25%-tcp6
364 Â20% +9569.9% 35198 Â 7% lkp-sb03/nepim/300s-25%-udp
312 Â 0% +12223.0% 38521 Â 1% lkp-sb03/nepim/300s-25%-udp6
308 Â 0% +6197.7% 19418 Â 0% lkp-sb03/nuttcp/300s
418 Â 4% +4948.2% 21101 Â 1% lkp-sb03/thrulay/300s
1.062e+09 Â 0% -2.9% 1.031e+09 Â 0% lkp-snb01/hackbench/50%-threads-pipe
18870 Â 0% +265.8% 69025 Â 0% lkp-snb01/will-it-scale/open2
20813 Â 0% +95.2% 40618 Â 0% xps/ftrace_onoff/5m

iperf tcp:

iperf.tcp.sender.bps

2.2e+10 ++-----*-*---------*-*--------------------------------------------+
| + + + .*. .* |
2.1e+10 ++ .* + .* *.*.*..*.*. .* *.. .*.*. + .*.. |
*.*. *.*. *. .* * *.*
| *.*. |
2e+10 ++ |
| |
1.9e+10 ++ |
| |
1.8e+10 ++ O |
| O O O |
O O O O O O O |
1.7e+10 ++O O O O |
| O O O |
1.6e+10 ++----------------------------------------------------------------+

[*] bisect-good sample
[O] bisect-bad sample


iperf.tcp.receiver.bps

2.2e+10 ++-----*-*---------*-*--------------------------------------------+
| + + + .*. .* |
2.1e+10 ++ .* + .* *.*.*..*.*. .* *.. .*.*. + .*.. |
*.*. *.*. *. .* * *.*
| *.*. |
2e+10 ++ |
| |
1.9e+10 ++ |
| |
1.8e+10 ++ O |
| O O O |
O O O O O O O |
1.7e+10 ++O O O O |
| O O O |
1.6e+10 ++----------------------------------------------------------------+


time.involuntary_context_switches

10000 ++----------------------------------O-------------------------------+
9000 ++ |
| O O O O O O O O |
8000 O+O O O O O |
7000 ++ O O O |
| |
6000 ++ |
5000 ++ |
4000 ++ |
| |
3000 ++ |
2000 ++*..* .*.* *
* + .*.*. + .*.*..*.*.. .*.*..*.*.. .*.*.*.. .*.. +|
1000 ++ *. *. * *.*. * *.*..* |
0 ++------------------------------------------------------------------+


qperf:
time.involuntary_context_switches

25000 ++------------------------------------------------------------------+
| |
| O O OO OO O OO |
20000 O+O OO O O O O OO O O O O O OO O O OO OO |
| O O O O O
| |
15000 ++ OO O O O |
| O O |
10000 ++ |
| |
| |
5000 ++ |
|.**.* .**.*.* .* .*. *.* .* .*. *.**.** |
* * * * * * * **.**.*.* |
0 ++------------------------------------------------------------------+


qperf.udp.send_bw

2.2e+09 *+**--*----*--**----**-*-----*-**----*---*------------------------+
| * ** * ** **.* ** *.* *.** |
2e+09 ++ |
1.8e+09 ++ |
| O O O O
1.6e+09 ++ O O O OO OO O OO OO O O O O |
| O O O O O O |
1.4e+09 ++ |
| |
1.2e+09 ++ |
1e+09 ++ |
| O |
8e+08 O+ O O O O O |
| O OO O OO O O O O O |
6e+08 ++----------------------------------------------------------------+


qperf.udp.recv_bw

2.2e+09 *+**--*----*--**----**-*-----*-**----*---*------------------------+
| * ** * ** **.* ** *.* *.** |
2e+09 ++ |
1.8e+09 ++ |
| O O O
1.6e+09 ++ O O OO OO OO O OO OO O O O O |
| O O O O O O |
1.4e+09 ++ |
| |
1.2e+09 ++ |
1e+09 ++ |
| O |
8e+08 O+ O O O O O |
| O OO O OO O O O O O |
6e+08 ++----------------------------------------------------------------+


will-it-scale unlink1:

time.voluntary_context_switches

40000 ++--O---------O-----------------------------------------------------+
| O |
35000 O+ O O O O O O |
30000 ++ O O O O O O |
| O O O O
25000 ++ |
| |
20000 ++ |
| |
15000 ++ |
10000 ++ ..*..*.. |
| .*...*..*... .*...*. . ..*..*...|
5000 *+..*..*...*. *...*..*...*. *..*. *
| |
0 ++------------------------------------------------------------------+


time.involuntary_context_switches

60000 ++------------------------------------------------------------------+
| O O O O |
50000 O+ O O O O O O O O O O O O O O O
| |
| |
40000 ++ |
| |
30000 ++ |
| |
20000 ++ |
| |
| *.. |
10000 ++ .. *.. |
| . . |
0 *+--*--*---*--*---*--*---*---*--*---*--*---*----------*--*---*--*---*

Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/