Re: [rcu] 5057f55e543: -23.5% qperf.udp.recv_bw

[Paul E. McKenney]

On Tue, Jun 03, 2014 at 06:08:41PM +0800, Jet Chen wrote:
> Hi Paul,
> 
> FYI, we noticed the below changes on
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/fixes
> commit 5057f55e543b7859cfd26bc281291795eac93f8a ("rcu: Bind RCU grace-period kthreads if NO_HZ_FULL")

My guess would be that some of these workloads generated enough callbacks
that binding all the rcuo callback-offloading kthreads to CPU 0 resulted
in a bottleneck.  If that was the case, CPU 0 would often hit 100%
CPU utilization, and there would be more wait time on other CPUs because
callback execution was delayed.

Does that match what you are seeing?

							Thanx, Paul

> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  2.127e+09 ~ 0%     -23.5%  1.628e+09 ~ 4%  bens/qperf/600s
>  2.127e+09 ~ 0%     -23.5%  1.628e+09 ~ 4%  TOTAL qperf.udp.recv_bw
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  2.128e+09 ~ 0%     -23.3%  1.633e+09 ~ 4%  bens/qperf/600s
>  2.128e+09 ~ 0%     -23.3%  1.633e+09 ~ 4%  TOTAL qperf.udp.send_bw
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  2.101e+10 ~ 2%     -18.7%  1.707e+10 ~ 2%  bens/iperf/300s-tcp
>  2.101e+10 ~ 2%     -18.7%  1.707e+10 ~ 2%  TOTAL iperf.tcp.sender.bps
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  2.101e+10 ~ 2%     -18.7%  1.707e+10 ~ 2%  bens/iperf/300s-tcp
>  2.101e+10 ~ 2%     -18.7%  1.707e+10 ~ 2%  TOTAL iperf.tcp.receiver.bps
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  1.331e+09 ~ 2%      -5.8%  1.255e+09 ~ 2%  bens/qperf/600s
>    2.4e+09 ~ 6%     -30.4%  1.671e+09 ~12%  brickland3/qperf/600s
>  2.384e+09 ~ 7%     -12.1%  2.096e+09 ~ 3%  lkp-sb03/qperf/600s
>  6.115e+09 ~ 5%     -17.9%  5.022e+09 ~ 6%  TOTAL qperf.sctp.bw
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>   2.83e+09 ~ 1%     -12.5%  2.476e+09 ~ 3%  bens/qperf/600s
>   2.83e+09 ~ 1%     -12.5%  2.476e+09 ~ 3%  TOTAL qperf.tcp.bw
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  2.272e+08 ~ 1%     -13.3%   1.97e+08 ~ 2%  bens/qperf/600s
>  2.272e+08 ~ 1%     -13.3%   1.97e+08 ~ 2%  TOTAL proc-vmstat.pgalloc_dma32
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>      53062 ~ 2%     -35.1%      34464 ~ 3%  bens/qperf/600s
>     109531 ~13%     +46.9%     160928 ~ 5%  brickland3/qperf/600s
>      67902 ~ 1%     +13.8%      77302 ~ 3%  lkp-sb03/qperf/600s
>     230496 ~ 7%     +18.3%     272694 ~ 4%  TOTAL softirqs.RCU
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>      80344 ~ 1%     -26.2%      59325 ~ 2%  bens/qperf/600s
>      80344 ~ 1%     -26.2%      59325 ~ 2%  TOTAL softirqs.SCHED
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>       1036 ~ 4%     -17.6%        853 ~ 4%  brickland3/qperf/600s
>       1036 ~ 4%     -17.6%        853 ~ 4%  TOTAL proc-vmstat.nr_page_table_pages
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>      48.12 ~ 0%     -11.7%      42.46 ~ 6%  brickland3/qperf/600s
>      48.12 ~ 0%     -11.7%      42.46 ~ 6%  TOTAL turbostat.%pc2
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>   74689352 ~ 1%     -13.3%   64771743 ~ 2%  bens/qperf/600s
>   74689352 ~ 1%     -13.3%   64771743 ~ 2%  TOTAL proc-vmstat.pgalloc_normal
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>  3.019e+08 ~ 1%     -13.3%  2.618e+08 ~ 2%  bens/qperf/600s
>  3.019e+08 ~ 1%     -13.3%  2.618e+08 ~ 2%  TOTAL proc-vmstat.pgfree
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>   23538414 ~ 0%     -12.9%   20506157 ~ 2%  bens/qperf/600s
>   23538414 ~ 0%     -12.9%   20506157 ~ 2%  TOTAL proc-vmstat.numa_local
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>   23538414 ~ 0%     -12.9%   20506157 ~ 2%  bens/qperf/600s
>   23538414 ~ 0%     -12.9%   20506157 ~ 2%  TOTAL proc-vmstat.numa_hit
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>      12789 ~ 1%     -10.9%      11391 ~ 2%  bens/qperf/600s
>      12789 ~ 1%     -10.9%      11391 ~ 2%  TOTAL softirqs.HRTIMER
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>     481253 ~ 0%      -8.9%     438624 ~ 0%  bens/qperf/600s
>     481253 ~ 0%      -8.9%     438624 ~ 0%  TOTAL softirqs.TIMER
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>       1297 ~33%    +565.9%       8640 ~ 7%  bens/iperf/300s-tcp
>       2788 ~ 3%    +588.8%      19204 ~ 4%  bens/qperf/600s
>       1191 ~ 5%   +1200.9%      15493 ~ 4%  brickland3/qperf/600s
>       1135 ~26%   +1195.9%      14709 ~ 4%  lkp-sb03/qperf/600s
>       6411 ~13%    +805.3%      58047 ~ 4%  TOTAL time.involuntary_context_switches
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>      72398 ~ 1%      -5.4%      68503 ~ 0%  bens/qperf/600s
>       8789 ~ 4%     +22.3%      10749 ~15%  lkp-sb03/qperf/600s
>      81187 ~ 1%      -2.4%      79253 ~ 2%  TOTAL vmstat.system.in
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>     141174 ~ 1%      -5.4%     133551 ~ 0%  bens/qperf/600s
>     143982 ~ 1%      -4.4%     137600 ~ 0%  brickland3/qperf/600s
>     285156 ~ 1%      -4.9%     271152 ~ 0%  TOTAL vmstat.system.cs
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>   42351859 ~ 0%      -5.3%   40114932 ~ 0%  bens/qperf/600s
>   43015383 ~ 1%      -4.4%   41143092 ~ 0%  brickland3/qperf/600s
>   85367242 ~ 1%      -4.8%   81258025 ~ 0%  TOTAL time.voluntary_context_switches
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>        146 ~ 0%      -2.2%        143 ~ 0%  bens/qperf/600s
>        147 ~ 1%      -4.8%        140 ~ 1%  brickland3/qperf/600s
>        293 ~ 0%      -3.5%        283 ~ 0%  TOTAL time.percent_of_cpu_this_job_got
> 
> 71a9b26963f8c2d  5057f55e543b7859cfd26bc28
> ---------------  -------------------------
>        872 ~ 0%      -2.3%        853 ~ 0%  bens/qperf/600s
>        874 ~ 1%      -4.6%        834 ~ 1%  brickland3/qperf/600s
>       1747 ~ 0%      -3.4%       1687 ~ 0%  TOTAL time.system_time
> 
> 
> Legend:
> 	~XX%    - stddev percent
> 	[+-]XX% - change percent
> 
> 
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> Thanks,
> Jet
> 
> 

> echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
> echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
> echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
> echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
> ./runtest.py unlink1 32 1 2 3 4

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/