Re: [rcu] 5057f55e543: -23.5% qperf.udp.recv_bw

[Paul E. McKenney]

On Wed, Jun 04, 2014 at 08:33:38PM +0800, Fengguang Wu wrote:
> Hi Paul,
> 
> On Tue, Jun 03, 2014 at 07:17:20AM -0700, Paul E. McKenney wrote:
> > On Tue, Jun 03, 2014 at 06:08:41PM +0800, Jet Chen wrote:
> > > Hi Paul,
> > > 
> > > FYI, we noticed the below changes on
> > > 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/fixes
> > > commit 5057f55e543b7859cfd26bc281291795eac93f8a ("rcu: Bind RCU grace-period kthreads if NO_HZ_FULL")
> > 
> > My guess would be that some of these workloads generated enough callbacks
> > that binding all the rcuo callback-offloading kthreads to CPU 0 resulted
> > in a bottleneck.  If that was the case, CPU 0 would often hit 100%
> > CPU utilization, and there would be more wait time on other CPUs because
> > callback execution was delayed.
> > 
> > Does that match what you are seeing?
> 
> This is the /proc/stat contents on completion of test
> brickland3/qperf/600s. It shows that CPU 0 is not busy.

Hmmm...  Might be sometimes busy, thus blocking grace periods?
I posted a speculative fix at a558e309f99cf8b809691417324c6770e10bb614,
please let me know how it goes!

							Thanx, Paul

> cpu  1707 0 57716 8569826 33 0 14572 0 0 0
> cpu0 2 0 575 69345 29 0 8 0 0 0
> cpu1 0 0 86 72203 3 0 1 0 0 0
> cpu2 116 0 386 71437 0 0 0 0 0 0
> 
> The columns of numbers are:
> 
> - user: normal processes executing in user mode
> - nice: niced processes executing in user mode
> - system: processes executing in kernel mode
> - idle: twiddling thumbs
> - iowait: waiting for I/O to complete
> - irq: servicing interrupts
> - softirq: servicing softirqs
> - steal: involuntary wait
> - guest: running a normal guest
> - guest_nice: running a niced guest
> 
> cpu3 32 0 59 72213 0 0 0 0 0 0
> cpu4 8 0 35 72272 0 0 0 0 0 0
> cpu5 1 0 3 72318 0 0 0 0 0 0
> cpu6 1 0 3 72320 0 0 0 0 0 0
> cpu7 0 0 2 72327 0 0 0 0 0 0
> cpu8 0 0 2 72323 0 0 0 0 0 0
> cpu9 0 0 1 72323 0 0 0 0 0 0
> cpu10 0 0 2 72314 0 0 0 0 0 0
> cpu11 0 0 2 72317 0 0 0 0 0 0
> cpu12 0 0 3 72311 0 0 0 0 0 0
> cpu13 0 0 2 72313 0 0 0 0 0 0
> cpu14 0 0 20 72296 0 0 0 0 0 0
> cpu15 0 0 90 72192 0 0 1 0 0 0
> cpu16 0 0 13 72288 0 0 1 0 0 0
> cpu17 115 0 137 71848 0 0 0 0 0 0
> cpu18 28 0 32 72106 0 0 0 0 0 0
> cpu19 7 0 5 72256 0 0 0 0 0 0
> cpu20 2 0 72 71938 0 0 0 0 0 0
> cpu21 0 0 3 72273 0 0 0 0 0 0
> cpu22 0 0 2 72284 0 0 0 0 0 0
> cpu23 0 0 1 72284 0 0 0 0 0 0
> cpu24 0 0 2 72279 0 0 0 0 0 0
> cpu25 0 0 1 72281 0 0 0 0 0 0
> cpu26 0 0 2 72274 0 0 0 0 0 0
> cpu27 0 0 5 72274 0 0 0 0 0 0
> cpu28 0 0 22 72255 0 0 0 0 0 0
> cpu29 0 0 1 72273 0 0 0 0 0 0
> cpu30 24 0 748 70165 0 0 407 0 0 0
> cpu31 28 0 906 70251 0 0 465 0 0 0
> cpu32 62 0 2276 68376 0 0 604 0 0 0
> cpu33 27 0 1100 70059 0 0 453 0 0 0
> cpu34 23 0 1359 69933 0 0 418 0 0 0
> cpu35 32 0 1663 69109 0 0 775 0 0 0
> cpu36 103 0 8569 60806 0 0 2306 0 0 0
> cpu37 88 0 7052 63141 0 0 518 0 0 0
> cpu38 51 0 2881 67927 0 0 498 0 0 0
> cpu39 36 0 1705 69166 0 0 430 0 0 0
> cpu40 99 0 8657 60720 0 0 1767 0 0 0
> cpu41 103 0 7088 61185 0 0 1749 0 0 0
> cpu42 42 0 1567 69244 0 0 525 0 0 0
> cpu43 43 0 1661 68865 0 0 457 0 0 0
> cpu44 39 0 2610 68046 0 0 660 0 0 0
> cpu45 3 0 248 71852 0 0 30 0 0 0
> cpu46 4 0 16 72140 0 0 6 0 0 0
> cpu47 115 0 297 71435 0 0 159 0 0 0
> cpu48 30 0 81 71940 0 0 61 0 0 0
> cpu49 14 0 163 71815 0 0 118 0 0 0
> cpu50 6 0 144 71832 0 0 129 0 0 0
> cpu51 12 0 508 71217 0 0 276 0 0 0
> cpu52 18 0 498 71082 0 0 378 0 0 0
> cpu53 11 0 440 71236 0 0 282 0 0 0
> cpu54 4 0 205 71710 0 0 144 0 0 0
> cpu55 4 0 136 71898 0 0 78 0 0 0
> cpu56 13 0 400 71305 0 0 288 0 0 0
> cpu57 14 0 377 71346 0 0 316 0 0 0
> cpu58 2 0 109 71954 0 0 73 0 0 0
> cpu59 1 0 8 72174 0 0 3 0 0 0
> cpu60 0 0 0 72191 0 0 0 0 0 0
> cpu61 86 0 313 71749 0 0 0 0 0 0
> cpu62 3 0 9 72173 0 0 0 0 0 0
> cpu63 2 0 8 72175 0 0 0 0 0 0
> cpu64 0 0 2 72178 0 0 0 0 0 0
> cpu65 0 0 2 72177 0 0 0 0 0 0
> cpu66 0 0 2 72175 0 0 0 0 0 0
> cpu67 0 0 2 72173 0 0 0 0 0 0
> cpu68 0 0 4 72169 0 0 0 0 0 0
> cpu69 0 0 2 72168 0 0 0 0 0 0
> cpu70 0 0 1 72166 0 0 0 0 0 0
> cpu71 0 0 1 72164 0 0 0 0 0 0
> cpu72 0 0 0 72163 0 0 0 0 0 0
> cpu73 0 0 0 72161 0 0 0 0 0 0
> cpu74 0 0 1 72158 0 0 0 0 0 0
> cpu75 0 0 0 72156 0 0 0 0 0 0
> cpu76 73 0 330 71709 0 0 0 0 0 0
> cpu77 4 0 10 72137 0 0 0 0 0 0
> cpu78 1 0 4 72145 0 0 0 0 0 0
> cpu79 0 0 50 72077 0 0 0 0 0 0
> cpu80 0 0 5 72138 0 0 0 0 0 0
> cpu81 0 0 1 72141 0 0 0 0 0 0
> cpu82 0 0 66 72074 0 0 0 0 0 0
> cpu83 0 0 1 72137 0 0 0 0 0 0
> cpu84 0 0 0 72136 0 0 0 0 0 0
> cpu85 0 0 1 72133 0 0 0 0 0 0
> cpu86 0 0 0 72131 0 0 0 0 0 0
> cpu87 0 0 0 72129 0 0 0 0 0 0
> cpu88 0 0 1 72127 0 0 0 0 0 0
> cpu89 0 0 0 72124 0 0 0 0 0 0
> cpu90 0 0 0 72122 0 0 0 0 0 0
> cpu91 63 0 1377 70501 0 0 162 0 0 0
> cpu92 2 0 14 72100 0 0 0 0 0 0
> cpu93 1 0 5 72109 0 0 0 0 0 0
> cpu94 0 0 4 72108 0 0 0 0 0 0
> cpu95 1 0 4 72106 0 0 0 0 0 0
> cpu96 0 0 1 72106 0 0 0 0 0 0
> cpu97 0 0 0 72105 0 0 0 0 0 0
> cpu98 0 0 1 72102 0 0 0 0 0 0
> cpu99 0 0 2 72099 0 0 0 0 0 0
> cpu100 0 0 1 72099 0 0 0 0 0 0
> cpu101 0 0 0 72096 0 0 0 0 0 0
> cpu102 0 0 0 72094 0 0 0 0 0 0
> cpu103 0 0 1 72092 0 0 0 0 0 0
> cpu104 0 0 0 72089 0 0 0 0 0 0
> cpu105 0 0 1 72086 0 0 0 0 0 0
> cpu106 66 0 363 71601 0 0 9 0 0 0
> cpu107 2 0 4 72075 0 0 0 0 0 0
> cpu108 0 0 2 72077 0 0 0 0 0 0
> cpu109 0 0 8 72069 0 0 0 0 0 0
> cpu110 0 0 1 72073 0 0 0 0 0 0
> cpu111 0 0 9 72063 0 0 0 0 0 0
> cpu112 0 0 1 72070 0 0 0 0 0 0
> cpu113 0 0 0 72068 0 0 0 0 0 0
> cpu114 0 0 1 72064 0 0 0 0 0 0
> cpu115 0 0 1 72063 0 0 0 0 0 0
> cpu116 0 0 1 72060 0 0 0 0 0 0
> cpu117 0 0 0 72059 0 0 0 0 0 0
> cpu118 0 0 0 72056 0 0 0 0 0 0
> cpu119 0 0 1 72053 0 0 0 0 0 0
> intr 5639794 132 0 0 9 856 0 0 0 1 0 0 0 0 0 0 0 31 0 0 0 0 0 0 311 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 42 0 2794 395 411 385 399 378 399 375 396 384 383 396 379 470 386 375 375 386 382 512 375 375 375 375 375 375 375 375 375 375 375 375 378 375 375 375 375 375 375 375 375 375 379 381 388 375 375 375 376 379 375 375 375 375 375 375 375 375 375 375 375 382 375 1 78 64 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> ctxt 84441175
> btime 1401871947
> processes 23608
> procs_running 2
> procs_blocked 0
> softirq 84702758 9 1574595 174 82369760 28 0 305 456968 84738 216181
> 
> Thanks,
> Fengguang
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/