Re: [Questions] perf c2c: What's the current status of perf c2c?

From: Joe Mario
Date: Wed Dec 09 2015 - 15:41:50 EST

On 12/09/2015 12:15 PM, Stephane Eranian wrote:
If I recall the c2c tool is giving you more than the bouncing line. It shows you
the offset inside the line and the participating CPUs.

Correct. It shows much more than the bouncing line.

Appended below is the output for running "perf c2c" on a 4-node system
running a multi-thread version of linpack. I've annotated it show describe
what some of the fields mean.

Note, your screen output has to be set pretty wide to read it. For those
not wanting to read it in their mailer, grab it from:
http://people.redhat.com/jmario/perf_c2c/perf_c2c_annotated_output.txt

Let me know of any questions. My annotations begin with "// ".

Joe
--------------------------------------------------------------------
// Perf c2c output from a linpack run.

// Set screen wide to view this.

// Here's a breakdown of all loads and stores sampled.
// It shows where they hit and missed.

=================================================
Trace Event Information
=================================================
Total records : 3229269
Locked Load/Store Operations : 64420
Load Operations : 1153827
Loads - uncacheable : 11
Loads - IO : 0
Loads - Miss : 11002
Loads - no mapping : 200
Load Fill Buffer Hit : 355942
Load L1D hit : 361303
Load L2D hit : 46792
Load LLC hit : 274265
Load Local HITM : 18647
Load Remote HITM : 55225
Load Remote HIT : 8917
Load Local DRAM : 11895
Load Remote DRAM : 28275
Load MESI State Exclusive : 40170
Load MESI State Shared : 0
Load LLC Misses : 104312
LLC Misses to Local DRAM : 11.4%
LLC Misses to Remote DRAM : 27.1%
LLC Misses to Remote cache (HIT) : 8.5%
LLC Misses to Remote cache (HITM) : 52.9%
Store Operations : 2069610
Store - uncacheable : 0
Store - no mapping : 3538
Store L1D Hit : 1889168
Store L1D Miss : 176904
No Page Map Rejects : 102146
Unable to parse data source : 5832


// Table showing activity on the shared cache lines.

=================================================
Global Shared Cache Line Event Information
=================================================
Total Shared Cache Lines : 14213
Load HITs on shared lines : 426101
Fill Buffer Hits on shared lines : 159230
L1D hits on shared lines : 77377
L2D hits on shared lines : 28897
LLC hits on shared lines : 94709
Locked Access on shared lines : 45704
Store HITs on shared lines : 198050
Store L1D hits on shared lines : 188136
Total Merged records : 271778


// In the next table, for each of the 10 hottest cache lines, breakout all the activity for that line.
// The sorting is done by remote hitm percentage, defined as loads that hit in a remote node's modified cacheline.

================================================================================================================================================================================================================
Shared Data Cache Line Table

Total %All Total ---- Core Load Hit ---- -- LLC Load Hit -- ----- LLC Load Hitm ----- -- Load Dram -- LLC ---- Store Reference ----
Index Phys Adrs Records Ld Miss %hitm Loads FB L1D L2D Lcl Rmt Total Lcl Rmt Lcl Rmt Ld Miss Total L1Hit L1Miss
================================================================================================================================================================================================================
0 0x7f9d8833bf80 12402 1.87% 3.54% 11748 6582 2259 72 24 20 2788 835 1953 2 1 1976 654 648 6
1 0x85fd0f0c5c0 5593 0.35% 0.65% 3813 797 2131 153 171 32 457 97 360 20 52 464 1780 1780 0
2 0x81fd0ddfd00 2174 0.21% 0.40% 1352 318 558 50 75 13 315 94 221 9 14 257 822 822 0
3 0x85fd0f0c8c0 1854 0.21% 0.39% 1436 184 655 153 92 28 279 65 214 13 32 287 418 418 0
4 0x83fffb17280 735 0.16% 0.31% 543 3 287 12 15 2 216 44 172 2 6 182 192 192 0
5 0x7fff81db1e00 3641 0.16% 0.30% 3331 1111 849 624 384 83 228 62 166 15 37 301 310 310 0
6 0x83fff857280 755 0.15% 0.29% 563 3 316 13 24 0 201 43 158 3 3 164 192 192 0
7 0x85fff917280 801 0.15% 0.28% 648 1 409 19 19 1 198 44 154 1 0 156 153 153 0
8 0x85fffc17280 767 0.14% 0.27% 664 4 403 27 15 2 205 54 151 2 6 161 103 103 0
9 0x85fffbf7280 718 0.14% 0.27% 495 5 262 15 14 0 193 42 151 1 5 157 223 223 0


======================================================================================================================================================================================================================================================================

Shared Cache Line Distribution Pareto

---- All ---- -- Shared -- ---- HITM ---- Load Inst Execute Latency Shared Data Participants
Data Misses Data Misses Remote Local -- Store Refs --
---- cycles ---- cpu
Num %dist %cumm %dist %cumm LLCmiss LLChit L1 hit L1 Miss Data Address Pid Tid Inst Address median mean CV cnt Symbol Object Node{cpus %hitms %stores} Node{cpus %hitms %stores} ...
======================================================================================================================================================================================================================================================================
-----------------------------------------------------------------------------------------------
0 1.9% 1.9% 3.5% 3.5% 1953 835 648 6 0x7f9d8833bf80 146560
-----------------------------------------------------------------------------------------------
0.3% 0.4% 97.2% 0.0% 0x10 146560 *** 0x9969fa 2036 2265 12.4% 80 __kmp_acquire_queuing_lock_wi xlinpack_xeon64 0{20 16.7% 25.2%} 1{21 33.3% 24.0%} 2{20 33.3% 25.7%} 3{19 16.7% 25.1%}
88.9% 89.1% 0.0% 0.0% 0x10 146560 *** 0x996dfc 304 349 0.4% 79 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{20 23.7% n/a} 1{21 24.5% n/a} 2{19 26.8% n/a} 3{19 25.0% n/a}
0.1% 0.0% 0.0% 0.0% 0x14 146560 146608 0x9969ac 410 410 0.0% 1 __kmp_acquire_queuing_lock_wi xlinpack_xeon64 2{ 1 100.0% n/a}
10.7% 10.5% 0.0% 0.0% 0x14 146560 *** 0x996def 303 352 1.2% 69 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{18 21.5% n/a} 1{18 27.8% n/a} 2{18 31.6% n/a} 3{15 19.1% n/a}
0.0% 0.0% 2.8% 100.0% 0x14 146560 *** 0x996e54 n/a n/a n/a 19 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{ 4 n/a 25.0%} 1{ 6 n/a 29.2%} 2{ 6 n/a 33.3%} 3{ 3 n/a 12.5%}

// Here's where the data gets interesting. Look at cacheline 0 above. The cacheline at data address 0x7f9d8833bf80 had the most contention.
//
// There were 1953 loads to that cacheline that hit in "remote-node modified cachelines". That's why this is the hottest cacheline.
// There were 835 loads that hit in a local modified cacheline. Also noted are the number of stores that both hit and missed the L1 cache.
// All accesses to that cacheline occurred at offsets 0x10 and 0x14.
// All accesses were done by one pid (146560), which was the Pid for linpack.
// I chose to display "***" when multiple thread ids (Tids) were involved in the same entry. Individual Tids can be displayed, but it makes for a long output.
// The instruction address of the load/store is displayed, along with the function and object name.
// Where loads are involved, the median and mean load latency cycles are displayed. Dumping the c2c raw records shows you individual worst offenders. It's not uncommon to see loads taking tens of thousands of cycles to complete when heavy contention is involved.
// The "cpu cnt" column shows how many individual cpus had samples contributing to a row of data. In the first row above, there were samples from 80 cpus for that row.
// The "Shared Data Participants" columns show the nodes the samples occured on, the number of cpus in each node that samples came from for that row, and for each node, the percentage of hitms and stores that came from that node.
//
// The above data shows how that hot cacheline is being concurrently read and written from cpus across all four nodes on the system. It's then easy to disassemble the binary (with line info) to identify the source code line numbers causing the false sharing.

-----------------------------------------------------------------------------------------------
1 0.3% 2.2% 0.7% 4.2% 360 97 1780 0 0xffff885fd0f0a5c0 ***
-----------------------------------------------------------------------------------------------
21.1% 29.9% 0.0% 0.0% 0x18 146560 *** 0xffffffff81196f40 1193 2404 13.9% 44 handle_mm_fault [kernel.kallsyms] 0{ 4 9.2% n/a} 1{12 27.6% n/a} 2{17 40.8% n/a} 3{11 22.4% n/a}
25.0% 21.6% 0.0% 0.0% 0x18 146560 *** 0xffffffff811a16d9 377 754 16.8% 43 mm_find_pmd [kernel.kallsyms] 0{11 23.3% n/a} 1{12 16.7% n/a} 2{11 41.1% n/a} 3{ 9 18.9% n/a}
17.2% 12.4% 0.0% 0.0% 0x18 0 0 0xffffffff8163accb 1994 2467 11.0% 58 __schedule [kernel.kallsyms] 0{16 24.2% n/a} 1{13 21.0% n/a} 2{13 21.0% n/a} 3{16 33.9% n/a}
3.9% 2.1% 24.1% 0.0% 0x24 *** *** 0xffffffff810b41ce 2066 3398 30.0% 136 finish_task_switch [kernel.kallsyms] 0{32 35.7% 23.5%} 1{36 14.3% 26.8%} 2{36 42.9% 26.1%} 3{32 7.1% 23.5%}
13.9% 11.3% 29.6% 0.0% 0x24 146560 *** 0xffffffff8163aa37 7230 7026 7.8% 139 __schedule [kernel.kallsyms] 0{34 26.0% 18.8%} 1{36 24.0% 28.5%} 2{36 34.0% 27.9%} 3{33 16.0% 24.7%}
0.0% 0.0% 0.1% 0.0% 0x24 146560 146665 0xffffffff8163aa3c n/a n/a n/a 1 __schedule [kernel.kallsyms] 2{ 1 n/a 100.0%}
5.3% 3.1% 0.0% 0.0% 0x38 146560 *** 0xffffffff810aa959 406 971 22.2% 20 down_read_trylock [kernel.kallsyms] 0{ 3 15.8% n/a} 1{ 9 52.6% n/a} 2{ 4 15.8% n/a} 3{ 4 15.8% n/a}
12.5% 15.5% 18.0% 0.0% 0x38 146560 *** 0xffffffff810aa965 2342 3845 13.9% 82 down_read_trylock [kernel.kallsyms] 0{21 15.6% 16.6%} 1{22 24.4% 31.6%} 2{19 35.6% 27.2%} 3{20 24.4% 24.7%}
1.1% 4.1% 28.3% 0.0% 0x38 146560 *** 0xffffffff810aa9c3 1376 1462 20.1% 87 up_read [kernel.kallsyms] 0{21 50.0% 16.7%} 1{23 50.0% 31.3%} 2{23 0.0% 28.6%} 3{20 0.0% 23.4%}

-----------------------------------------------------------------------------------------------
2 0.2% 2.4% 0.4% 4.6% 221 94 822 0 0xffff881fd0dddd00 146560
-----------------------------------------------------------------------------------------------
2.7% 2.1% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a31ff 328 550 25.9% 8 page_lock_anon_vma_read [kernel.kallsyms] 0{ 2 16.7% n/a} 1{ 1 16.7% n/a} 2{ 3 33.3% n/a} 3{ 2 33.3% n/a}
9.5% 9.6% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a36f5 429 814 19.2% 24 try_to_unmap_anon [kernel.kallsyms] 0{ 5 19.0% n/a} 1{ 9 42.9% n/a} 2{ 5 23.8% n/a} 3{ 5 14.3% n/a}
30.8% 27.7% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a39f0 373 541 9.4% 37 rmap_walk [kernel.kallsyms] 0{12 32.4% n/a} 1{ 9 20.6% n/a} 2{ 8 25.0% n/a} 3{ 8 22.1% n/a}
24.0% 20.2% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a3a6a 424 692 11.0% 35 rmap_walk [kernel.kallsyms] 0{ 9 26.4% n/a} 1{10 24.5% n/a} 2{ 9 32.1% n/a} 3{ 7 17.0% n/a}
0.9% 0.0% 0.0% 0.0% 0x08 146560 *** 0xffffffff810aa959 295 295 4.4% 2 down_read_trylock [kernel.kallsyms] 0{ 2 100.0% n/a}
2.3% 2.1% 38.3% 0.0% 0x08 146560 *** 0xffffffff810aa965 537 821 25.9% 48 down_read_trylock [kernel.kallsyms] 0{15 40.0% 25.4%} 1{12 0.0% 28.6%} 2{11 0.0% 26.7%} 3{10 60.0% 19.4%}
3.2% 4.3% 15.0% 0.0% 0x08 146560 *** 0xffffffff810aa9c3 746 834 17.1% 44 up_read [kernel.kallsyms] 0{13 14.3% 23.6%} 1{12 14.3% 23.6%} 2{10 28.6% 26.0%} 3{ 9 42.9% 26.8%}
4.1% 3.2% 22.9% 0.0% 0x08 146560 *** 0xffffffff8163a0a5 950 934 15.7% 46 down_read [kernel.kallsyms] 0{14 44.4% 31.9%} 1{13 22.2% 22.3%} 2{11 22.2% 25.5%} 3{ 8 11.1% 20.2%}
12.7% 17.0% 0.0% 0.0% 0x28 146560 *** 0xffffffff811a3140 337 827 26.1% 28 page_get_anon_vma [kernel.kallsyms] 0{ 7 21.4% n/a} 1{ 9 42.9% n/a} 2{ 6 10.7% n/a} 3{ 6 25.0% n/a}
2.3% 2.1% 17.0% 0.0% 0x28 146560 *** 0xffffffff811a3151 1288 1870 29.9% 43 page_get_anon_vma [kernel.kallsyms] 0{13 40.0% 33.6%} 1{11 20.0% 19.3%} 2{10 0.0% 27.1%} 3{ 9 40.0% 20.0%}
0.0% 0.0% 0.1% 0.0% 0x28 146560 146602 0xffffffff811a31aa n/a n/a n/a 1 page_get_anon_vma [kernel.kallsyms] 1{ 1 n/a 100.0%}
2.7% 3.2% 6.7% 0.0% 0x28 146560 *** 0xffffffff811c77b4 843 1950 43.3% 35 migrate_pages [kernel.kallsyms] 0{10 16.7% 36.4%} 1{ 6 16.7% 18.2%} 2{10 16.7% 21.8%} 3{ 9 50.0% 23.6%}
5.0% 8.5% 0.0% 0.0% 0x30 146560 *** 0xffffffff81190b15 336 561 19.6% 15 anon_vma_interval_tree_iter_f [kernel.kallsyms] 0{ 5 36.4% n/a} 1{ 4 9.1% n/a} 2{ 3 36.4% n/a} 3{ 3 18.2% n/a}

-----------------------------------------------------------------------------------------------
3 0.2% 2.6% 0.4% 5.0% 214 65 418 0 0xffff885fd0f0a8c0 ***
-----------------------------------------------------------------------------------------------
7.9% 13.8% 0.0% 0.0% 0x08 0 0 0xffffffff81065c26 430 724 17.2% 20 leave_mm [kernel.kallsyms] 0{10 58.8% n/a} 1{10 41.2% n/a}
0.5% 0.0% 12.7% 0.0% 0x08 0 0 0xffffffff81065c38 1343 1343 0.0% 25 leave_mm [kernel.kallsyms] 0{15 0.0% 54.7%} 1{10 100.0% 45.3%}
1.9% 1.5% 5.5% 0.0% 0x08 0 0 0xffffffff810b0c74 1902 3074 32.9% 18 cpumask_set_cpu [kernel.kallsyms] 0{ 9 25.0% 60.9%} 1{ 9 75.0% 39.1%}
11.2% 9.2% 0.0% 0.0% 0x08 *** *** 0xffffffff8163acb5 401 554 15.9% 20 __schedule [kernel.kallsyms] 0{13 58.3% n/a} 1{ 7 41.7% n/a}
6.5% 6.2% 0.0% 0.0% 0x0c 0 0 0xffffffff81065c26 310 540 22.2% 14 leave_mm [kernel.kallsyms] 1{ 2 14.3% n/a} 2{10 78.6% n/a} 3{ 2 7.1% n/a}
0.5% 0.0% 11.2% 0.0% 0x0c 0 0 0xffffffff81065c38 3638 3638 0.0% 25 leave_mm [kernel.kallsyms] 1{ 1 0.0% 4.3%} 2{16 100.0% 68.1%} 3{ 8 0.0% 27.7%}
2.8% 4.6% 3.8% 0.0% 0x0c 0 0 0xffffffff810b0c74 1774 2290 25.7% 20 cpumask_set_cpu [kernel.kallsyms] 1{ 3 0.0% 18.8%} 2{12 100.0% 50.0%} 3{ 5 0.0% 31.2%}
15.4% 9.2% 0.0% 0.0% 0x0c *** *** 0xffffffff8163acb5 415 768 23.1% 21 __schedule [kernel.kallsyms] 1{ 2 3.0% n/a} 2{12 63.6% n/a} 3{ 7 33.3% n/a}
10.7% 15.4% 0.0% 0.0% 0x10 0 0 0xffffffff81065c26 428 619 14.2% 18 leave_mm [kernel.kallsyms] 0{13 65.2% n/a} 1{ 3 13.0% n/a} 3{ 2 21.7% n/a}
0.0% 0.0% 14.1% 0.0% 0x10 0 0 0xffffffff81065c38 n/a n/a n/a 27 leave_mm [kernel.kallsyms] 0{15 n/a 55.9%} 1{ 5 n/a 18.6%} 3{ 7 n/a 25.4%}
0.0% 4.6% 6.9% 0.0% 0x10 0 0 0xffffffff810b0c74 n/a n/a n/a 18 cpumask_set_cpu [kernel.kallsyms] 0{10 n/a 41.4%} 1{ 5 n/a 41.4%} 3{ 3 n/a 17.2%}
9.3% 7.7% 0.0% 0.0% 0x10 *** *** 0xffffffff8163acb5 404 640 25.5% 18 __schedule [kernel.kallsyms] 0{ 6 35.0% n/a} 1{ 5 25.0% n/a} 3{ 7 40.0% n/a}
13.1% 9.2% 0.0% 0.0% 0x14 0 0 0xffffffff81065c26 340 458 9.8% 21 leave_mm [kernel.kallsyms] 1{ 7 32.1% n/a} 2{13 64.3% n/a} 3{ 1 3.6% n/a}
0.5% 0.0% 20.3% 0.0% 0x14 0 0 0xffffffff81065c38 2662 2662 0.0% 27 leave_mm [kernel.kallsyms] 1{11 0.0% 32.9%} 2{14 100.0% 57.6%} 3{ 2 0.0% 9.4%}
1.9% 0.0% 8.9% 0.0% 0x14 0 0 0xffffffff810b0c74 2892 3086 28.5% 20 cpumask_set_cpu [kernel.kallsyms] 1{ 6 0.0% 32.4%} 2{13 100.0% 64.9%} 3{ 1 0.0% 2.7%}
0.0% 0.0% 0.2% 0.0% 0x14 0 0 0xffffffff810b0c78 n/a n/a n/a 1 cpumask_set_cpu [kernel.kallsyms] 2{ 1 n/a 100.0%}
8.9% 4.6% 0.0% 0.0% 0x14 0 0 0xffffffff8163acb5 437 1041 20.5% 16 __schedule [kernel.kallsyms] 1{ 8 42.1% n/a} 2{ 7 52.6% n/a} 3{ 1 5.3% n/a}
3.3% 7.7% 0.0% 0.0% 0x18 0 0 0xffffffff81065c26 407 450 15.8% 8 leave_mm [kernel.kallsyms] 3{ 8 100.0% n/a}
0.0% 0.0% 12.4% 0.0% 0x18 0 0 0xffffffff81065c38 n/a n/a n/a 16 leave_mm [kernel.kallsyms] 3{16 n/a 100.0%}
0.9% 1.5% 3.8% 0.0% 0x18 0 0 0xffffffff810b0c74 1007 1230 18.2% 12 cpumask_set_cpu [kernel.kallsyms] 3{12 100.0% 100.0%}
4.7% 4.6% 0.0% 0.0% 0x18 *** *** 0xffffffff8163acb5 342 629 25.1% 10 __schedule [kernel.kallsyms] 3{10 100.0% n/a}

-----------------------------------------------------------------------------------------------
4 0.2% 2.8% 0.3% 5.3% 172 44 192 0 0xffff883fffb15280 146560
-----------------------------------------------------------------------------------------------
0.6% 0.0% 85.4% 0.0% 0x00 146560 *** 0xffffffff810e6e13 298 298 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 1{ 1 100.0% 100.0%}
98.8% 97.7% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 291 363 4.9% 44 llist_add_batch [kernel.kallsyms] 0{13 27.1% n/a} 1{10 13.5% n/a} 2{12 29.4% n/a} 3{ 9 30.0% n/a}
0.6% 2.3% 14.6% 0.0% 0x00 146560 *** 0xffffffff813067e1 892 1255 0.0% 21 llist_add_batch [kernel.kallsyms] 0{ 7 0.0% 35.7%} 1{ 2 0.0% 7.1%} 2{ 6 100.0% 28.6%} 3{ 6 0.0% 28.6%}

-----------------------------------------------------------------------------------------------
5 0.2% 3.0% 0.3% 5.6% 166 62 310 0 0xffffffff81dafe00 ***
-----------------------------------------------------------------------------------------------
0.6% 0.0% 0.0% 0.0% 0x00 146560 146696 0xffffffff810bcc6c 1363 1363 0.0% 1 update_cfs_rq_h_load [kernel.kallsyms] 1{ 1 100.0% n/a}
22.3% 21.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff810bdf56 1421 2209 15.8% 38 select_task_rq_fair [kernel.kallsyms] 0{13 32.4% n/a} 1{ 7 16.2% n/a} 2{12 35.1% n/a} 3{ 6 16.2% n/a}
4.2% 6.5% 0.0% 0.0% 0x00 146560 *** 0xffffffff810bdfaf 1500 5357 64.4% 10 select_task_rq_fair [kernel.kallsyms] 0{ 1 0.0% n/a} 1{ 4 71.4% n/a} 2{ 2 0.0% n/a} 3{ 3 28.6% n/a}
6.0% 11.3% 0.0% 0.0% 0x00 *** *** 0xffffffff810befcf 3172 2506 25.8% 15 update_blocked_averages [kernel.kallsyms] 0{ 2 10.0% n/a} 1{ 4 20.0% n/a} 2{ 4 30.0% n/a} 3{ 5 40.0% n/a}
54.8% 54.8% 0.0% 0.0% 0x00 *** *** 0xffffffff810c1963 1884 2815 13.1% 81 update_cfs_shares [kernel.kallsyms] 0{16 23.1% n/a} 1{21 24.2% n/a} 2{23 27.5% n/a} 3{21 25.3% n/a}
1.2% 0.0% 0.0% 0.0% 0x08 146560 *** 0xffffffff810b5fb7 1365 1365 51.4% 2 set_task_cpu [kernel.kallsyms] 0{ 1 50.0% n/a} 1{ 1 50.0% n/a}
3.0% 3.2% 0.0% 0.0% 0x08 146560 *** 0xffffffff810bf901 815 1186 45.9% 6 can_migrate_task [kernel.kallsyms] 0{ 2 20.0% n/a} 1{ 2 40.0% n/a} 2{ 2 40.0% n/a}
3.6% 0.0% 44.8% 0.0% 0x20 *** *** 0xffffffff810bcf6f 5364 6967 41.5% 94 update_cfs_rq_blocked_load [kernel.kallsyms] 0{22 33.3% 17.3%} 1{27 0.0% 36.0%} 2{23 16.7% 23.0%} 3{22 50.0% 23.7%}
0.0% 0.0% 1.0% 0.0% 0x28 *** *** 0xffffffff810bf3a9 n/a n/a n/a 3 update_blocked_averages [kernel.kallsyms] 0{ 1 n/a 33.3%} 3{ 2 n/a 66.7%}
0.0% 0.0% 0.3% 0.0% 0x28 0 0 0xffffffff810c102a n/a n/a n/a 1 idle_enter_fair [kernel.kallsyms] 1{ 1 n/a 100.0%}
0.6% 3.2% 29.7% 0.0% 0x28 146560 *** 0xffffffff810c2455 1647 1647 0.0% 68 dequeue_task_fair [kernel.kallsyms] 0{23 100.0% 35.9%} 1{18 0.0% 26.1%} 2{19 0.0% 25.0%} 3{ 8 0.0% 13.0%}
0.0% 0.0% 0.6% 0.0% 0x28 146560 *** 0xffffffff810c2ba0 n/a n/a n/a 2 task_tick_fair [kernel.kallsyms] 2{ 1 n/a 50.0%} 3{ 1 n/a 50.0%}
3.0% 0.0% 23.5% 0.0% 0x28 *** *** 0xffffffff810c4203 2085 2958 31.8% 61 enqueue_task_fair [kernel.kallsyms] 0{14 20.0% 30.1%} 1{14 0.0% 21.9%} 2{13 20.0% 21.9%} 3{20 60.0% 26.0%}
0.6% 0.0% 0.0% 0.0% 0x30 146560 146593 0xffffffff810b5fe7 3070 3070 0.0% 1 set_task_cpu [kernel.kallsyms] 1{ 1 100.0% n/a}

-----------------------------------------------------------------------------------------------
6 0.2% 3.1% 0.3% 5.9% 158 43 192 0 0xffff883fff855280 146560
-----------------------------------------------------------------------------------------------
0.0% 0.0% 87.5% 0.0% 0x00 146560 146569 0xffffffff810e6e13 n/a n/a n/a 1 flush_smp_call_function_queue [kernel.kallsyms] 1{ 1 n/a 100.0%}
98.7% 100.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 317 406 5.9% 44 llist_add_batch [kernel.kallsyms] 0{13 32.1% n/a} 1{10 11.5% n/a} 2{11 30.1% n/a} 3{10 26.3% n/a}
1.3% 0.0% 12.5% 0.0% 0x00 146560 *** 0xffffffff813067e1 1040 1040 23.1% 20 llist_add_batch [kernel.kallsyms] 0{ 6 50.0% 29.2%} 1{ 5 0.0% 20.8%} 2{ 6 50.0% 25.0%} 3{ 3 0.0% 25.0%}

-----------------------------------------------------------------------------------------------
7 0.1% 3.3% 0.3% 6.2% 154 44 153 0 0xffff885fff915280 146560
-----------------------------------------------------------------------------------------------
0.6% 0.0% 86.9% 0.0% 0x00 146560 146624 0xffffffff810e6e13 1341 1341 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 100.0% 100.0%}
98.7% 100.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 287 377 9.2% 43 llist_add_batch [kernel.kallsyms] 0{12 36.8% n/a} 1{11 24.3% n/a} 2{10 11.8% n/a} 3{10 27.0% n/a}
0.6% 0.0% 13.1% 0.0% 0x00 146560 *** 0xffffffff813067e1 539 539 0.0% 17 llist_add_batch [kernel.kallsyms] 0{ 6 0.0% 30.0%} 1{ 6 0.0% 40.0%} 2{ 1 0.0% 5.0%} 3{ 4 100.0% 25.0%}

-----------------------------------------------------------------------------------------------
8 0.1% 3.4% 0.3% 6.4% 151 54 103 0 0xffff885fffc15280 146560
-----------------------------------------------------------------------------------------------
0.7% 0.0% 69.9% 0.0% 0x00 146560 146607 0xffffffff810e6e13 902 902 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 100.0% 100.0%}
99.3% 98.1% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 278 377 5.3% 45 llist_add_batch [kernel.kallsyms] 0{13 34.7% n/a} 1{11 24.7% n/a} 2{11 10.7% n/a} 3{10 30.0% n/a}
0.0% 1.9% 30.1% 0.0% 0x00 146560 *** 0xffffffff813067e1 n/a n/a n/a 23 llist_add_batch [kernel.kallsyms] 0{ 6 n/a 22.6%} 1{ 7 n/a 32.3%} 2{ 6 n/a 25.8%} 3{ 4 n/a 19.4%}

-----------------------------------------------------------------------------------------------
9 0.1% 3.5% 0.3% 6.7% 151 42 223 0 0xffff885fffbf5280 146560
-----------------------------------------------------------------------------------------------
0.0% 0.0% 87.4% 0.0% 0x00 146560 *** 0xffffffff810e6e13 n/a n/a n/a 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 n/a 100.0%}
99.3% 97.6% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 309 346 2.5% 44 llist_add_batch [kernel.kallsyms] 0{11 28.0% n/a} 1{11 23.3% n/a} 2{11 13.3% n/a} 3{11 35.3% n/a}
0.7% 2.4% 12.6% 0.0% 0x00 146560 *** 0xffffffff813067e1 1028 1696 0.0% 20 llist_add_batch [kernel.kallsyms] 0{ 5 0.0% 25.0%} 1{ 2 0.0% 7.1%} 2{ 6 100.0% 35.7%} 3{ 7 0.0% 32.1%}

=====================================================================================================================================
Object Name, Path & Reference Counts

Index Records Object Name Object Path
=====================================================================================================================================
0 2032703 xlinpack_xeon64 /home/joe/linpack/xlinpack_xeon64
1 1059580 [kernel.kallsyms] /proc/kcore
2 21352 libpthread-2.17.so /usr/lib64/libpthread-2.17.so
3 3882 perf /home/root/git/rhel7/tools/perf/perf
4 26 libc-2.17.so /usr/lib64/libc-2.17.so
5 5 libpython2.7.so.1.0 /usr/lib64/libpython2.7.so.1.0
6 3 ld-2.17.so /usr/lib64/ld-2.17.so
7 1 sendmail.sendmail /usr/sbin/sendmail.sendmail
8 1 irqbalance /usr/sbin/irqbalance




--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/