Re: [patch 00/13] Syslets, "Threadlets", generic AIO support, v3

From: Evgeniy Polyakov
Date: Thu Mar 01 2007 - 09:49:31 EST

On Thu, Mar 01, 2007 at 02:12:50PM +0100, Eric Dumazet (dada1@xxxxxxxxxxxxx) wrote:
> On Thursday 01 March 2007 12:47, Evgeniy Polyakov wrote:
> >
> > Could you provide at least remote way to find it?
> >
>
> Sure :)
>
> > I only found the same problem at
> > http://lkml.org/lkml/2006/10/27/3
> >
> > but without any hits to solve the problem.
> >
> > I will try CVS oprofile, if it works I will provide details of course.
> >
>
> # cat CVS/Root
> CVS/Root::pserver:anonymous@xxxxxxxxxxxxxxxxxxxxxxxxxxxx:/cvsroot/oprofile
>
> # cvs diff >/tmp/oprofile.diff
>
> Hope it helps

One of the hunks failed, since it was in CVS already.
After setting up some mirrors, I've installed all bits needed for
oprofile.
Attached kevent and epoll profiles.

In that tests I got epoll perf about 4400 req/s, kevent was about 5300.

epoll does not contain gettimeofday() call.

--
Evgeniy Polyakov
CPU: AMD64 processors, speed 2210.08 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
195750 67.3097 cpu_idle
14111 4.8521 enter_idle
4979 1.7121 IRQ0x51_interrupt
4765 1.6385 tcp_v4_rcv
3316 1.1402 tcp_ack
3138 1.0790 kmem_cache_free
2619 0.9006 kfree
2323 0.7988 memset_c
1747 0.6007 schedule
1682 0.5784 csum_partial_copy_generic
1646 0.5660 ip_output
1563 0.5374 tcp_rcv_state_process
1506 0.5178 dev_queue_xmit
1412 0.4855 handle_IRQ_event
1266 0.4353 ip_rcv
1026 0.3528 ip_queue_xmit
1004 0.3452 __do_softirq
1001 0.3442 ip_local_deliver
906 0.3115 csum_partial
902 0.3102 ip_route_input
889 0.3057 __d_lookup
880 0.3026 kmem_cache_alloc
847 0.2912 tcp_v4_do_rcv
841 0.2892 netif_receive_skb
830 0.2854 tcp_sendmsg
819 0.2816 system_call
788 0.2710 kfree_skbmem
780 0.2682 tcp_transmit_skb
742 0.2551 preempt_schedule
731 0.2514 __tcp_push_pending_frames
699 0.2404 __link_path_walk
687 0.2362 pfifo_fast_dequeue
672 0.2311 local_bh_enable
657 0.2259 __alloc_skb
650 0.2235 net_rx_action
627 0.2156 pfifo_fast_enqueue
583 0.2005 sock_wfree
571 0.1963 get_unused_fd
547 0.1881 tcp_parse_options
546 0.1877 copy_user_generic_string
533 0.1833 _atomic_dec_and_lock
529 0.1819 number
524 0.1802 ret_from_intr
509 0.1750 skb_clone
507 0.1743 fget
507 0.1743 tcp_rcv_established
498 0.1712 __kfree_skb
492 0.1692 tcp_poll
481 0.1654 rt_hash_code
471 0.1620 dput
454 0.1561 sock_def_readable
422 0.1451 mask_and_ack_8259A
421 0.1448 sysret_check
419 0.1441 __fput
413 0.1420 exit_idle
396 0.1362 ip_append_data
374 0.1286 sock_poll
371 0.1276 tcp_data_queue
359 0.1234 __tcp_select_window
356 0.1224 tcp_recvmsg
348 0.1197 lock_timer_base
340 0.1169 cache_alloc_refill
338 0.1162 thread_return
319 0.1097 sys_close
318 0.1093 do_path_lookup
318 0.1093 ret_from_sys_call
311 0.1069 vsnprintf
307 0.1056 eth_type_trans
303 0.1042 find_next_zero_bit
302 0.1038 __mod_timer
298 0.1025 d_alloc
296 0.1018 rb_erase
293 0.1007 call_softirq
290 0.0997 __dentry_open
276 0.0949 cache_grow
274 0.0942 __ip_route_output_key
273 0.0939 try_to_wake_up
258 0.0887 dentry_iput
258 0.0887 sk_stream_mem_schedule
257 0.0884 do_lookup
244 0.0839 strncpy_from_user
234 0.0805 do_generic_mapping_read
231 0.0794 memcmp
229 0.0787 tcp_current_mss
228 0.0784 tcp_rtt_estimator
214 0.0736 restore_args
205 0.0705 sys_recvfrom
204 0.0701 fput
203 0.0698 tcp_send_fin
200 0.0688 release_sock
193 0.0664 memcpy_c
191 0.0657 common_interrupt
189 0.0650 fget_light
185 0.0636 skb_checksum
182 0.0626 generic_drop_inode
180 0.0619 do_sys_open
174 0.0598 get_page_from_freelist
168 0.0578 link_path_walk
165 0.0567 schedule_timeout
163 0.0560 del_timer
162 0.0557 rb_insert_color
160 0.0550 percpu_counter_mod
159 0.0547 __up_read
155 0.0533 expand_files
154 0.0530 sys_fcntl
150 0.0516 tcp_v4_send_check
146 0.0502 fd_install
145 0.0499 bictcp_cong_avoid
143 0.0492 call_rcu
141 0.0485 __down_read
141 0.0485 sock_close
140 0.0481 copy_page_c
138 0.0475 __skb_checksum_complete
138 0.0475 lookup_mnt
137 0.0471 getname
132 0.0454 generic_permission
131 0.0450 find_get_page
130 0.0447 __do_page_cache_readahead
130 0.0447 update_send_head
127 0.0437 get_empty_filp
126 0.0433 __path_lookup_intent_open
124 0.0426 mod_timer
122 0.0420 half_md4_transform
121 0.0416 page_fault
121 0.0416 tcp_sync_mss
120 0.0413 __wake_up
120 0.0413 current_fs_time
119 0.0409 remove_wait_queue
118 0.0406 groups_search
116 0.0399 __handle_mm_fault
115 0.0395 tcp_send_ack
114 0.0392 get_task_mm
109 0.0375 tcp_snd_test
107 0.0368 new_inode
106 0.0364 sock_release
106 0.0364 tcp_init_tso_segs
104 0.0358 inotify_dentry_parent_queue_event
104 0.0358 try_to_del_timer_sync
102 0.0351 add_wait_queue
99 0.0340 cond_resched
97 0.0334 __follow_mount
96 0.0330 __put_unused_fd
95 0.0327 open_namei
94 0.0323 file_move
93 0.0320 clear_inode
93 0.0320 permission
92 0.0316 rw_verify_area
91 0.0313 may_open
91 0.0313 memmove
86 0.0296 __up_write
86 0.0296 dnotify_flush
86 0.0296 tcp_select_initial_window
84 0.0289 sock_common_recvmsg
83 0.0285 __wake_up_bit
83 0.0285 page_cache_readahead
82 0.0282 IRQ0x20_interrupt
81 0.0279 put_page
77 0.0265 inet_sendmsg
76 0.0261 skb_copy_datagram_iovec
76 0.0261 tcp_init_cwnd
75 0.0258 sock_recvmsg
75 0.0258 tcp_event_data_recv
74 0.0254 inet_sk_rebuild_header
74 0.0254 locks_remove_posix
73 0.0251 d_instantiate
68 0.0234 sock_sendmsg
66 0.0227 __rb_rotate_right
65 0.0224 __down_write_nested
65 0.0224 retint_kernel
64 0.0220 alloc_inode
64 0.0220 clear_page_c
64 0.0220 init_timer
63 0.0217 mntput_no_expire
63 0.0217 sk_reset_timer
63 0.0217 tcp_setsockopt
61 0.0210 tcp_check_space
61 0.0210 unmap_vmas
60 0.0206 filp_close
60 0.0206 tcp_v4_tw_remember_stamp
59 0.0203 tcp_rcv_space_adjust
56 0.0193 inotify_inode_queue_event
54 0.0186 __delay
54 0.0186 touch_atime
53 0.0182 sk_stream_rfree
53 0.0182 sys_open
53 0.0182 tcp_cwnd_validate
50 0.0172 copy_to_user
50 0.0172 file_kill
49 0.0168 generic_file_open
46 0.0158 tcp_cong_avoid
44 0.0151 do_filp_open
44 0.0151 inet_sock_destruct
43 0.0148 __rb_rotate_left
43 0.0148 __tcp_ack_snd_check
43 0.0148 inode_init_once
43 0.0148 sprintf
42 0.0144 exit_intr
42 0.0144 sock_fasync
42 0.0144 wake_up_inode
41 0.0141 memset
39 0.0134 __alloc_pages
39 0.0134 free_hot_cold_page
39 0.0134 vfs_permission
38 0.0131 bit_waitqueue
38 0.0131 do_page_fault
38 0.0131 locks_remove_flock
38 0.0131 tcp_unhash
36 0.0124 del_timer_sync
36 0.0124 iput
36 0.0124 iret_label
35 0.0120 file_free_rcu
35 0.0120 file_ra_state_init
35 0.0120 tcp_v4_destroy_sock
34 0.0117 sk_stop_timer
33 0.0113 __lookup_mnt
33 0.0113 inet_getname
33 0.0113 zone_watermark_ok
30 0.0103 copy_page_range
29 0.0100 blockable_page_cache_readahead
29 0.0100 do_wp_page
29 0.0100 hrtimer_run_queues
29 0.0100 sk_alloc
26 0.0089 _spin_lock_bh
26 0.0089 in_group_p
25 0.0086 __put_user_8
25 0.0086 __wake_up_locked
25 0.0086 find_vma
25 0.0086 init_once
24 0.0083 __lock_text_start
23 0.0079 destroy_inode
23 0.0079 tcp_slow_start
22 0.0076 mark_page_accessed
21 0.0072 flush_tlb_page
20 0.0069 _read_lock_irqsave
20 0.0069 memcpy_toiovec
18 0.0062 apic_timer_interrupt
18 0.0062 retint_restore_args
18 0.0062 vm_normal_page
14 0.0048 __down_write
14 0.0048 __tcp_checksum_complete_user
14 0.0048 invalidate_inode_buffers
13 0.0045 mutex_lock
11 0.0038 __get_user_4
11 0.0038 error_exit
11 0.0038 wake_up_bit
10 0.0034 __block_write_full_page
10 0.0034 __down_read_trylock
10 0.0034 copy_from_user
9 0.0031 copy_process
9 0.0031 timespec_trunc
7 0.0024 dup_fd
7 0.0024 filemap_nopage
7 0.0024 lru_cache_add_active
7 0.0024 memcpy
7 0.0024 nameidata_to_filp
7 0.0024 unlink_file_vma
6 0.0021 __find_get_block
6 0.0021 __mutex_init
6 0.0021 find_get_pages_tag
6 0.0021 inode_has_buffers
6 0.0021 mmput
6 0.0021 page_remove_rmap
6 0.0021 sys_mprotect
5 0.0017 __mark_inode_dirty
5 0.0017 do_mmap_pgoff
5 0.0017 error_sti
5 0.0017 free_hot_page
5 0.0017 load_elf_binary
5 0.0017 page_waitqueue
5 0.0017 radix_tree_tag_clear
5 0.0017 rcu_start_batch
5 0.0017 retint_swapgs
4 0.0014 _spin_lock_irqsave
4 0.0014 copy_strings
4 0.0014 free_pages
4 0.0014 free_pgd_range
4 0.0014 free_pgtables
4 0.0014 page_add_file_rmap
4 0.0014 run_local_timers
4 0.0014 sys_rt_sigprocmask
4 0.0014 unlock_page
4 0.0014 vma_link
3 0.0010 __getblk
3 0.0010 __pagevec_lru_add_active
3 0.0010 __strnlen_user
3 0.0010 _write_lock_bh
3 0.0010 anon_vma_prepare
3 0.0010 anon_vma_unlink
3 0.0010 bio_alloc_bioset
3 0.0010 dnotify_parent
3 0.0010 do_exit
3 0.0010 do_wait
3 0.0010 exit_mmap
3 0.0010 kthread_should_stop
3 0.0010 prio_tree_insert
3 0.0010 remove_vma
3 0.0010 retint_check
3 0.0010 vma_prio_tree_add
2 6.9e-04 __block_prepare_write
2 6.9e-04 __bread
2 6.9e-04 __end_that_request_first
2 6.9e-04 __find_get_block_slow
2 6.9e-04 __free_pages
2 6.9e-04 __vm_enough_memory
2 6.9e-04 alloc_page_vma
2 6.9e-04 arch_unmap_area
2 6.9e-04 bio_init
2 6.9e-04 blk_recount_segments
2 6.9e-04 block_write_full_page
2 6.9e-04 clear_page_dirty_for_io
2 6.9e-04 do_group_exit
2 6.9e-04 do_munmap
2 6.9e-04 do_notify_resume
2 6.9e-04 find_vma_prepare
2 6.9e-04 find_vma_prev
2 6.9e-04 generic_fillattr
2 6.9e-04 get_index
2 6.9e-04 get_unmapped_area
2 6.9e-04 get_vma_policy
2 6.9e-04 prio_tree_remove
2 6.9e-04 prio_tree_replace
2 6.9e-04 proc_get_inode
2 6.9e-04 proc_lookup
2 6.9e-04 release_pages
2 6.9e-04 show_vfsmnt
2 6.9e-04 strchr
2 6.9e-04 sys_dup2
2 6.9e-04 sys_faccessat
2 6.9e-04 sys_mmap
2 6.9e-04 test_set_page_writeback
2 6.9e-04 unmap_region
2 6.9e-04 vm_acct_memory
2 6.9e-04 vm_stat_account
2 6.9e-04 vma_adjust
2 6.9e-04 vma_merge
2 6.9e-04 vma_prio_tree_remove
2 6.9e-04 zonelist_policy
1 3.4e-04 __brelse
1 3.4e-04 __clear_user
1 3.4e-04 __d_path
1 3.4e-04 __lookup_hash
1 3.4e-04 __make_request
1 3.4e-04 __page_set_anon_rmap
1 3.4e-04 __pagevec_lru_add
1 3.4e-04 __pmd_alloc
1 3.4e-04 __pollwait
1 3.4e-04 __pte_alloc
1 3.4e-04 __pud_alloc
1 3.4e-04 __put_user_4
1 3.4e-04 add_to_page_cache
1 3.4e-04 alloc_pages_current
1 3.4e-04 blk_rq_map_sg
1 3.4e-04 can_share_swap_page
1 3.4e-04 can_vma_merge_after
1 3.4e-04 copy_semundo
1 3.4e-04 cp_new_stat
1 3.4e-04 cpuset_update_task_memory_state
1 3.4e-04 d_delete
1 3.4e-04 d_path
1 3.4e-04 dcache_readdir
1 3.4e-04 default_llseek
1 3.4e-04 do_arch_prctl
1 3.4e-04 do_brk
1 3.4e-04 do_fsync
1 3.4e-04 do_mpage_readpage
1 3.4e-04 do_sigaction
1 3.4e-04 do_sigaltstack
1 3.4e-04 dupfd
1 3.4e-04 elf_map
1 3.4e-04 end_page_writeback
1 3.4e-04 error_entry
1 3.4e-04 exit_sem
1 3.4e-04 fasync_helper
1 3.4e-04 file_permission
1 3.4e-04 file_read_actor
1 3.4e-04 flush_signal_handlers
1 3.4e-04 flush_thread
1 3.4e-04 generic_delete_inode
1 3.4e-04 generic_file_aio_write
1 3.4e-04 generic_file_buffered_write
1 3.4e-04 generic_file_mmap
1 3.4e-04 get_request_wait
1 3.4e-04 get_signal_to_deliver
1 3.4e-04 get_stack
1 3.4e-04 hrtimer_init
1 3.4e-04 kill_fasync
1 3.4e-04 kmem_flagcheck
1 3.4e-04 meminfo_read_proc
1 3.4e-04 mempool_free
1 3.4e-04 memscan
1 3.4e-04 mutex_unlock
1 3.4e-04 notify_change
1 3.4e-04 nr_blockdev_pages
1 3.4e-04 page_add_new_anon_rmap
1 3.4e-04 path_release
1 3.4e-04 pipe_ioctl
1 3.4e-04 pipe_iov_copy_from_user
1 3.4e-04 poll_freewait
1 3.4e-04 proc_file_read
1 3.4e-04 proc_pident_lookup
1 3.4e-04 ptregscall_common
1 3.4e-04 rb_next
1 3.4e-04 recalc_bh_state
1 3.4e-04 release_task
1 3.4e-04 retint_careful
1 3.4e-04 schedule_tail
1 3.4e-04 seq_escape
1 3.4e-04 set_brk
1 3.4e-04 set_normalized_timespec
1 3.4e-04 si_swapinfo
1 3.4e-04 split_vma
1 3.4e-04 stub_execve
1 3.4e-04 sync_sb_inodes
1 3.4e-04 sys_brk
1 3.4e-04 sys_lseek
1 3.4e-04 sys_munmap
1 3.4e-04 sys_newfstat
1 3.4e-04 sys_newstat
1 3.4e-04 sys_select
1 3.4e-04 sys_write
1 3.4e-04 truncate_complete_page
1 3.4e-04 tty_ioctl
1 3.4e-04 udp_rcv
1 3.4e-04 unlock_buffer
1 3.4e-04 vfs_readdir
1 3.4e-04 vma_prio_tree_insert
1 3.4e-04 worker_thread
CPU: AMD64 processors, speed 2210.08 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
samples % symbol name
124605 59.0887 cpu_idle
13754 6.5223 enter_idle
4677 2.2179 tcp_v4_rcv
3888 1.8437 IRQ0x51_interrupt
3115 1.4772 tcp_ack
2571 1.2192 kmem_cache_free
2151 1.0200 kfree
2121 1.0058 memset_c
1632 0.7739 csum_partial_copy_generic
1611 0.7639 ip_output
1488 0.7056 schedule
1389 0.6587 dev_queue_xmit
1352 0.6411 tcp_rcv_state_process
1272 0.6032 ip_rcv
1184 0.5615 handle_IRQ_event
951 0.4510 ip_queue_xmit
916 0.4344 __do_softirq
900 0.4268 ip_route_input
888 0.4211 csum_partial
870 0.4126 tcp_v4_do_rcv
856 0.4059 ip_local_deliver
851 0.4036 system_call
829 0.3931 netif_receive_skb
812 0.3851 tcp_sendmsg
812 0.3851 tcp_transmit_skb
805 0.3817 __alloc_skb
683 0.3239 kmem_cache_alloc
683 0.3239 local_bh_enable
678 0.3215 __tcp_push_pending_frames
673 0.3191 fget
665 0.3153 __d_lookup
653 0.3097 pfifo_fast_enqueue
639 0.3030 copy_user_generic_string
607 0.2878 preempt_schedule
606 0.2874 net_rx_action
593 0.2812 pfifo_fast_dequeue
589 0.2793 kfree_skbmem
552 0.2618 __fput
550 0.2608 __link_path_walk
528 0.2504 skb_clone
523 0.2480 tcp_parse_options
515 0.2442 ret_from_intr
512 0.2428 number
496 0.2352 _atomic_dec_and_lock
493 0.2338 __kfree_skb
477 0.2262 rt_hash_code
476 0.2257 sock_wfree
459 0.2177 tcp_poll
449 0.2129 tcp_rcv_established
415 0.1968 dput
398 0.1887 sysret_check
366 0.1736 exit_idle
364 0.1726 __mod_timer
352 0.1669 tcp_recvmsg
352 0.1669 thread_return
346 0.1641 sys_close
341 0.1617 get_unused_fd
340 0.1612 rb_erase
337 0.1598 __tcp_select_window
331 0.1570 lock_timer_base
316 0.1498 mask_and_ack_8259A
299 0.1418 ip_append_data
290 0.1375 cache_alloc_refill
286 0.1356 d_alloc
269 0.1276 ret_from_sys_call
268 0.1271 do_path_lookup
267 0.1266 call_softirq
265 0.1257 eth_type_trans
262 0.1242 tcp_current_mss
262 0.1242 tcp_data_queue
262 0.1242 vsnprintf
261 0.1238 __ip_route_output_key
255 0.1209 sk_stream_mem_schedule
249 0.1181 cache_grow
247 0.1171 tcp_rtt_estimator
239 0.1133 __dentry_open
221 0.1048 sys_fcntl
220 0.1043 find_next_zero_bit
216 0.1024 dentry_iput
208 0.0986 try_to_wake_up
197 0.0934 call_rcu
196 0.0929 strncpy_from_user
195 0.0925 do_lookup
191 0.0906 sock_recvmsg
189 0.0896 tcp_send_fin
188 0.0892 sys_recvfrom
182 0.0863 sock_def_readable
178 0.0844 restore_args
176 0.0835 common_interrupt
174 0.0825 release_sock
171 0.0811 do_generic_mapping_read
167 0.0792 schedule_timeout
163 0.0773 generic_drop_inode
163 0.0773 get_page_from_freelist
163 0.0773 percpu_counter_mod
160 0.0759 skb_checksum
159 0.0754 del_timer
155 0.0735 do_sys_open
153 0.0726 memcpy_c
149 0.0707 memcmp
145 0.0688 __skb_checksum_complete
144 0.0683 link_path_walk
144 0.0683 tcp_v4_send_check
141 0.0669 fd_install
140 0.0664 get_empty_filp
139 0.0659 fget_light
131 0.0621 current_fs_time
126 0.0598 mod_timer
125 0.0593 bictcp_cong_avoid
123 0.0583 tcp_init_tso_segs
121 0.0574 update_send_head
120 0.0569 __put_unused_fd
119 0.0564 __do_page_cache_readahead
119 0.0564 alloc_inode
118 0.0560 rb_insert_color
115 0.0545 prepare_to_wait
112 0.0531 locks_remove_posix
112 0.0531 new_inode
111 0.0526 half_md4_transform
109 0.0517 fput
108 0.0512 tcp_sync_mss
105 0.0498 get_task_mm
104 0.0493 clear_inode
102 0.0484 find_get_page
102 0.0484 tcp_select_initial_window
101 0.0479 lookup_mnt
101 0.0479 tcp_send_ack
99 0.0469 sock_close
98 0.0465 try_to_del_timer_sync
97 0.0460 page_cache_readahead
97 0.0460 tcp_snd_test
94 0.0446 generic_permission
94 0.0446 getname
92 0.0436 may_open
91 0.0432 tcp_event_data_recv
90 0.0427 open_namei
89 0.0422 IRQ0x20_interrupt
88 0.0417 inotify_dentry_parent_queue_event
87 0.0413 put_page
85 0.0403 copy_page_c
84 0.0398 d_instantiate
81 0.0384 finish_wait
80 0.0379 __wake_up
79 0.0375 expand_files
77 0.0365 groups_search
75 0.0356 inet_sendmsg
75 0.0356 skb_copy_datagram_iovec
73 0.0346 file_free_rcu
72 0.0341 __path_lookup_intent_open
72 0.0341 permission
71 0.0337 __follow_mount
71 0.0337 memmove
70 0.0332 dnotify_flush
69 0.0327 cond_resched
69 0.0327 igrab
68 0.0322 sk_reset_timer
66 0.0313 sockfd_lookup
65 0.0308 tcp_setsockopt
64 0.0303 sock_sendmsg
62 0.0294 __wake_up_bit
62 0.0294 file_move
62 0.0294 inotify_inode_queue_event
62 0.0294 retint_kernel
62 0.0294 tcp_v4_tw_remember_stamp
60 0.0285 sprintf
56 0.0266 copy_to_user
56 0.0266 sock_common_recvmsg
56 0.0266 tcp_check_space
56 0.0266 tcp_cwnd_validate
55 0.0261 file_kill
55 0.0261 inet_sk_rebuild_header
55 0.0261 tcp_init_cwnd
54 0.0256 __handle_mm_fault
54 0.0256 init_timer
52 0.0247 filp_close
52 0.0247 mutex_unlock
52 0.0247 rw_verify_area
51 0.0242 sock_release
48 0.0228 __tcp_ack_snd_check
48 0.0228 sys_open
47 0.0223 inode_init_once
47 0.0223 locks_remove_flock
47 0.0223 mntput_no_expire
47 0.0223 touch_atime
46 0.0218 sk_stream_rfree
45 0.0213 page_fault
45 0.0213 unmap_vmas
45 0.0213 wake_up_inode
44 0.0209 clear_page_c
44 0.0209 memset
42 0.0199 __delay
42 0.0199 tcp_cong_avoid
41 0.0194 __rb_rotate_left
39 0.0185 do_filp_open
39 0.0185 memcpy_toiovec
39 0.0185 sock_fasync
38 0.0180 __put_user_8
38 0.0180 exit_intr
35 0.0166 zone_watermark_ok
34 0.0161 iput
33 0.0156 iret_label
32 0.0152 free_hot_cold_page
32 0.0152 tcp_unhash
31 0.0147 __alloc_pages
31 0.0147 sk_alloc
30 0.0142 __lookup_mnt
30 0.0142 mutex_lock
29 0.0138 generic_file_open
28 0.0133 _spin_lock_bh
28 0.0133 bit_waitqueue
28 0.0133 tcp_rcv_space_adjust
27 0.0128 inet_sock_destruct
26 0.0123 sk_stop_timer
26 0.0123 tcp_slow_start
24 0.0114 hrtimer_run_queues
24 0.0114 inet_getname
22 0.0104 file_ra_state_init
22 0.0104 vfs_permission
21 0.0100 blockable_page_cache_readahead
21 0.0100 tcp_v4_destroy_sock
20 0.0095 init_once
19 0.0090 do_wp_page
18 0.0085 destroy_inode
16 0.0076 apic_timer_interrupt
16 0.0076 copy_from_user
16 0.0076 del_timer_sync
15 0.0071 in_group_p
15 0.0071 invalidate_inode_buffers
14 0.0066 do_page_fault
13 0.0062 mark_page_accessed
13 0.0062 retint_restore_args
12 0.0057 __get_user_4
11 0.0052 copy_page_range
11 0.0052 find_vma
11 0.0052 wake_up_bit
10 0.0047 __down_read
9 0.0043 rcu_start_batch
8 0.0038 __tcp_checksum_complete_user
8 0.0038 __up_read
8 0.0038 flush_tlb_page
8 0.0038 free_pages
8 0.0038 timespec_trunc
6 0.0028 __find_get_block
6 0.0028 __rb_rotate_right
6 0.0028 error_sti
6 0.0028 kmem_flagcheck
6 0.0028 retint_swapgs
5 0.0024 __down_read_trylock
5 0.0024 _spin_lock_irqsave
5 0.0024 inode_has_buffers
5 0.0024 retint_check
4 0.0019 __mutex_init
4 0.0019 nameidata_to_filp
4 0.0019 prio_tree_insert
4 0.0019 proc_lookup
4 0.0019 run_workqueue
3 0.0014 __getblk
3 0.0014 __iget
3 0.0014 __pagevec_lru_add_active
3 0.0014 __strnlen_user
3 0.0014 _write_lock_bh
3 0.0014 copy_process
3 0.0014 error_exit
3 0.0014 filemap_nopage
3 0.0014 mmput
3 0.0014 sys_mprotect
3 0.0014 unlink_file_vma
3 0.0014 vma_adjust
2 9.5e-04 __clear_user
2 9.5e-04 __end_that_request_first
2 9.5e-04 __pte_alloc
2 9.5e-04 __set_page_dirty_nobuffers
2 9.5e-04 _read_lock_bh
2 9.5e-04 _read_lock_irqsave
2 9.5e-04 add_to_page_cache
2 9.5e-04 anon_vma_prepare
2 9.5e-04 anon_vma_unlink
2 9.5e-04 d_lookup
2 9.5e-04 do_exit
2 9.5e-04 do_mmap_pgoff
2 9.5e-04 do_munmap
2 9.5e-04 do_wait
2 9.5e-04 dup_fd
2 9.5e-04 flush_signal_handlers
2 9.5e-04 free_hot_page
2 9.5e-04 generic_file_aio_read
2 9.5e-04 generic_fillattr
2 9.5e-04 generic_make_request
2 9.5e-04 kthread_should_stop
2 9.5e-04 load_elf_binary
2 9.5e-04 lru_add_drain
2 9.5e-04 lru_cache_add_active
2 9.5e-04 memcpy
2 9.5e-04 mempool_free
2 9.5e-04 mm_init
2 9.5e-04 page_remove_rmap
2 9.5e-04 put_unused_fd
2 9.5e-04 radix_tree_tag_clear
2 9.5e-04 rb_next
2 9.5e-04 retint_with_reschedule
2 9.5e-04 run_local_timers
2 9.5e-04 strchr
2 9.5e-04 vfs_ioctl
2 9.5e-04 vma_link
1 4.7e-04 __add_entropy_words
1 4.7e-04 __anon_vma_link
1 4.7e-04 __brelse
1 4.7e-04 __d_path
1 4.7e-04 __down_write
1 4.7e-04 __find_get_block_slow
1 4.7e-04 __free_pages
1 4.7e-04 __generic_file_aio_write_nolock
1 4.7e-04 __lookup_hash
1 4.7e-04 __make_request
1 4.7e-04 __mark_inode_dirty
1 4.7e-04 __page_set_anon_rmap
1 4.7e-04 __pagevec_lru_add
1 4.7e-04 __pagevec_release
1 4.7e-04 __remove_shared_vm_struct
1 4.7e-04 __up_write
1 4.7e-04 __user_walk_fd
1 4.7e-04 __vm_enough_memory
1 4.7e-04 __writeback_single_inode
1 4.7e-04 alloc_buffer_head
1 4.7e-04 alloc_page_buffers
1 4.7e-04 alloc_pages_current
1 4.7e-04 bio_alloc_bioset
1 4.7e-04 blk_done_softirq
1 4.7e-04 blk_recount_segments
1 4.7e-04 blk_rq_map_sg
1 4.7e-04 can_vma_merge_before
1 4.7e-04 cfq_set_request
1 4.7e-04 copy_strings
1 4.7e-04 cpuset_update_task_memory_state
1 4.7e-04 create_empty_buffers
1 4.7e-04 deny_write_access
1 4.7e-04 do_arch_prctl
1 4.7e-04 do_mpage_readpage
1 4.7e-04 do_select
1 4.7e-04 do_sigaction
1 4.7e-04 exit_mm
1 4.7e-04 exit_mmap
1 4.7e-04 find_vma_prepare
1 4.7e-04 find_vma_prev
1 4.7e-04 free_pgtables
1 4.7e-04 get_locked_pte
1 4.7e-04 get_request_wait
1 4.7e-04 get_signal_to_deliver
1 4.7e-04 get_unmapped_area
1 4.7e-04 init_once
1 4.7e-04 init_request_from_bio
1 4.7e-04 inode_setattr
1 4.7e-04 ll_rw_block
1 4.7e-04 make_ahead_window
1 4.7e-04 mapping_tagged
1 4.7e-04 mempool_alloc
1 4.7e-04 n_tty_write_wakeup
1 4.7e-04 notify_change
1 4.7e-04 page_add_file_rmap
1 4.7e-04 pipe_iov_copy_from_user
1 4.7e-04 pipe_write
1 4.7e-04 preempt_schedule_irq
1 4.7e-04 proc_file_read
1 4.7e-04 remove_vma
1 4.7e-04 sched_balance_self
1 4.7e-04 sched_fork
1 4.7e-04 search_binary_handler
1 4.7e-04 sigprocmask
1 4.7e-04 stub_execve
1 4.7e-04 sync_dirty_buffer
1 4.7e-04 sys_execve
1 4.7e-04 sys_ftruncate
1 4.7e-04 tty_ldisc_ref_wait
1 4.7e-04 vfs_write
1 4.7e-04 writeback_inodes