Re: >10% performance degradation since 2.6.18

[Andi Kleen]

On Mon, Jul 06, 2009 at 02:16:38PM -0700, Ma, Chinang wrote:
> >
> >> 0.8246 kmem_cache_free             0.8801 kmem_cache_free
> >> 0.7108 schedule                    0.7774 __blockdev_direct_IO
> >> 0.6733 scsi_request_fn             0.7031 scsi_request_fn
> >> 0.6114 kmem_cache_alloc            0.5317 __schedule
> >> 0.4207 follow_hugetlb_page         0.3922 task_rq_lock
> >> 0.4062 list_del                    0.3629 sd_prep_fn
> >> 0.3400 __switch_to                 0.3504 list_del
> >
> >A callgraph for the list_del/list_dels would be interesting.
> >Presumably that's cache misses, maybe we can do something
> >here with some strategic prefetches or more batching.
> >
> >> 0.3339 generic_make_request        0.3382 __sigsetjmp
> >
> >__sigsetjmp is in glibc, isn't it?
> 
> Yes, __sigsetjmp from glibc

Strange that it takes that much CPU time.

> 
> >
> >> 0.3204 memmove                     0.3270 __switch_to
> >> 0.3088 __sigsetjmp                 0.3257 generic_make_request
> >> 0.2848 get_request                 0.3116 kfree
> >> 0.2804 lock_timer_base             0.2895 memmove
> >
> >memmove? Is that glibc or the kernel? (it would be good if that
> >was marked in the listing)
> 
> Memmove is from glibc. I will filter out all user space libraries in the future report 

it would be good to find out why __errno_location is that costly.
in theory it should be only
	
	mov %fs:xxxx,%rax
	ret

> >A callgraph for it would be useful if it's kernel.
> >
> >> 0.2572 __end_that_request_first    0.2402 aio_complete
> >> 0.2567 fget_light                  0.2382 mptscsih_qcmd
> >> 0.2531 submit_page_section         0.2342 fget
> >> 0.2428 mempool_alloc               0.2277 gup_huge_pmd
> >> 0.2428 __generic_file_aio_read     0.2264 submit_page_section
> >> 0.2368 touch_atime                 0.2204 touch_atime
> >
> >Did you not run with noatime? If it runs with noatime and touch_atime
> >still takes that many cycles something must be wrong.
> >
> 
> Did not try noatime option yet. Will try it next time.

noatime would only help with the patch from the previous message.
Otherwise it'll still take the lock all the time.

Also 2.6.30 defaults to "relatime" which should be often
equivalent to noatime. But still the patch might help.
> 
> Will try collect this data.

It's probably not needed for now; if you can get callgraphs.
The memmove part would have been only interesting if the hot memmove
was in kernel. 

-Andi
-- 
ak@xxxxxxxxxxxxxxx -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/