Re: [lkp] [jbd2] de92c8caf16: no primary result change, +270.9% vmstat.procs.b, -61.0% vmstat.procs.r
From: Jan Kara
Date: Thu Jun 18 2015 - 05:14:34 EST
On Thu 18-06-15 11:20:59, Huang Ying wrote:
> FYI, we noticed the below changes on
>
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> commit de92c8caf16ca84926fa31b7a5590c0fb9c0d5ca ("jbd2: speedup jbd2_journal_get_[write|undo]_access()")
>
> It appears that more processes are put in uninterruptible state after the
> commit.
Thanks for info! So looking at the overall perf statistics we seem to be
using somewhat less CPU and more quickly proceeding to wait for the IO
which is the bottleneck for this workload. So I think we are fine.
Honza
> =========================================================================================
> tbox_group/testcase/rootfs/kconfig/compiler/wait_disks_timeout/runtime/disk/md/iosched/fs/nr_threads:
> lkp-st02/dd-write/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/300/5m/11HDD/JBOD/cfq/ext4/10dd
>
> commit:
> 8b00f400eedf91d074f831077003c0d4d9147377
> de92c8caf16ca84926fa31b7a5590c0fb9c0d5ca
>
> 8b00f400eedf91d0 de92c8caf16ca84926fa31b7a5
> ---------------- --------------------------
> %stddev %change %stddev
> \ | \
> 48241 ± 8% +139.2% 115399 ± 2% softirqs.SCHED
> 1565 ± 5% +12.4% 1759 ± 6% time.involuntary_context_switches
> 391202 ± 5% +110.7% 824392 ± 1% proc-vmstat.pgactivate
> 70008 ± 56% -73.1% 18828 ± 50% proc-vmstat.pgrotated
> 21.50 ± 10% +270.9% 79.75 ± 1% vmstat.procs.b
> 86.00 ± 1% -61.0% 33.50 ± 2% vmstat.procs.r
> 23827 ± 0% -2.7% 23187 ± 0% vmstat.system.in
> 2310 ± 4% -16.9% 1919 ± 10% slabinfo.ext4_io_end.active_objs
> 2310 ± 4% -16.9% 1919 ± 10% slabinfo.ext4_io_end.num_objs
> 2110 ± 18% +18.4% 2499 ± 3% slabinfo.kmalloc-96.active_objs
> 2163 ± 13% +15.5% 2499 ± 3% slabinfo.kmalloc-96.num_objs
> 1.683e+12 ± 1% -6.1% 1.58e+12 ± 0% perf-stat.L1-dcache-loads
> 6.332e+08 ± 2% -6.8% 5.904e+08 ± 1% perf-stat.L1-dcache-prefetches
> 9.849e+11 ± 1% -4.0% 9.457e+11 ± 0% perf-stat.L1-dcache-stores
> 6.932e+10 ± 1% +6.7% 7.397e+10 ± 0% perf-stat.L1-icache-load-misses
> 5.064e+12 ± 1% -5.7% 4.777e+12 ± 0% perf-stat.L1-icache-loads
> 1.823e+09 ± 1% +8.0% 1.969e+09 ± 0% perf-stat.LLC-load-misses
> 8.519e+11 ± 1% -3.5% 8.222e+11 ± 0% perf-stat.branch-instructions
> 8.99e+09 ± 2% -23.1% 6.916e+09 ± 0% perf-stat.branch-load-misses
> 8.513e+11 ± 1% -3.5% 8.216e+11 ± 0% perf-stat.branch-loads
> 8.967e+09 ± 2% -23.5% 6.858e+09 ± 0% perf-stat.branch-misses
> 8.059e+11 ± 1% -3.4% 7.786e+11 ± 0% perf-stat.bus-cycles
> 3.658e+09 ± 1% +5.6% 3.863e+09 ± 0% perf-stat.cache-misses
> 1.612e+11 ± 2% +5.6% 1.701e+11 ± 0% perf-stat.cache-references
> 6.452e+12 ± 1% -3.5% 6.227e+12 ± 0% perf-stat.cpu-cycles
> 92243 ± 6% +175.6% 254241 ± 2% perf-stat.cpu-migrations
> 1.305e+10 ± 12% -15.2% 1.106e+10 ± 4% perf-stat.dTLB-load-misses
> 1.683e+12 ± 1% -6.2% 1.579e+12 ± 0% perf-stat.dTLB-loads
> 9.845e+11 ± 1% -4.0% 9.446e+11 ± 0% perf-stat.dTLB-stores
> 31933083 ± 9% -31.3% 21923765 ± 2% perf-stat.iTLB-load-misses
> 4.638e+12 ± 1% -3.6% 4.471e+12 ± 0% perf-stat.iTLB-loads
> 4.639e+12 ± 1% -3.6% 4.471e+12 ± 0% perf-stat.instructions
> 6.448e+12 ± 1% -3.4% 6.229e+12 ± 0% perf-stat.ref-cycles
> 1.36 ± 4% +17.9% 1.60 ± 3% perf-profile.cpu-cycles.__clear_user.iov_iter_zero.read_iter_zero.__vfs_read.vfs_read
> 3.48 ± 4% +11.1% 3.87 ± 2% perf-profile.cpu-cycles.__ext4_get_inode_loc.ext4_get_inode_loc.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode
> 5.15 ± 1% +22.6% 6.31 ± 2% perf-profile.cpu-cycles.__ext4_handle_dirty_metadata.ext4_mark_iloc_dirty.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
> 7.90 ± 3% -87.7% 0.98 ± 14% perf-profile.cpu-cycles.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
> 8.53 ± 0% +17.7% 10.04 ± 1% perf-profile.cpu-cycles.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter
> 0.91 ± 3% +11.3% 1.01 ± 9% perf-profile.cpu-cycles.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end
> 2.99 ± 2% +17.4% 3.50 ± 4% perf-profile.cpu-cycles.__ext4_journal_stop.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter
> 0.78 ± 6% +22.4% 0.96 ± 6% perf-profile.cpu-cycles.__ext4_journal_stop.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end
> 1.19 ± 3% +29.1% 1.53 ± 4% perf-profile.cpu-cycles.__find_get_block.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc.ext4_reserve_inode_write
> 1.88 ± 4% +20.2% 2.26 ± 4% perf-profile.cpu-cycles.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc.ext4_reserve_inode_write.ext4_mark_inode_dirty
> 26.69 ± 0% -16.5% 22.29 ± 1% perf-profile.cpu-cycles.__mark_inode_dirty.generic_write_end.ext4_da_write_end.generic_perform_write.__generic_file_write_iter
> 3.39 ± 3% +18.7% 4.02 ± 2% perf-profile.cpu-cycles.__vfs_read.vfs_read.sys_read.system_call_fastpath
> 0.99 ± 3% +36.5% 1.35 ± 5% perf-profile.cpu-cycles._raw_read_lock.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write
> 4.13 ± 6% -99.8% 0.01 ±-10000% perf-profile.cpu-cycles.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty
> 34.70 ± 0% -10.1% 31.19 ± 1% perf-profile.cpu-cycles.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write
> 26.13 ± 1% -16.6% 21.80 ± 1% perf-profile.cpu-cycles.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end.generic_perform_write
> 3.76 ± 3% +9.2% 4.11 ± 2% perf-profile.cpu-cycles.ext4_get_inode_loc.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty
> 10.35 ± 0% +13.4% 11.74 ± 2% perf-profile.cpu-cycles.ext4_mark_iloc_dirty.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
> 23.99 ± 1% -19.4% 19.35 ± 2% perf-profile.cpu-cycles.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end
> 11.94 ± 4% -53.5% 5.55 ± 3% perf-profile.cpu-cycles.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end
> 30.83 ± 0% -13.8% 26.58 ± 1% perf-profile.cpu-cycles.generic_write_end.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter
> 7.42 ± 0% +16.8% 8.67 ± 1% perf-profile.cpu-cycles.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter
> 1.62 ± 7% -100.0% 0.00 ± -1% perf-profile.cpu-cycles.jbd2_journal_add_journal_head.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty
> 3.63 ± 1% +19.7% 4.34 ± 3% perf-profile.cpu-cycles.jbd2_journal_dirty_metadata.__ext4_handle_dirty_metadata.ext4_mark_iloc_dirty.ext4_mark_inode_dirty.ext4_dirty_inode
> 7.43 ± 3% -92.3% 0.57 ± 22% perf-profile.cpu-cycles.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode
> 1.01 ± 9% +45.9% 1.47 ± 5% perf-profile.cpu-cycles.jbd2_journal_grab_journal_head.jbd2_journal_dirty_metadata.__ext4_handle_dirty_metadata.ext4_mark_iloc_dirty.ext4_mark_inode_dirty
> 1.00 ± 6% -100.0% 0.00 ± -1% perf-profile.cpu-cycles.jbd2_journal_put_journal_head.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty
> 2.68 ± 1% +17.4% 3.15 ± 5% perf-profile.cpu-cycles.jbd2_journal_stop.__ext4_journal_stop.ext4_da_write_end.generic_perform_write.__generic_file_write_iter
> 3.89 ± 3% +18.0% 4.58 ± 2% perf-profile.cpu-cycles.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write
> 6.77 ± 1% +14.2% 7.74 ± 2% perf-profile.cpu-cycles.sys_read.system_call_fastpath
> 0.90 ± 10% -100.0% 0.00 ± -1% perf-profile.cpu-cycles.unlock_buffer.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write
> 5.98 ± 1% +16.8% 6.98 ± 1% perf-profile.cpu-cycles.vfs_read.sys_read.system_call_fastpath
> 1323 ± 24% -67.3% 433.50 ± 14% sched_debug.cfs_rq[0]:/.utilization_load_avg
> 1153750 ± 5% +12.2% 1294517 ± 1% sched_debug.cfs_rq[1]:/.min_vruntime
> -260439 ±-69% -103.8% 9885 ±199% sched_debug.cfs_rq[1]:/.spread0
> 1237 ± 18% -78.8% 262.50 ± 28% sched_debug.cfs_rq[1]:/.utilization_load_avg
> 87.25 ± 17% +165.6% 231.75 ± 56% sched_debug.cfs_rq[2]:/.load
> 942.25 ± 16% -78.5% 203.00 ± 36% sched_debug.cfs_rq[2]:/.utilization_load_avg
> 1749 ± 26% -48.3% 903.50 ± 72% sched_debug.cfs_rq[3]:/.blocked_load_avg
> 206.25 ± 39% -44.5% 114.50 ± 14% sched_debug.cfs_rq[3]:/.load
> 162.00 ± 3% -25.8% 120.25 ± 21% sched_debug.cfs_rq[3]:/.runnable_load_avg
> -241296 ±-76% -102.5% 5936 ±118% sched_debug.cfs_rq[3]:/.spread0
> 1930 ± 22% -46.9% 1024 ± 63% sched_debug.cfs_rq[3]:/.tg_load_contrib
> 1203 ± 9% -72.3% 333.50 ± 46% sched_debug.cfs_rq[3]:/.utilization_load_avg
> 921.50 ± 95% +131.9% 2137 ± 37% sched_debug.cfs_rq[4]:/.blocked_load_avg
> 1080842 ± 3% +17.8% 1273061 ± 0% sched_debug.cfs_rq[4]:/.min_vruntime
> -333430 ±-48% -96.5% -11621 ±-147% sched_debug.cfs_rq[4]:/.spread0
> 1077 ± 86% +110.7% 2269 ± 35% sched_debug.cfs_rq[4]:/.tg_load_contrib
> 1288 ± 19% -76.7% 300.50 ± 23% sched_debug.cfs_rq[4]:/.utilization_load_avg
> 1205 ± 14% -68.7% 376.75 ± 52% sched_debug.cfs_rq[5]:/.utilization_load_avg
> 1095954 ± 4% +17.7% 1290354 ± 0% sched_debug.cfs_rq[6]:/.min_vruntime
> -318381 ±-49% -101.8% 5587 ±256% sched_debug.cfs_rq[6]:/.spread0
> 1265 ± 14% -71.3% 363.50 ± 46% sched_debug.cfs_rq[6]:/.utilization_load_avg
> 1251 ± 20% -76.9% 289.75 ± 32% sched_debug.cfs_rq[7]:/.utilization_load_avg
> 10.00 ± 23% -75.0% 2.50 ± 20% sched_debug.cpu#0.nr_running
> 10.50 ± 14% -76.2% 2.50 ± 44% sched_debug.cpu#1.nr_running
> 1793 ± 12% -26.9% 1311 ± 12% sched_debug.cpu#2.curr->pid
> 8.00 ± 26% -71.9% 2.25 ± 36% sched_debug.cpu#2.nr_running
> -246.00 ±-46% -93.2% -16.75 ±-607% sched_debug.cpu#2.nr_uninterruptible
> 10.50 ± 14% -73.8% 2.75 ± 30% sched_debug.cpu#3.nr_running
> 128.50 ± 11% -20.0% 102.75 ± 8% sched_debug.cpu#4.cpu_load[0]
> 127.75 ± 9% -16.0% 107.25 ± 11% sched_debug.cpu#4.cpu_load[1]
> 10.50 ± 20% -73.8% 2.75 ± 30% sched_debug.cpu#4.nr_running
> 9.75 ± 15% -71.8% 2.75 ± 15% sched_debug.cpu#5.nr_running
> 122.00 ± 9% +19.7% 146.00 ± 8% sched_debug.cpu#6.cpu_load[1]
> 120.00 ± 7% +25.8% 151.00 ± 15% sched_debug.cpu#6.cpu_load[2]
> 10.75 ± 21% -72.1% 3.00 ± 52% sched_debug.cpu#6.nr_running
> 1971 ± 9% -34.0% 1300 ± 17% sched_debug.cpu#7.curr->pid
> 10.00 ± 23% -82.5% 1.75 ± 24% sched_debug.cpu#7.nr_running
> 40663 ± 2% -18.6% 33116 ± 0% latency_stats.avg.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 508303 ± 3% +3.3% 525071 ± 2% latency_stats.avg.blk_execute_rq.sg_io.scsi_cmd_ioctl.scsi_cmd_blk_ioctl.sd_ioctl.[sd_mod].blkdev_ioctl.block_ioctl.do_vfs_ioctl.SyS_ioctl.system_call_fastpath
> 65195 ± 4% -13.7% 56254 ± 3% latency_stats.avg.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time.__generic_file_write_iter.ext4_file_write_iter.__vfs_write
> 3440 ± 71% -100.0% 0.00 ± -1% latency_stats.avg.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate.SyS_ftruncate
> 453.25 ± 4% +67.1% 757.50 ± 1% latency_stats.avg.do_wait.SyS_wait4.system_call_fastpath
> 968.50 ± 16% +28.9% 1248 ± 10% latency_stats.avg.ep_poll.SyS_epoll_wait.system_call_fastpath
> 438.50 ± 8% -32.7% 295.00 ± 1% latency_stats.avg.ring_buffer_wait.wait_on_pipe.tracing_wait_pipe.tracing_read_pipe.__vfs_read.vfs_read.SyS_read.system_call_fastpath
> 686.00 ± 22% -21.1% 541.50 ± 16% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_open
> 874.50 ± 55% -47.7% 457.75 ± 11% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
> 1169 ±140% -86.2% 161.75 ± 5% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_lookup.[cifs].lookup_real.__lookup_hash.do_unlinkat.SyS_unlink.system_call_fastpath
> 199.00 ± 7% -19.5% 160.25 ± 15% latency_stats.avg.wait_for_response.[cifs].SendReceive2.[cifs].CIFSSMBWrite2.[cifs].cifs_sync_write.[cifs].cifs_write.[cifs].cifs_write_end.[cifs].generic_perform_write.__generic_file_write_iter.generic_file_write_iter.cifs_strict_writev.[cifs].__vfs_write.vfs_write
> 107505 ± 14% +51.7% 163094 ± 11% latency_stats.avg.wait_transaction_locked.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write
> 103169 ± 12% +514.6% 634103 ± 1% latency_stats.hits.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 1.00 ± 81% +225.0% 3.25 ± 50% latency_stats.hits.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve
> 26.00 ± 14% +30.8% 34.00 ± 13% latency_stats.hits.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write
> 2835 ± 5% +11.4% 3158 ± 3% latency_stats.hits.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time.__generic_file_write_iter.ext4_file_write_iter.__vfs_write
> 663.50 ± 10% -40.2% 396.75 ± 8% latency_stats.hits.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter
> 1.75 ± 47% -100.0% 0.00 ± -1% latency_stats.hits.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate.SyS_ftruncate
> 1661 ± 0% +17.3% 1948 ± 1% latency_stats.hits.do_wait.SyS_wait4.system_call_fastpath
> 142.50 ± 6% +15.3% 164.25 ± 11% latency_stats.hits.ep_poll.SyS_epoll_wait.system_call_fastpath
> 394313 ± 0% -10.2% 354275 ± 4% latency_stats.hits.hrtimer_nanosleep.SyS_nanosleep.system_call_fastpath
> 465945 ± 23% -37.9% 289496 ± 4% latency_stats.hits.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
> 337392 ± 8% -49.8% 169230 ± 8% latency_stats.hits.ring_buffer_wait.wait_on_pipe.tracing_wait_pipe.tracing_read_pipe.__vfs_read.vfs_read.SyS_read.system_call_fastpath
> 91.75 ± 47% -87.6% 11.33 ±141% latency_stats.hits.wait_iff_congested.shrink_inactive_list.shrink_lruvec.shrink_zone.do_try_to_free_pages.try_to_free_pages.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
> 1123 ± 3% -41.6% 656.00 ± 6% latency_stats.hits.wait_transaction_locked.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 4105 ± 0% -38.2% 2535 ± 3% latency_stats.hits.wait_transaction_locked.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_dirty_inode.__mark_inode_dirty.generic_update_time.file_update_time.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write
> 1030093 ± 9% +11.9% 1152679 ± 6% latency_stats.max.blk_execute_rq.sg_io.scsi_cmd_ioctl.scsi_cmd_blk_ioctl.sd_ioctl.[sd_mod].blkdev_ioctl.block_ioctl.do_vfs_ioctl.SyS_ioctl.system_call_fastpath
> 309120 ± 43% -41.6% 180418 ± 23% latency_stats.max.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write
> 4459 ± 75% -100.0% 0.00 ± -1% latency_stats.max.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate.SyS_ftruncate
> 4823 ± 1% +2.5% 4943 ± 0% latency_stats.max.ep_poll.SyS_epoll_wait.system_call_fastpath
> 1271 ± 69% -55.1% 570.75 ± 8% latency_stats.max.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_do_create.[cifs].cifs_atomic_open.[cifs].do_last.path_openat.do_filp_open.do_sys_open.SyS_open.system_call_fastpath
> 11400 ±126% -91.8% 939.75 ± 37% latency_stats.max.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
> 1169 ±140% -86.2% 161.75 ± 5% latency_stats.max.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_lookup.[cifs].lookup_real.__lookup_hash.do_unlinkat.SyS_unlink.system_call_fastpath
> 4.2e+09 ± 13% +399.9% 2.1e+10 ± 1% latency_stats.sum.balance_dirty_pages.balance_dirty_pages_ratelimited.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 24398593 ± 3% +3.3% 25203443 ± 2% latency_stats.sum.blk_execute_rq.sg_io.scsi_cmd_ioctl.scsi_cmd_blk_ioctl.sd_ioctl.[sd_mod].blkdev_ioctl.block_ioctl.do_vfs_ioctl.SyS_ioctl.system_call_fastpath
> 10100882 ± 15% -24.3% 7648970 ± 12% latency_stats.sum.call_rwsem_down_read_failed.ext4_da_get_block_prep.__block_write_begin.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 6.00 ± 81% +216.7% 19.00 ± 56% latency_stats.sum.call_rwsem_down_write_failed.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.SyS_execve.return_from_execve
> 38146064 ± 11% -40.7% 22612285 ± 6% latency_stats.sum.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty.generic_write_end.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter
> 7994 ±100% -100.0% 0.00 ± -1% latency_stats.sum.do_get_write_access.jbd2_journal_get_write_access.__ext4_journal_get_write_access.ext4_reserve_inode_write.ext4_mark_inode_dirty.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate.do_sys_ftruncate.SyS_ftruncate
> 753440 ± 3% +96.0% 1476939 ± 2% latency_stats.sum.do_wait.SyS_wait4.system_call_fastpath
> 139159 ± 21% +49.1% 207524 ± 22% latency_stats.sum.ep_poll.SyS_epoll_wait.system_call_fastpath
> 22735513 ± 0% -10.0% 20470551 ± 4% latency_stats.sum.hrtimer_nanosleep.SyS_nanosleep.system_call_fastpath
> 2.014e+08 ± 6% -51.4% 97800459 ± 4% latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
> 1.488e+08 ± 14% -66.4% 49982280 ± 7% latency_stats.sum.ring_buffer_wait.wait_on_pipe.tracing_wait_pipe.tracing_read_pipe.__vfs_read.vfs_read.SyS_read.system_call_fastpath
> 79639 ± 22% -21.2% 62767 ± 17% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_open
> 25189 ± 56% -49.1% 12823 ± 11% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
> 1169 ±140% -86.2% 161.75 ± 5% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_lookup.[cifs].lookup_real.__lookup_hash.do_unlinkat.SyS_unlink.system_call_fastpath
> 24794 ± 13% -25.1% 18564 ± 14% latency_stats.sum.wait_for_response.[cifs].SendReceive2.[cifs].CIFSSMBWrite2.[cifs].cifs_sync_write.[cifs].cifs_write.[cifs].cifs_write_end.[cifs].generic_perform_write.__generic_file_write_iter.generic_file_write_iter.cifs_strict_writev.[cifs].__vfs_write.vfs_write
> 108340 ± 27% -81.3% 20256 ±141% latency_stats.sum.wait_iff_congested.shrink_inactive_list.shrink_lruvec.shrink_zone.do_try_to_free_pages.try_to_free_pages.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin
> 1.015e+08 ± 11% -36.3% 64626646 ± 16% latency_stats.sum.wait_transaction_locked.start_this_handle.jbd2__journal_start.__ext4_journal_start_sb.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.__vfs_write.vfs_write.SyS_write.system_call_fastpath
> 26524 ± 5% +16.1% 30791 ± 4% ftrace.balance_dirty_pages.sdb.bdi_dirty
> 179.00 ± 15% -46.4% 96.00 ± 4% ftrace.balance_dirty_pages.sdb.dirtied
> 179.25 ± 15% -45.0% 98.50 ± 4% ftrace.balance_dirty_pages.sdb.dirtied_pause
> 58927 ± 13% -80.8% 11327 ± 3% ftrace.balance_dirty_pages.sdb.dirty_ratelimit
> -3430677 ±-173% -100.0% -11.00 ±-75% ftrace.balance_dirty_pages.sdb.pause
> 22.50 ± 25% +76.7% 39.75 ± 3% ftrace.balance_dirty_pages.sdb.period
> 80869 ± 18% -86.9% 10600 ± 2% ftrace.balance_dirty_pages.sdb.task_ratelimit
> 3430696 ±173% -100.0% 49.25 ± 18% ftrace.balance_dirty_pages.sdb.think
> 190.25 ± 13% -51.8% 91.75 ± 5% ftrace.balance_dirty_pages.sdc.dirtied
> 191.75 ± 13% -51.1% 93.75 ± 5% ftrace.balance_dirty_pages.sdc.dirtied_pause
> 58193 ± 19% -81.2% 10953 ± 5% ftrace.balance_dirty_pages.sdc.dirty_ratelimit
> -424.50 ± -2% -98.4% -7.00 ±-286% ftrace.balance_dirty_pages.sdc.pause
> 20.00 ± 17% +448.8% 109.75 ±114% ftrace.balance_dirty_pages.sdc.period
> 76965 ± 21% -84.8% 11683 ± 7% ftrace.balance_dirty_pages.sdc.task_ratelimit
> 443.75 ± 2% -90.4% 42.75 ± 47% ftrace.balance_dirty_pages.sdc.think
> 23458 ± 9% +22.1% 28637 ± 7% ftrace.balance_dirty_pages.sdd.bdi_dirty
> 185.75 ± 14% -49.7% 93.50 ± 5% ftrace.balance_dirty_pages.sdd.dirtied
> 186.00 ± 14% -49.2% 94.50 ± 4% ftrace.balance_dirty_pages.sdd.dirtied_pause
> 61258 ± 14% -81.7% 11213 ± 3% ftrace.balance_dirty_pages.sdd.dirty_ratelimit
> -416.00 ± -4% -100.1% 0.50 ±830% ftrace.balance_dirty_pages.sdd.pause
> 19.25 ± 22% +97.4% 38.00 ± 4% ftrace.balance_dirty_pages.sdd.period
> 86687 ± 20% -87.2% 11105 ± 2% ftrace.balance_dirty_pages.sdd.task_ratelimit
> 433.50 ± 4% -91.5% 36.75 ± 7% ftrace.balance_dirty_pages.sdd.think
> 186.75 ± 13% -53.1% 87.50 ± 0% ftrace.balance_dirty_pages.sde.dirtied
> 188.25 ± 13% -52.6% 89.25 ± 0% ftrace.balance_dirty_pages.sde.dirtied_pause
> 60931 ± 16% -82.8% 10461 ± 1% ftrace.balance_dirty_pages.sde.dirty_ratelimit
> -3387398 ±-173% -100.0% -1.25 ±-1079% ftrace.balance_dirty_pages.sde.pause
> 19.25 ± 14% +94.8% 37.50 ± 1% ftrace.balance_dirty_pages.sde.period
> 80501 ± 23% -86.6% 10750 ± 3% ftrace.balance_dirty_pages.sde.task_ratelimit
> 3387416 ±173% -100.0% 37.00 ± 34% ftrace.balance_dirty_pages.sde.think
> 191.50 ± 8% -52.3% 91.25 ± 3% ftrace.balance_dirty_pages.sdf.dirtied
> 192.50 ± 8% -51.8% 92.75 ± 4% ftrace.balance_dirty_pages.sdf.dirtied_pause
> 59318 ± 10% -81.4% 11026 ± 1% ftrace.balance_dirty_pages.sdf.dirty_ratelimit
> -425.75 ± -1% -96.6% -14.50 ±-84% ftrace.balance_dirty_pages.sdf.pause
> 21.50 ± 14% +77.9% 38.25 ± 3% ftrace.balance_dirty_pages.sdf.period
> 78892 ± 14% -85.6% 11371 ± 2% ftrace.balance_dirty_pages.sdf.task_ratelimit
> 446.00 ± 1% -88.5% 51.50 ± 25% ftrace.balance_dirty_pages.sdf.think
> 29879 ± 6% -10.9% 26610 ± 2% ftrace.balance_dirty_pages.sdg.bdi_dirty
> 181.75 ± 14% -51.3% 88.50 ± 0% ftrace.balance_dirty_pages.sdg.dirtied
> 182.50 ± 14% -50.3% 90.75 ± 1% ftrace.balance_dirty_pages.sdg.dirtied_pause
> 49491 ± 23% -78.7% 10564 ± 2% ftrace.balance_dirty_pages.sdg.dirty_ratelimit
> -372.25 ± -5% -99.3% -2.75 ±-376% ftrace.balance_dirty_pages.sdg.pause
> 27.75 ± 16% +48.6% 41.25 ± 17% ftrace.balance_dirty_pages.sdg.period
> 63160 ± 27% -83.3% 10550 ± 5% ftrace.balance_dirty_pages.sdg.task_ratelimit
> 398.25 ± 3% -90.4% 38.25 ± 24% ftrace.balance_dirty_pages.sdg.think
> 184.00 ± 14% -50.7% 90.75 ± 5% ftrace.balance_dirty_pages.sdh.dirtied
> 185.50 ± 14% -50.0% 92.75 ± 4% ftrace.balance_dirty_pages.sdh.dirtied_pause
> 55615 ± 12% -80.6% 10769 ± 5% ftrace.balance_dirty_pages.sdh.dirty_ratelimit
> -388.00 ±-10% -97.9% -8.00 ±-168% ftrace.balance_dirty_pages.sdh.pause
> 23.50 ± 16% +62.8% 38.25 ± 3% ftrace.balance_dirty_pages.sdh.period
> 70106 ± 20% -84.2% 11100 ± 6% ftrace.balance_dirty_pages.sdh.task_ratelimit
> 410.25 ± 9% -89.0% 45.00 ± 32% ftrace.balance_dirty_pages.sdh.think
> 176.25 ± 14% -47.1% 93.25 ± 2% ftrace.balance_dirty_pages.sdi.dirtied
> 177.25 ± 14% -46.3% 95.25 ± 2% ftrace.balance_dirty_pages.sdi.dirtied_pause
> 45550 ± 19% -75.7% 11062 ± 3% ftrace.balance_dirty_pages.sdi.dirty_ratelimit
> -363.75 ± -8% -95.3% -17.25 ±-85% ftrace.balance_dirty_pages.sdi.pause
> 2.00 ± 50% -87.5% 0.25 ±173% ftrace.balance_dirty_pages.sdi.paused
> 29.25 ± 4% +35.0% 39.50 ± 1% ftrace.balance_dirty_pages.sdi.period
> 54404 ± 24% -80.4% 10681 ± 6% ftrace.balance_dirty_pages.sdi.task_ratelimit
> 391.75 ± 7% -85.8% 55.75 ± 26% ftrace.balance_dirty_pages.sdi.think
> 180.50 ± 11% -48.2% 93.50 ± 4% ftrace.balance_dirty_pages.sdk.dirtied
> 181.75 ± 12% -47.5% 95.50 ± 3% ftrace.balance_dirty_pages.sdk.dirtied_pause
> 53838 ± 14% -79.4% 11111 ± 3% ftrace.balance_dirty_pages.sdk.dirty_ratelimit
> -383.50 ± -8% -97.1% -11.25 ±-84% ftrace.balance_dirty_pages.sdk.pause
> 23.75 ± 22% +65.3% 39.25 ± 3% ftrace.balance_dirty_pages.sdk.period
> 73059 ± 22% -84.8% 11122 ± 6% ftrace.balance_dirty_pages.sdk.task_ratelimit
> 406.50 ± 6% -88.0% 48.75 ± 20% ftrace.balance_dirty_pages.sdk.think
> 179.75 ± 14% -49.5% 90.75 ± 3% ftrace.balance_dirty_pages.sdl.dirtied
> 180.50 ± 14% -48.2% 93.50 ± 3% ftrace.balance_dirty_pages.sdl.dirtied_pause
> 59262 ± 9% -81.7% 10868 ± 4% ftrace.balance_dirty_pages.sdl.dirty_ratelimit
> -401.25 ± -2% -98.0% -8.00 ±-115% ftrace.balance_dirty_pages.sdl.pause
> 21.00 ± 10% +82.1% 38.25 ± 2% ftrace.balance_dirty_pages.sdl.period
> 77465 ± 15% -86.1% 10789 ± 4% ftrace.balance_dirty_pages.sdl.task_ratelimit
> 421.75 ± 1% -89.4% 44.75 ± 21% ftrace.balance_dirty_pages.sdl.think
> 185.75 ± 13% -49.0% 94.75 ± 4% ftrace.balance_dirty_pages.sdm.dirtied
> 186.75 ± 13% -48.7% 95.75 ± 4% ftrace.balance_dirty_pages.sdm.dirtied_pause
> 59214 ± 18% -81.0% 11276 ± 3% ftrace.balance_dirty_pages.sdm.dirty_ratelimit
> -402.75 ± -3% -98.8% -5.00 ±-183% ftrace.balance_dirty_pages.sdm.pause
> 1.50 ± 57% -100.0% 0.00 ± 0% ftrace.balance_dirty_pages.sdm.paused
> 21.25 ± 20% +84.7% 39.25 ± 2% ftrace.balance_dirty_pages.sdm.period
> 80010 ± 25% -85.6% 11485 ± 7% ftrace.balance_dirty_pages.sdm.task_ratelimit
> 422.00 ± 3% -89.9% 42.50 ± 20% ftrace.balance_dirty_pages.sdm.think
> 60203 ± 14% -81.5% 11163 ± 4% ftrace.bdi_dirty_ratelimit.sdb.balanced_dirty_ratelimit
> 58856 ± 13% -80.9% 11269 ± 3% ftrace.bdi_dirty_ratelimit.sdb.dirty_ratelimit
> 80299 ± 18% -87.2% 10313 ± 3% ftrace.bdi_dirty_ratelimit.sdb.task_ratelimit
> 58684 ± 18% -81.5% 10842 ± 3% ftrace.bdi_dirty_ratelimit.sdc.balanced_dirty_ratelimit
> 58173 ± 19% -81.3% 10902 ± 5% ftrace.bdi_dirty_ratelimit.sdc.dirty_ratelimit
> 77036 ± 21% -85.1% 11441 ± 9% ftrace.bdi_dirty_ratelimit.sdc.task_ratelimit
> 63739 ± 14% -82.8% 10955 ± 1% ftrace.bdi_dirty_ratelimit.sdd.balanced_dirty_ratelimit
> 61299 ± 14% -81.8% 11184 ± 3% ftrace.bdi_dirty_ratelimit.sdd.dirty_ratelimit
> 86718 ± 20% -87.4% 10969 ± 1% ftrace.bdi_dirty_ratelimit.sdd.task_ratelimit
> 62431 ± 16% -82.8% 10760 ± 3% ftrace.bdi_dirty_ratelimit.sde.balanced_dirty_ratelimit
> 60940 ± 16% -82.9% 10415 ± 1% ftrace.bdi_dirty_ratelimit.sde.dirty_ratelimit
> 80144 ± 22% -86.9% 10513 ± 4% ftrace.bdi_dirty_ratelimit.sde.task_ratelimit
> 61030 ± 11% -81.6% 11236 ± 1% ftrace.bdi_dirty_ratelimit.sdf.balanced_dirty_ratelimit
> 59341 ± 10% -81.5% 10997 ± 1% ftrace.bdi_dirty_ratelimit.sdf.dirty_ratelimit
> 78573 ± 14% -85.5% 11382 ± 2% ftrace.bdi_dirty_ratelimit.sdf.task_ratelimit
> 49805 ± 24% -78.4% 10782 ± 5% ftrace.bdi_dirty_ratelimit.sdg.balanced_dirty_ratelimit
> 49586 ± 23% -78.8% 10508 ± 2% ftrace.bdi_dirty_ratelimit.sdg.dirty_ratelimit
> 62941 ± 28% -83.5% 10409 ± 6% ftrace.bdi_dirty_ratelimit.sdg.task_ratelimit
> 56232 ± 13% -80.5% 10962 ± 7% ftrace.bdi_dirty_ratelimit.sdh.balanced_dirty_ratelimit
> 55658 ± 12% -80.7% 10737 ± 6% ftrace.bdi_dirty_ratelimit.sdh.dirty_ratelimit
> 69833 ± 20% -84.4% 10906 ± 7% ftrace.bdi_dirty_ratelimit.sdh.task_ratelimit
> 44050 ± 17% -75.7% 10685 ± 4% ftrace.bdi_dirty_ratelimit.sdi.balanced_dirty_ratelimit
> 45546 ± 19% -75.8% 11013 ± 3% ftrace.bdi_dirty_ratelimit.sdi.dirty_ratelimit
> 54325 ± 24% -80.6% 10513 ± 7% ftrace.bdi_dirty_ratelimit.sdi.task_ratelimit
> 54470 ± 16% -80.2% 10809 ± 2% ftrace.bdi_dirty_ratelimit.sdk.balanced_dirty_ratelimit
> 53799 ± 14% -79.4% 11069 ± 4% ftrace.bdi_dirty_ratelimit.sdk.dirty_ratelimit
> 72874 ± 22% -85.2% 10787 ± 6% ftrace.bdi_dirty_ratelimit.sdk.task_ratelimit
> 60161 ± 10% -82.0% 10837 ± 2% ftrace.bdi_dirty_ratelimit.sdl.balanced_dirty_ratelimit
> 59266 ± 9% -81.7% 10819 ± 5% ftrace.bdi_dirty_ratelimit.sdl.dirty_ratelimit
> 77010 ± 16% -86.3% 10563 ± 2% ftrace.bdi_dirty_ratelimit.sdl.task_ratelimit
> 60632 ± 18% -81.0% 11544 ± 3% ftrace.bdi_dirty_ratelimit.sdm.balanced_dirty_ratelimit
> 59157 ± 18% -81.0% 11246 ± 3% ftrace.bdi_dirty_ratelimit.sdm.dirty_ratelimit
> 79666 ± 25% -86.0% 11179 ± 6% ftrace.bdi_dirty_ratelimit.sdm.task_ratelimit
> 263022 ± 7% -10.6% 235031 ± 4% ftrace.writeback_single_inode.sde.index
> 2635 ± 5% -12.1% 2317 ± 2% ftrace.writeback_single_inode.sde.wrote
> 2810 ± 8% -12.2% 2466 ± 4% ftrace.writeback_single_inode.sdf.wrote
> 1.00 ± 0% -100.0% 0.00 ± 0% ftrace.writeback_single_inode.sdg.age
> 3173 ± 10% -25.6% 2360 ± 4% ftrace.writeback_single_inode.sdg.wrote
> 1.00 ± 0% -100.0% 0.00 ± 0% ftrace.writeback_single_inode.sdh.age
> 3521 ± 14% -26.7% 2581 ± 7% ftrace.writeback_single_inode.sdi.wrote
>
>
> lkp-st02: Core2
> Memory: 8G
>
>
> softirqs.SCHED
>
> 200000 ++-----------------------------------------------------------------+
> 180000 ++ O |
> | O O |
> 160000 ++ O O |
> 140000 O+ O O O O O O O O |
> | O O O O O |
> 120000 ++ O O O O O O O
> 100000 ++ |
> 80000 ++ |
> | |
> 60000 ++ .*.*.. .*. .*.. .*. |
> 40000 ++ *.*..*.*. *. *..*.*. *.*..*.*. * |
> | + |
> 20000 ++ |
> 0 *+-O---------------------------------------------------------------+
>
>
> perf-stat.cpu-migrations
>
> 400000 ++-------------------O---------------------------------------------+
> | O |
> 350000 ++ O O O |
> O O O O O O |
> 300000 ++ O O O O O O O O |
> 250000 ++ O O O O O O O
> | |
> 200000 ++ |
> | |
> 150000 ++ |
> 100000 ++ .*.*.. .*. .*.. |
> | *.*..*.*. *. *..*.*. *.*..*.*..*.* |
> 50000 +++ |
> |+ |
> 0 *+-O---------------------------------------------------------------+
>
>
> vmstat.procs.r
>
> 100 ++--------------------------------------------------------------------+
> 90 ++ *. .*..*. .*.. |
> | : *..*..*. .*.. .*. *. *.*..*..*.* |
> 80 ++ : *..*. * |
> 70 ++: |
> | : |
> 60 ++: |
> 50 ++: |
> 40 ++ |
> |: O O O O O O
> 30 O+ O O O O O O O O O O O O O O O O O |
> 20 ++ O O |
> | |
> 10 ++ |
> 0 *+-O------------------------------------------------------------------+
>
>
> vmstat.procs.b
>
> 100 ++--------------------------------------------------------------------+
> 90 ++ O O O O |
> O O O O O O O O O O O O |
> 80 ++ O O O O O O O O O O
> 70 ++ |
> | |
> 60 ++ |
> 50 ++ |
> 40 ++ |
> | .*.. * |
> 30 ++ .*. .. + .*.. |
> 20 ++ .*..*..* * *..*.. .*.. .*.*. *.* |
> | * * *. |
> 10 ++. |
> 0 *+-O------------------------------------------------------------------+
>
>
> proc-vmstat.pgactivate
>
> 900000 ++----------------O--O------O--O----------------------O--O---------+
> O O O O O O O O O O O O O O O O O O O O
> 800000 ++ |
> 700000 ++ |
> | |
> 600000 ++ |
> 500000 ++ |
> | .*.*.. .*. |
> 400000 ++ .*.. .*. *. *.. .*..*.*..*.* |
> 300000 ++ * * *.*..*..* |
> | : |
> 200000 ++: |
> 100000 ++ |
> |: |
> 0 *+-O---------------------------------------------------------------+
>
>
> ftrace.balance_dirty_pages.sdc.pause
>
> 50 ++--------------------------------------------------------------------+
> 0 O+ O O O O O O O O O O O O O O O O O
> | O O O O O O O O O |
> -50 ++ |
> -100 ++ |
> |: |
> -150 ++ |
> -200 ++: |
> -250 ++: |
> | : |
> -300 ++: *. .* |
> -350 ++ : * * .. *. + |
> | : .. + + + .* + |
> -400 ++ *.*.. .*.* + + *. *. .*..*. |
> -450 ++------*----------*--------------------*-------*---------------------+
>
>
> ftrace.balance_dirty_pages.sdc.think
>
> 500 ++--------------------------------------------------------------------+
> 450 ++ .*..*. *.. .*.. .* |
> | *.*. *.. + .*.. * *..* |
> 400 ++ : + * *.. + |
> 350 ++ : * .*.. + |
> | : * * |
> 300 ++: |
> 250 ++: |
> 200 ++: |
> |: |
> 150 ++ |
> 100 ++ |
> |: O O O |
> 50 O+ O O O O O O O O O O O O O O O O O O O O O
> 0 *+-O--------------------------O---------------------------------------+
>
>
> ftrace.balance_dirty_pages.sdg.pause
>
> 50 ++--------------------------------------------------------------------+
> 0 O+ O O O O O O O O O O O O O O O O O O
> | O O O O O O O O |
> -50 ++ |
> -100 ++ |
> |: |
> -150 ++ |
> -200 ++: |
> -250 ++: |
> | : * * |
> -300 ++: .* : : + + |
> -350 ++ : *. + : : + *..*.. .*.. * |
> | : + + : : .* *.*. + |
> -400 ++ *.*..*..* * *. * |
> -450 ++--------------------------------------------------------------------+
>
>
> ftrace.balance_dirty_pages.sdg.think
>
> 450 ++------*--*-------*----*---------------------------------------------+
> | *.*. : + : : .*. .*. |
> 400 ++ : : + : : * .*. *..*. * |
> 350 ++ : *..* : : + *. |
> | : * + + |
> 300 ++: * |
> 250 ++: |
> | : |
> 200 ++ |
> 150 ++ |
> |: |
> 100 ++ |
> 50 ++ O O O O |
> O O O O O O O O O O O O O O O O O O O O O O
> 0 *+-O------------------------------------------------------------------+
>
>
> ftrace.balance_dirty_pages.sdl.pause
>
> 50 ++--------------------------------------------------------------------+
> 0 O+ O O O O O O O O O O O O O O
> | O O O O O O O O O O O O |
> -50 ++ |
> -100 ++ |
> |: |
> -150 ++ |
> -200 ++: |
> -250 ++: * |
> | : : : |
> -300 ++: : : * .*. |
> -350 ++ : : : .. : *. *..*.. |
> | *. : * : .. .*.. |
> -400 ++ *.. .*.* * *.*. *.* |
> -450 ++------*-------------------------------------------------------------+
>
>
> ftrace.balance_dirty_pages.sdl.think
>
> 500 ++--------------------------------------------------------------------+
> 450 ++ .*.. |
> | .*. *.* *.. *.. *.*.. .*.* |
> 400 ++ * : : + .. *. |
> 350 ++ : : : * *.. .*..* |
> | : : : * |
> 300 ++: * |
> 250 ++: |
> 200 ++: |
> |: |
> 150 ++ |
> 100 ++ |
> |: O O O |
> 50 O+ O O O O O O O O O O O O O O O O O O O O O
> 0 *+-O---------------O--------------------------------------------------+
>
> [*] bisect-good sample
> [O] bisect-bad sample
>
> To reproduce:
>
> apt-get install ruby
> git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
> cd lkp-tests
> bin/setup-local job.yaml # the job file attached in this email
> bin/run-local job.yaml
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Ying Huang
> -------------------------------------
> lkp@xxxxxxxxxxxxxxxxx
> https://eclists.intel.com/sympa/info/lkp
> Unsubscribe by sending email to sympa@xxxxxxxxxxxxxxxxx with subject "Unsubscribe lkp"
>
> ---
> LKP_SERVER: inn
> LKP_CGI_PORT: 80
> LKP_CIFS_PORT: 139
> testcase: dd-write
> default-monitors:
> wait: pre-test
> uptime:
> iostat:
> vmstat:
> numa-numastat:
> numa-vmstat:
> numa-meminfo:
> proc-vmstat:
> proc-stat:
> interval: 10
> meminfo:
> slabinfo:
> interrupts:
> lock_stat:
> latency_stats:
> softirqs:
> bdi_dev_mapping:
> diskstats:
> nfsstat:
> cpuidle:
> cpufreq-stats:
> turbostat:
> pmeter:
> sched_debug:
> interval: 60
> default-watchdogs:
> watch-oom:
> watchdog:
> cpufreq_governor:
> commit: a3aca33b3af0f6961719c68a5b1b561206eaa4b8
> model: Core2
> memory: 8G
> nr_hdd_partitions: 12
> wait_disks_timeout: 300
> hdd_partitions: "/dev/disk/by-id/scsi-35000c5000???????"
> swap_partitions:
> runtime: 5m
> disk: 11HDD
> md: JBOD
> iosched: cfq
> fs: ext4
> fs2:
> monitors:
> perf-stat:
> perf-profile:
> ftrace:
> events: balance_dirty_pages bdi_dirty_ratelimit global_dirty_state writeback_single_inode
> nr_threads: 10dd
> dd:
> testbox: lkp-st02
> tbox_group: lkp-st02
> kconfig: x86_64-rhel
> enqueue_time: 2015-06-15 11:39:27.015808126 +08:00
> user: lkp
> queue: cyclic
> compiler: gcc-4.9
> head_commit: a3aca33b3af0f6961719c68a5b1b561206eaa4b8
> base_commit: 0f57d86787d8b1076ea8f9cbdddda2a46d534a27
> branch: linux-devel/devel-hourly-2015061609
> kernel: "/pkg/linux/x86_64-rhel/gcc-4.9/a3aca33b3af0f6961719c68a5b1b561206eaa4b8/vmlinuz-4.1.0-rc8-wl-04178-ga3aca33"
> rootfs: debian-x86_64-2015-02-07.cgz
> result_root: "/result/dd-write/300-5m-11HDD-JBOD-cfq-ext4-10dd/lkp-st02/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/a3aca33b3af0f6961719c68a5b1b561206eaa4b8/0"
> job_file: "/lkp/scheduled/lkp-st02/cyclic_dd-write-300-5m-11HDD-JBOD-cfq-ext4-10dd-x86_64-rhel-CYCLIC_HEAD-a3aca33b3af0f6961719c68a5b1b561206eaa4b8-0-20150615-96200-tjp09g.yaml"
> dequeue_time: 2015-06-16 09:49:33.945435115 +08:00
> nr_cpu: "$(nproc)"
> max_uptime: 1500
> initrd: "/osimage/debian/debian-x86_64-2015-02-07.cgz"
> bootloader_append:
> - root=/dev/ram0
> - user=lkp
> - job=/lkp/scheduled/lkp-st02/cyclic_dd-write-300-5m-11HDD-JBOD-cfq-ext4-10dd-x86_64-rhel-CYCLIC_HEAD-a3aca33b3af0f6961719c68a5b1b561206eaa4b8-0-20150615-96200-tjp09g.yaml
> - ARCH=x86_64
> - kconfig=x86_64-rhel
> - branch=linux-devel/devel-hourly-2015061609
> - commit=a3aca33b3af0f6961719c68a5b1b561206eaa4b8
> - BOOT_IMAGE=/pkg/linux/x86_64-rhel/gcc-4.9/a3aca33b3af0f6961719c68a5b1b561206eaa4b8/vmlinuz-4.1.0-rc8-wl-04178-ga3aca33
> - max_uptime=1500
> - RESULT_ROOT=/result/dd-write/300-5m-11HDD-JBOD-cfq-ext4-10dd/lkp-st02/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/a3aca33b3af0f6961719c68a5b1b561206eaa4b8/0
> - LKP_SERVER=inn
> - |2-
>
>
> earlyprintk=ttyS0,115200 systemd.log_level=err
> debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100
> panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
> console=ttyS0,115200 console=tty0 vga=normal
>
> rw
> lkp_initrd: "/lkp/lkp/lkp-x86_64.cgz"
> modules_initrd: "/pkg/linux/x86_64-rhel/gcc-4.9/a3aca33b3af0f6961719c68a5b1b561206eaa4b8/modules.cgz"
> bm_initrd: "/osimage/deps/debian-x86_64-2015-02-07.cgz/lkp.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/run-ipconfig.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/turbostat.cgz,/lkp/benchmarks/turbostat.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/fs.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/fs2.cgz"
> job_state: finished
> loadavg: 65.04 66.52 31.46 1/169 5575
> start_time: '1434419421'
> end_time: '1434419742'
> version: "/lkp/lkp/.src-20150616-071222"
> time_delta: '1434419386.307633460'
> mkfs -t ext4 -q -F /dev/sdh
> mkfs -t ext4 -q -F /dev/sdi
> mount -t ext4 /dev/sdb /fs/sdb
> mount -t ext4 /dev/sdg /fs/sdg
> mount -t ext4 /dev/sdi /fs/sdi
> mount -t ext4 /dev/sdh /fs/sdh
> mount -t ext4 /dev/sdl /fs/sdl
> mount -t ext4 /dev/sdf /fs/sdf
> mount -t ext4 /dev/sdm /fs/sdm
> mount -t ext4 /dev/sdk /fs/sdk
> mount -t ext4 /dev/sdd /fs/sdd
> mount -t ext4 /dev/sde /fs/sde
> mount -t ext4 /dev/sdc /fs/sdc
> echo 1 > /sys/kernel/debug/tracing/events/writeback/balance_dirty_pages/enable
> echo 1 > /sys/kernel/debug/tracing/events/writeback/bdi_dirty_ratelimit/enable
> echo 1 > /sys/kernel/debug/tracing/events/writeback/global_dirty_state/enable
> echo 1 > /sys/kernel/debug/tracing/events/writeback/writeback_single_inode/enable
> dd if=/dev/zero of=/fs/sdb/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-1 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-2 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-3 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-4 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-5 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-6 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-7 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-8 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-9 status=noxfer &
> dd if=/dev/zero of=/fs/sdb/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdg/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdi/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdh/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdl/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdf/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdm/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdk/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdd/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sde/zero-10 status=noxfer &
> dd if=/dev/zero of=/fs/sdc/zero-10 status=noxfer &
> sleep 289
> killall -9 dd
--
Jan Kara <jack@xxxxxxx>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/