Re: [linus:master] [iomap] c5c810b94c: stress-ng.metamix.ops_per_sec -98.4% regression
From: Brian Foster
Date: Mon Oct 14 2024 - 12:33:46 EST
On Mon, Oct 14, 2024 at 03:55:24PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a -98.4% regression of stress-ng.metamix.ops_per_sec on:
>
>
> commit: c5c810b94cfd818fc2f58c96feee58a9e5ead96d ("iomap: fix handling of dirty folios over unwritten extents")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> testcase: stress-ng
> config: x86_64-rhel-8.3
> compiler: gcc-12
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
>
> nr_threads: 100%
> disk: 1HDD
> testtime: 60s
> fs: xfs
> test: metamix
> cpufreq_governor: performance
>
>
>
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
> | Closes: https://lore.kernel.org/oe-lkp/202410141536.1167190b-oliver.sang@xxxxxxxxx
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20241014/202410141536.1167190b-oliver.sang@xxxxxxxxx
>
So I basically just run this on a >64xcpu guest and reproduce the delta:
stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --metamix 64
The short of it is that with tracing enabled, I see a very large number
of extending writes across unwritten mappings, which basically means XFS
eof zeroing is calling zero range and hitting the newly introduced
flush. This is all pretty much expected given the patch.
I ran a quick experiment to skip the flush on sub-4k ranges in favor of
doing explicit folio zeroing. The idea with that is that the range is
likely restricted to single folio and since it's dirty, we can assume
unwritten conversion is imminent and just explicitly zero the range. I
still see a decent number of flushes from larger ranges in that
experiment, but that still seems to get things pretty close to my
baseline test (on a 6.10 distro kernel).
That said, the next thing on my todo list in this area is to try out the
revalidation based iteration scheme that was discussed on earlier review
passes of the flush fix, with the idea that would elide the flush
entirely on filesystems that support revalidation (i.e. XFS). Given that
the current approach was intended to be a simple and correct fallback
for fs' without revalidation support, I'm a little reluctant to
introduce a heuristic just to appease this benchmark on XFS.
So... my current thinking is that is I'm going to prioritize hacking on
the reval scheme over optimizing the fallback solution and see how that
goes. If that doesn't pan out for whatever reason and/or if some other
more practical workload comes up where this is a practical problem, then
I suspect it will be easy enough to work around with a heuristic to
balance flushes vs. explicit folio zeroing.
For example, I still have another prototype lying around that let
iomap_truncate_page() default to zeroing rather than flushing since the
operation is by design restricted to a partial range of the EOF folio.
With the caveat that I haven't fully dug into this stress-ng test to
know exactly what it's doing, I suspect something like that could help
elide flushes entirely from this workload with some fairly small tweaks
in the fs.
Brian
> =========================================================================================
> compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
> gcc-12/performance/1HDD/xfs/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/metamix/stress-ng/60s
>
> commit:
> 6f634eb080 ("filemap: fix htmldoc warning for mapping_align_index()")
> c5c810b94c ("iomap: fix handling of dirty folios over unwritten extents")
>
> 6f634eb080161baa c5c810b94cfd818fc2f58c96fee
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 5.762e+09 ± 6% +1784.6% 1.086e+11 ± 3% cpuidle..time
> 537216 ± 8% +846.9% 5087100 ± 3% cpuidle..usage
> 5.83 ± 13% -72.0% 1.63 ± 3% iostat.cpu.idle
> 93.48 +5.1% 98.28 iostat.cpu.iowait
> 134.32 ± 4% +1197.6% 1742 ± 3% uptime.boot
> 2789 ± 3% +54.0% 4295 ± 4% uptime.idle
> 47.17 ±144% -81.6% 8.67 ± 24% perf-c2c.DRAM.local
> 89.00 ± 77% -58.4% 37.00 ± 9% perf-c2c.HITM.local
> 42.33 ± 86% -63.0% 15.67 ± 26% perf-c2c.HITM.remote
> 609419 ± 10% +144.5% 1489941 ± 18% numa-numastat.node0.local_node
> 628123 ± 10% +142.2% 1521154 ± 17% numa-numastat.node0.numa_hit
> 537937 ± 4% +288.1% 2087519 ± 10% numa-numastat.node1.local_node
> 585710 ± 4% +262.4% 2122854 ± 10% numa-numastat.node1.numa_hit
> 33057 ± 5% -94.2% 1926 ± 2% vmstat.io.bo
> 1.72 ± 6% -37.5% 1.08 vmstat.procs.r
> 5454 ± 6% -45.2% 2990 vmstat.system.cs
> 5999 ± 4% -53.5% 2790 vmstat.system.in
> 3.79 ± 19% -2.3 1.52 ± 3% mpstat.cpu.all.idle%
> 0.02 ± 23% -0.0 0.01 ± 3% mpstat.cpu.all.irq%
> 0.01 ± 4% -0.0 0.00 mpstat.cpu.all.soft%
> 0.25 ± 6% -0.2 0.05 ± 4% mpstat.cpu.all.sys%
> 0.40 ± 5% -0.4 0.03 ± 7% mpstat.cpu.all.usr%
> 6.44 ± 5% -13.8% 5.55 ± 3% mpstat.max_utilization_pct
> 1991 ± 14% -68.8% 621.17 ± 9% stress-ng.metamix.ops
> 23.12 ± 10% -98.4% 0.37 ± 7% stress-ng.metamix.ops_per_sec
> 87.77 ± 6% +1831.7% 1695 ± 3% stress-ng.time.elapsed_time
> 87.77 ± 6% +1831.7% 1695 ± 3% stress-ng.time.elapsed_time.max
> 120134 ± 2% -5.1% 114001 stress-ng.time.minor_page_faults
> 5.67 ± 8% -82.4% 1.00 stress-ng.time.percent_of_cpu_this_job_got
> 4.90 ± 7% +331.2% 21.13 ± 4% stress-ng.time.system_time
> 63630 ± 9% +1332.9% 911761 ± 3% stress-ng.time.voluntary_context_switches
> 25272 ± 6% +100.3% 50631 meminfo.Active
> 20787 ± 10% +96.7% 40898 ± 2% meminfo.Active(anon)
> 4485 ± 15% +117.0% 9733 meminfo.Active(file)
> 207516 ± 6% +216.6% 656984 ± 10% meminfo.AnonHugePages
> 377749 ± 4% +31.5% 496804 meminfo.Dirty
> 1808866 -20.6% 1436964 meminfo.Inactive
> 902066 -43.7% 508085 meminfo.Inactive(file)
> 6425133 -10.6% 5746563 meminfo.Memused
> 11.47 +17676.6% 2038 meminfo.Mlocked
> 102534 +22.7% 125834 meminfo.Shmem
> 119495 -95.0% 5925 ± 3% meminfo.Writeback
> 7239789 -14.4% 6193951 meminfo.max_used_kB
> 2230 ± 16% +72.8% 3853 ± 5% numa-meminfo.node0.Active(file)
> 84262 ± 27% +532.6% 533075 ± 25% numa-meminfo.node0.AnonHugePages
> 401080 ± 65% +81.0% 726026 ± 10% numa-meminfo.node0.AnonPages.max
> 450689 ± 3% -59.7% 181817 ± 4% numa-meminfo.node0.Inactive(file)
> 15857 ± 4% -12.5% 13880 ± 4% numa-meminfo.node0.KernelStack
> 5.73 ±100% +14883.5% 858.83 ± 96% numa-meminfo.node0.Mlocked
> 59744 ± 3% -96.6% 2047 ± 4% numa-meminfo.node0.Writeback
> 16267 ± 11% +121.2% 35987 ± 16% numa-meminfo.node1.Active
> 14010 ± 13% +114.9% 30108 ± 20% numa-meminfo.node1.Active(anon)
> 2257 ± 15% +160.4% 5879 ± 4% numa-meminfo.node1.Active(file)
> 188486 ± 7% +69.0% 318533 ± 3% numa-meminfo.node1.Dirty
> 956395 ± 30% -43.4% 541330 ± 20% numa-meminfo.node1.Inactive
> 452002 ± 5% -27.8% 326319 ± 3% numa-meminfo.node1.Inactive(file)
> 150109 ± 13% +16.7% 175200 ± 5% numa-meminfo.node1.Slab
> 59749 ± 4% -93.5% 3891 ± 5% numa-meminfo.node1.Writeback
> 556.82 ± 16% +73.0% 963.40 ± 5% numa-vmstat.node0.nr_active_file
> 41.21 ± 27% +531.7% 260.29 ± 25% numa-vmstat.node0.nr_anon_transparent_hugepages
> 379768 ± 11% -23.8% 289236 ± 3% numa-vmstat.node0.nr_dirtied
> 112900 ± 2% -59.7% 45451 ± 4% numa-vmstat.node0.nr_inactive_file
> 15877 ± 4% -12.6% 13883 ± 3% numa-vmstat.node0.nr_kernel_stack
> 1.44 ±100% +14824.7% 214.67 ± 96% numa-vmstat.node0.nr_mlock
> 14977 ± 3% -96.6% 512.83 ± 4% numa-vmstat.node0.nr_writeback
> 379768 ± 11% -25.6% 282645 ± 3% numa-vmstat.node0.nr_written
> 556.84 ± 16% +73.0% 963.40 ± 5% numa-vmstat.node0.nr_zone_active_file
> 112900 ± 2% -59.7% 45451 ± 4% numa-vmstat.node0.nr_zone_inactive_file
> 62482 ± 3% -27.8% 45088 ± 4% numa-vmstat.node0.nr_zone_write_pending
> 625814 ± 10% +143.0% 1520756 ± 17% numa-vmstat.node0.numa_hit
> 607109 ± 10% +145.4% 1489543 ± 18% numa-vmstat.node0.numa_local
> 3496 ± 13% +115.3% 7527 ± 20% numa-vmstat.node1.nr_active_anon
> 563.82 ± 16% +160.7% 1469 ± 4% numa-vmstat.node1.nr_active_file
> 380179 ± 9% +38.2% 525240 ± 5% numa-vmstat.node1.nr_dirtied
> 47231 ± 7% +68.6% 79622 ± 3% numa-vmstat.node1.nr_dirty
> 113239 ± 5% -28.0% 81571 ± 3% numa-vmstat.node1.nr_inactive_file
> 14977 ± 4% -93.5% 974.17 ± 5% numa-vmstat.node1.nr_writeback
> 380179 ± 9% +35.0% 513207 ± 5% numa-vmstat.node1.nr_written
> 3496 ± 13% +115.3% 7527 ± 20% numa-vmstat.node1.nr_zone_active_anon
> 563.82 ± 16% +160.7% 1469 ± 4% numa-vmstat.node1.nr_zone_active_file
> 113239 ± 5% -28.0% 81571 ± 3% numa-vmstat.node1.nr_zone_inactive_file
> 62209 ± 6% +29.6% 80597 ± 3% numa-vmstat.node1.nr_zone_write_pending
> 583795 ± 4% +263.5% 2121826 ± 10% numa-vmstat.node1.numa_hit
> 535988 ± 4% +289.3% 2086491 ± 10% numa-vmstat.node1.numa_local
> 5190 ± 10% +97.0% 10224 ± 2% proc-vmstat.nr_active_anon
> 1122 ± 14% +116.7% 2433 proc-vmstat.nr_active_file
> 208668 +2.2% 213362 proc-vmstat.nr_anon_pages
> 101.32 ± 6% +216.6% 320.79 ± 10% proc-vmstat.nr_anon_transparent_hugepages
> 94628 ± 4% +31.3% 124208 proc-vmstat.nr_dirty
> 1051112 -8.7% 959304 proc-vmstat.nr_file_pages
> 226820 +2.4% 232205 proc-vmstat.nr_inactive_anon
> 225925 -43.8% 127025 proc-vmstat.nr_inactive_file
> 29400 -6.6% 27458 proc-vmstat.nr_kernel_stack
> 2.88 +17612.4% 509.76 proc-vmstat.nr_mlock
> 22780 -6.0% 21412 ± 3% proc-vmstat.nr_page_table_pages
> 25696 ± 2% +22.4% 31461 proc-vmstat.nr_shmem
> 26966 +2.2% 27573 proc-vmstat.nr_slab_reclaimable
> 63926 +2.0% 65209 proc-vmstat.nr_slab_unreclaimable
> 29903 -95.0% 1484 ± 3% proc-vmstat.nr_writeback
> 5190 ± 10% +97.0% 10224 ± 2% proc-vmstat.nr_zone_active_anon
> 1122 ± 14% +116.7% 2433 proc-vmstat.nr_zone_active_file
> 226820 +2.4% 232205 proc-vmstat.nr_zone_inactive_anon
> 225925 -43.8% 127025 proc-vmstat.nr_zone_inactive_file
> 1215783 ± 6% +199.9% 3646420 ± 3% proc-vmstat.numa_hit
> 1149305 ± 7% +211.5% 3579877 ± 3% proc-vmstat.numa_local
> 89633 ± 7% +349.7% 403114 ± 3% proc-vmstat.pgactivate
> 1312602 ± 6% +200.5% 3944776 ± 3% proc-vmstat.pgalloc_normal
> 415149 ± 3% +897.4% 4140633 ± 3% proc-vmstat.pgfault
> 1306213 ± 6% +200.3% 3923139 ± 3% proc-vmstat.pgfree
> 16256 ± 3% +1057.0% 188090 ± 3% proc-vmstat.pgreuse
> 1.82 ± 4% -9.6% 1.65 perf-stat.i.MPKI
> 3.76e+08 ± 6% -89.1% 40978627 ± 4% perf-stat.i.branch-instructions
> 2.30 ± 3% -0.6 1.66 perf-stat.i.branch-miss-rate%
> 17129191 ± 7% -93.0% 1196180 ± 7% perf-stat.i.branch-misses
> 16.84 ± 3% -6.5 10.36 perf-stat.i.cache-miss-rate%
> 1341312 ± 4% -81.2% 251609 perf-stat.i.cache-misses
> 7592299 ± 3% -73.3% 2030865 perf-stat.i.cache-references
> 5488 ± 6% -45.6% 2986 perf-stat.i.context-switches
> 1.35 ± 4% +8.7% 1.47 perf-stat.i.cpi
> 1.867e+09 ± 5% -86.0% 2.607e+08 ± 6% perf-stat.i.cpu-cycles
> 199.44 ± 2% -59.4% 80.94 perf-stat.i.cpu-migrations
> 1655 ± 13% -41.4% 969.67 perf-stat.i.cycles-between-cache-misses
> 1.841e+09 ± 6% -89.2% 1.994e+08 ± 4% perf-stat.i.instructions
> 0.87 -18.5% 0.71 perf-stat.i.ipc
> 0.57 ± 40% -97.6% 0.01 ± 43% perf-stat.i.major-faults
> 0.01 ±141% +30993.6% 2.48 ± 5% perf-stat.i.metric.K/sec
> 3665 ± 3% -36.0% 2345 perf-stat.i.minor-faults
> 3666 ± 3% -36.0% 2345 perf-stat.i.page-faults
> 0.73 ± 5% +72.7% 1.26 ± 4% perf-stat.overall.MPKI
> 4.55 -1.6 2.92 ± 3% perf-stat.overall.branch-miss-rate%
> 17.66 ± 2% -5.3 12.39 perf-stat.overall.cache-miss-rate%
> 1.02 ± 5% +28.1% 1.31 ± 3% perf-stat.overall.cpi
> 1397 ± 5% -25.8% 1036 ± 5% perf-stat.overall.cycles-between-cache-misses
> 0.98 ± 5% -22.1% 0.77 ± 3% perf-stat.overall.ipc
> 3.728e+08 ± 6% -89.0% 41014226 ± 4% perf-stat.ps.branch-instructions
> 16957001 ± 7% -92.9% 1198002 ± 7% perf-stat.ps.branch-misses
> 1332233 ± 4% -81.1% 251650 perf-stat.ps.cache-misses
> 7543069 ± 3% -73.1% 2030809 perf-stat.ps.cache-references
> 5443 ± 6% -45.2% 2985 perf-stat.ps.context-switches
> 63285 +1.1% 63962 perf-stat.ps.cpu-clock
> 1.859e+09 ± 5% -86.0% 2.611e+08 ± 6% perf-stat.ps.cpu-cycles
> 198.47 ± 3% -59.2% 80.91 perf-stat.ps.cpu-migrations
> 1.826e+09 ± 6% -89.1% 1.996e+08 ± 4% perf-stat.ps.instructions
> 0.58 ± 41% -97.6% 0.01 ± 43% perf-stat.ps.major-faults
> 3640 ± 3% -35.6% 2344 perf-stat.ps.minor-faults
> 3640 ± 3% -35.6% 2344 perf-stat.ps.page-faults
> 63285 +1.1% 63962 perf-stat.ps.task-clock
> 1.637e+11 ± 5% +106.8% 3.387e+11 ± 4% perf-stat.total.instructions
> 0.01 ± 31% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mempool_alloc_noprof.bio_alloc_bioset.iomap_writepage_map_blocks.iomap_writepage_map
> 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra
> 0.00 ± 16% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 0.00 ± 17% -100.0% 0.00 perf-sched.sch_delay.avg.ms.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64
> 0.00 ± 12% -100.0% 0.00 perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 0.01 ± 8% -100.0% 0.00 perf-sched.sch_delay.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
> 0.02 ±135% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mempool_alloc_noprof.bio_alloc_bioset.iomap_writepage_map_blocks.iomap_writepage_map
> 0.14 ± 79% -88.5% 0.02 ±103% perf-sched.sch_delay.max.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
> 0.02 ± 77% -67.9% 0.01 ± 5% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
> 0.02 ± 38% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra
> 0.01 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 0.00 ± 26% -100.0% 0.00 perf-sched.sch_delay.max.ms.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 0.01 ±103% -100.0% 0.00 perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64
> 0.00 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 0.01 ± 11% -100.0% 0.00 perf-sched.sch_delay.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
> 8119 ± 9% -50.9% 3990 ± 15% perf-sched.total_wait_and_delay.count.ms
> 765.01 ± 48% -100.0% 0.01 perf-sched.wait_and_delay.avg.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
> 8.46 ± 6% +27.6% 10.79 ± 16% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 50.66 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 134.86 ± 13% +158.8% 348.98 ± 18% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
> 65.33 ± 70% +353.1% 296.00 ± 64% perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
> 1366 ± 18% -78.6% 293.00 ± 64% perf-sched.wait_and_delay.count.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
> 97.17 -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.xfsaild.kthread.ret_from_fork
> 3007 ± 18% -77.6% 674.17 ± 29% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
> 2858 ± 50% -100.0% 0.04 ± 72% perf-sched.wait_and_delay.max.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
> 103.17 ± 10% +73.5% 179.00 ± 24% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 59.66 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 765.00 ± 48% -100.0% 0.00 perf-sched.wait_time.avg.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
> 93.17 ± 32% -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra
> 8.45 ± 6% +27.6% 10.78 ± 16% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 50.65 -100.0% 0.00 perf-sched.wait_time.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 134.64 ± 13% +158.9% 348.54 ± 18% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
> 0.01 ± 22% -100.0% 0.00 perf-sched.wait_time.avg.ms.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 202.39 ± 62% -100.0% 0.00 perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64
> 248.16 ± 50% -100.0% 0.00 perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 219.73 ± 46% -100.0% 0.00 perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
> 2858 ± 50% -100.0% 0.00 perf-sched.wait_time.max.ms.io_schedule.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range
> 468.55 ± 29% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra
> 103.16 ± 10% +73.5% 179.00 ± 24% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 59.65 ± 10% -100.0% 0.00 perf-sched.wait_time.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork
> 0.02 ± 25% -100.0% 0.00 perf-sched.wait_time.max.ms.xlog_force_lsn.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 348.84 ± 58% -100.0% 0.00 perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64
> 486.25 ± 23% -100.0% 0.00 perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xfs_log_force_seq.xfs_file_fsync.__x64_sys_fdatasync
> 473.19 ± 38% -100.0% 0.00 perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread
> 7346 ± 4% +23.0% 9033 ± 7% sched_debug.cfs_rq:/.avg_vruntime.avg
> 848.55 ± 8% +123.3% 1894 ± 11% sched_debug.cfs_rq:/.avg_vruntime.min
> 0.14 ± 14% -71.2% 0.04 ± 2% sched_debug.cfs_rq:/.h_nr_running.avg
> 0.32 ± 9% -41.3% 0.19 ± 2% sched_debug.cfs_rq:/.h_nr_running.stddev
> 139317 ±191% -88.9% 15437 ± 4% sched_debug.cfs_rq:/.load.avg
> 8499863 ±201% -92.8% 613636 ± 4% sched_debug.cfs_rq:/.load.max
> 247.33 ± 32% -85.6% 35.54 ± 46% sched_debug.cfs_rq:/.load_avg.avg
> 632.63 ± 71% -74.1% 163.67 ± 68% sched_debug.cfs_rq:/.load_avg.stddev
> 7346 ± 4% +23.0% 9033 ± 7% sched_debug.cfs_rq:/.min_vruntime.avg
> 848.55 ± 8% +123.3% 1894 ± 11% sched_debug.cfs_rq:/.min_vruntime.min
> 0.14 ± 14% -71.2% 0.04 ± 2% sched_debug.cfs_rq:/.nr_running.avg
> 0.32 ± 9% -41.3% 0.19 ± 2% sched_debug.cfs_rq:/.nr_running.stddev
> 62.04 ± 22% -89.5% 6.51 ± 42% sched_debug.cfs_rq:/.removed.load_avg.avg
> 597.33 ± 31% -63.0% 220.77 ± 53% sched_debug.cfs_rq:/.removed.load_avg.max
> 171.33 ± 9% -80.1% 34.06 ± 47% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 18.96 ± 25% -85.8% 2.70 ± 45% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> 308.25 ± 31% -64.0% 111.09 ± 54% sched_debug.cfs_rq:/.removed.runnable_avg.max
> 60.15 ± 17% -74.0% 15.67 ± 51% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
> 18.96 ± 25% -85.8% 2.70 ± 45% sched_debug.cfs_rq:/.removed.util_avg.avg
> 308.25 ± 31% -64.0% 111.09 ± 54% sched_debug.cfs_rq:/.removed.util_avg.max
> 60.15 ± 17% -74.0% 15.67 ± 51% sched_debug.cfs_rq:/.removed.util_avg.stddev
> 237.66 ± 5% -85.8% 33.72 ± 3% sched_debug.cfs_rq:/.runnable_avg.avg
> 1104 ± 12% -45.6% 600.94 ± 2% sched_debug.cfs_rq:/.runnable_avg.max
> 226.39 ± 5% -57.8% 95.63 ± 2% sched_debug.cfs_rq:/.runnable_avg.stddev
> 236.68 ± 5% -85.9% 33.42 ± 4% sched_debug.cfs_rq:/.util_avg.avg
> 1103 ± 12% -46.3% 591.99 sched_debug.cfs_rq:/.util_avg.max
> 226.37 ± 5% -58.2% 94.62 ± 2% sched_debug.cfs_rq:/.util_avg.stddev
> 18.18 ± 24% -83.7% 2.96 ± 17% sched_debug.cfs_rq:/.util_est.avg
> 338.67 ± 13% -67.8% 109.03 ± 15% sched_debug.cfs_rq:/.util_est.max
> 67.93 ± 17% -76.8% 15.79 ± 14% sched_debug.cfs_rq:/.util_est.stddev
> 803668 ± 2% +18.9% 955672 sched_debug.cpu.avg_idle.avg
> 198446 ± 5% -41.6% 115938 ± 11% sched_debug.cpu.avg_idle.stddev
> 74790 +1064.4% 870887 ± 3% sched_debug.cpu.clock.avg
> 74795 +1064.4% 870893 ± 3% sched_debug.cpu.clock.max
> 74786 +1064.5% 870879 ± 3% sched_debug.cpu.clock.min
> 2.48 ± 4% +26.3% 3.14 ± 8% sched_debug.cpu.clock.stddev
> 74525 +1068.1% 870538 ± 3% sched_debug.cpu.clock_task.avg
> 74776 +1064.6% 870850 ± 3% sched_debug.cpu.clock_task.max
> 66679 +1193.0% 862189 ± 3% sched_debug.cpu.clock_task.min
> 339.14 ± 15% +27.9% 433.69 ± 4% sched_debug.cpu.curr->pid.avg
> 4069 +498.1% 24338 ± 3% sched_debug.cpu.curr->pid.max
> 895.15 ± 7% +243.7% 3076 ± 3% sched_debug.cpu.curr->pid.stddev
> 0.14 ± 15% -74.6% 0.04 ± 8% sched_debug.cpu.nr_running.avg
> 0.31 ± 11% -44.7% 0.17 ± 6% sched_debug.cpu.nr_running.stddev
> 4679 ± 4% +794.9% 41879 ± 4% sched_debug.cpu.nr_switches.avg
> 27295 ± 11% +571.5% 183299 ± 4% sched_debug.cpu.nr_switches.max
> 1238 ± 20% +985.0% 13436 ± 12% sched_debug.cpu.nr_switches.min
> 4550 ± 12% +581.7% 31022 ± 7% sched_debug.cpu.nr_switches.stddev
> 8.07 +60.9% 12.99 ± 2% sched_debug.cpu.nr_uninterruptible.avg
> 35.50 ± 10% +49.5% 53.06 ± 14% sched_debug.cpu.nr_uninterruptible.max
> 10.63 ± 7% +33.5% 14.19 ± 12% sched_debug.cpu.nr_uninterruptible.stddev
> 74787 +1064.5% 870884 ± 3% sched_debug.cpu_clk
> 73624 +1081.3% 869721 ± 3% sched_debug.ktime
> 75383 +1056.2% 871587 ± 3% sched_debug.sched_clk
> 12.22 ± 39% -12.2 0.00 perf-profile.calltrace.cycles-pp.fdatasync.stress_metamix
> 12.15 ± 39% -12.2 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fdatasync.stress_metamix
> 12.15 ± 39% -12.2 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fdatasync.stress_metamix
> 12.02 ± 40% -12.0 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_fdatasync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fdatasync.stress_metamix
> 12.02 ± 40% -12.0 0.00 perf-profile.calltrace.cycles-pp.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fdatasync
> 9.67 ± 50% -9.2 0.44 ±112% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write.ksys_write.do_syscall_64
> 8.34 ± 50% -7.9 0.41 ±108% perf-profile.calltrace.cycles-pp.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write.ksys_write
> 7.12 ± 30% -7.1 0.00 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.81 ± 32% -6.8 0.00 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.__x64_sys_fdatasync.do_syscall_64
> 6.81 ± 32% -6.8 0.00 perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.__x64_sys_fdatasync
> 6.80 ± 32% -6.8 0.00 perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
> 6.80 ± 32% -6.8 0.00 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range
> 6.44 ± 49% -6.3 0.12 ±223% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write
> 5.25 ± 45% -5.2 0.00 perf-profile.calltrace.cycles-pp.iomap_writepage_map.iomap_writepages.xfs_vm_writepages.do_writepages.filemap_fdatawrite_wbc
> 4.70 ± 47% -4.7 0.00 perf-profile.calltrace.cycles-pp.read.stress_metamix
> 4.69 ± 46% -4.7 0.00 perf-profile.calltrace.cycles-pp.iomap_writepage_map_blocks.iomap_writepage_map.iomap_writepages.xfs_vm_writepages.do_writepages
> 4.68 ± 63% -4.7 0.00 perf-profile.calltrace.cycles-pp.unlink.stress_metamix
> 4.66 ± 63% -4.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink.stress_metamix
> 4.66 ± 63% -4.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink.stress_metamix
> 4.64 ± 63% -4.6 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink.stress_metamix
> 4.64 ± 63% -4.6 0.00 perf-profile.calltrace.cycles-pp.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
> 4.12 ± 48% -4.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read.stress_metamix
> 4.08 ± 48% -4.1 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.stress_metamix
> 4.01 ± 64% -4.0 0.00 perf-profile.calltrace.cycles-pp.evict.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 4.00 ± 64% -4.0 0.00 perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink.do_syscall_64
> 3.96 ± 48% -4.0 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read.stress_metamix
> 3.88 ± 52% -3.9 0.00 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.iomap_writepage_map_blocks.iomap_writepage_map.iomap_writepages.xfs_vm_writepages
> 3.87 ± 53% -3.9 0.00 perf-profile.calltrace.cycles-pp.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepage_map_blocks.iomap_writepage_map.iomap_writepages
> 3.86 ± 53% -3.9 0.00 perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepage_map_blocks.iomap_writepage_map
> 6.96 ± 11% -3.3 3.62 ± 29% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
> 6.29 ± 14% -3.1 3.16 ± 28% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
> 0.65 ± 64% +0.5 1.12 ± 25% perf-profile.calltrace.cycles-pp._nohz_idle_balance.handle_softirqs.__irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single
> 0.22 ±141% +0.6 0.85 ± 28% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 0.11 ±223% +0.6 0.74 ± 28% perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.22 ±141% +0.7 0.88 ± 27% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
> 0.22 ±141% +0.7 0.88 ± 27% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
> 0.50 ± 76% +0.7 1.15 ± 25% perf-profile.calltrace.cycles-pp.__sysvec_posted_msi_notification.sysvec_posted_msi_notification.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_enter
> 0.22 ±142% +0.7 0.94 ± 25% perf-profile.calltrace.cycles-pp.__open64_nocancel.setlocale
> 0.27 ±141% +0.8 1.07 ± 28% perf-profile.calltrace.cycles-pp.write
> 0.33 ±102% +0.8 1.14 ± 22% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
> 0.22 ±145% +0.8 1.05 ± 24% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
> 0.43 ±108% +0.9 1.35 ± 24% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
> 0.16 ±223% +1.0 1.17 ± 36% perf-profile.calltrace.cycles-pp.blk_mq_submit_bio.__submit_bio.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepages
> 0.15 ±223% +1.1 1.20 ± 38% perf-profile.calltrace.cycles-pp.blk_mq_dispatch_rq_list.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue
> 0.66 ± 98% +1.1 1.76 ± 33% perf-profile.calltrace.cycles-pp.cmd_stat.run_builtin.handle_internal_command.main
> 0.66 ± 98% +1.1 1.76 ± 33% perf-profile.calltrace.cycles-pp.dispatch_events.cmd_stat.run_builtin.handle_internal_command.main
> 0.66 ± 98% +1.1 1.76 ± 33% perf-profile.calltrace.cycles-pp.process_interval.dispatch_events.cmd_stat.run_builtin.handle_internal_command
> 0.54 ±119% +1.1 1.68 ± 34% perf-profile.calltrace.cycles-pp.exit_mmap.mmput.exit_mm.do_exit.do_group_exit
> 0.54 ±118% +1.2 1.70 ± 33% perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
> 0.54 ±118% +1.2 1.70 ± 33% perf-profile.calltrace.cycles-pp.mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group
> 1.56 ± 19% +1.2 2.73 ± 22% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
> 0.00 +1.2 1.18 ± 40% perf-profile.calltrace.cycles-pp.scsi_queue_rq.blk_mq_dispatch_rq_list.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests
> 0.45 ±148% +1.2 1.66 ± 33% perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat.run_builtin
> 0.72 ± 98% +1.3 1.99 ± 32% perf-profile.calltrace.cycles-pp.handle_internal_command.main
> 0.72 ± 98% +1.3 1.99 ± 32% perf-profile.calltrace.cycles-pp.main
> 0.72 ± 98% +1.3 1.99 ± 32% perf-profile.calltrace.cycles-pp.run_builtin.handle_internal_command.main
> 0.48 ±151% +1.3 1.76 ± 32% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 1.80 ± 20% +1.3 3.10 ± 23% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
> 1.82 ± 20% +1.3 3.12 ± 23% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
> 0.64 ±114% +1.4 2.00 ± 32% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.64 ±114% +1.4 2.00 ± 32% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64
> 0.64 ±114% +1.4 2.00 ± 32% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.52 ±109% +1.4 1.88 ± 27% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.53 ±109% +1.4 1.92 ± 28% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.93 ± 83% +1.4 2.32 ± 32% perf-profile.calltrace.cycles-pp.x64_sys_call.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.91 ± 89% +1.5 2.41 ± 42% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 0.71 ±110% +1.6 2.32 ± 42% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
> 0.66 ±112% +1.7 2.35 ± 30% perf-profile.calltrace.cycles-pp.setlocale
> 2.08 ± 26% +1.7 3.82 ± 29% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
> 1.05 ± 78% +1.9 2.94 ± 28% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
> 1.46 ± 49% +1.9 3.35 ± 19% perf-profile.calltrace.cycles-pp.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
> 1.10 ± 78% +1.9 3.02 ± 28% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
> 1.10 ± 78% +1.9 3.02 ± 28% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
> 1.30 ± 69% +2.0 3.28 ± 32% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
> 1.32 ± 68% +2.0 3.30 ± 31% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
> 1.32 ± 68% +2.0 3.30 ± 31% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
> 1.43 ± 70% +2.0 3.45 ± 30% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 1.11 ± 78% +2.1 3.24 ± 27% perf-profile.calltrace.cycles-pp.read
> 2.01 ± 71% +3.0 4.99 ± 22% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
> 2.01 ± 71% +3.0 5.00 ± 22% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
> 2.01 ± 71% +3.0 5.00 ± 22% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
> 2.01 ± 71% +3.0 5.00 ± 22% perf-profile.calltrace.cycles-pp.execve
> 2.00 ± 70% +3.0 4.99 ± 22% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
> 1.01 ± 54% +3.3 4.27 ± 31% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests
> 1.01 ± 54% +3.4 4.36 ± 30% perf-profile.calltrace.cycles-pp._raw_spin_lock.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests
> 1.05 ± 55% +3.4 4.47 ± 31% perf-profile.calltrace.cycles-pp.dd_dispatch_request.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue
> 2.33 ± 39% +4.0 6.29 ± 28% perf-profile.calltrace.cycles-pp.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_dispatch_plug_list.blk_mq_flush_plug_list
> 2.33 ± 39% +4.0 6.31 ± 28% perf-profile.calltrace.cycles-pp.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_dispatch_plug_list.blk_mq_flush_plug_list.__blk_flush_plug
> 2.35 ± 39% +4.0 6.35 ± 27% perf-profile.calltrace.cycles-pp.blk_mq_run_hw_queue.blk_mq_dispatch_plug_list.blk_mq_flush_plug_list.__blk_flush_plug.__submit_bio
> 1.71 ± 50% +4.5 6.23 ± 29% perf-profile.calltrace.cycles-pp.__blk_mq_do_dispatch_sched.__blk_mq_sched_dispatch_requests.blk_mq_sched_dispatch_requests.blk_mq_run_hw_queue.blk_mq_dispatch_plug_list
> 3.24 ± 66% +4.8 8.01 ± 27% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 3.24 ± 66% +4.8 8.04 ± 27% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 3.55 ± 40% +5.2 8.77 ± 29% perf-profile.calltrace.cycles-pp.blk_mq_dispatch_plug_list.blk_mq_flush_plug_list.__blk_flush_plug.__submit_bio.submit_bio_noacct_nocheck
> 3.55 ± 40% +5.2 8.77 ± 29% perf-profile.calltrace.cycles-pp.blk_mq_flush_plug_list.__blk_flush_plug.__submit_bio.submit_bio_noacct_nocheck.iomap_submit_ioend
> 3.98 ± 55% +8.1 12.11 ± 30% perf-profile.calltrace.cycles-pp.xfs_file_write_checks.xfs_file_buffered_write.vfs_write.ksys_write.do_syscall_64
> 0.56 ±164% +8.2 8.77 ± 29% perf-profile.calltrace.cycles-pp.__blk_flush_plug.__submit_bio.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepages
> 3.33 ± 56% +8.3 11.60 ± 31% perf-profile.calltrace.cycles-pp.iomap_zero_range.xfs_file_write_checks.xfs_file_buffered_write.vfs_write.ksys_write
> 0.91 ±108% +9.0 9.94 ± 29% perf-profile.calltrace.cycles-pp.__submit_bio.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepages.xfs_vm_writepages
> 0.91 ±108% +9.0 9.94 ± 29% perf-profile.calltrace.cycles-pp.submit_bio_noacct_nocheck.iomap_submit_ioend.iomap_writepages.xfs_vm_writepages.do_writepages
> 0.92 ±107% +9.0 9.95 ± 29% perf-profile.calltrace.cycles-pp.iomap_submit_ioend.iomap_writepages.xfs_vm_writepages.do_writepages.filemap_fdatawrite_wbc
> 0.00 +10.3 10.29 ± 30% perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.filemap_write_and_wait_range.iomap_zero_range
> 0.00 +10.3 10.29 ± 30% perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.filemap_write_and_wait_range
> 0.00 +10.4 10.40 ± 30% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.filemap_write_and_wait_range.iomap_zero_range.xfs_file_write_checks.xfs_file_buffered_write
> 0.00 +10.4 10.40 ± 30% perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.filemap_write_and_wait_range.iomap_zero_range.xfs_file_write_checks
> 0.00 +11.2 11.22 ± 30% perf-profile.calltrace.cycles-pp.filemap_write_and_wait_range.iomap_zero_range.xfs_file_write_checks.xfs_file_buffered_write.vfs_write
> 12.24 ± 39% -12.2 0.00 perf-profile.children.cycles-pp.fdatasync
> 12.02 ± 40% -12.0 0.00 perf-profile.children.cycles-pp.__x64_sys_fdatasync
> 12.02 ± 40% -12.0 0.00 perf-profile.children.cycles-pp.xfs_file_fsync
> 9.68 ± 50% -9.1 0.57 ± 69% perf-profile.children.cycles-pp.iomap_file_buffered_write
> 8.37 ± 50% -7.8 0.53 ± 66% perf-profile.children.cycles-pp.iomap_write_iter
> 7.12 ± 30% -7.1 0.00 perf-profile.children.cycles-pp.file_write_and_wait_range
> 6.48 ± 49% -6.1 0.35 ± 65% perf-profile.children.cycles-pp.iomap_write_begin
> 5.25 ± 45% -5.0 0.22 ± 74% perf-profile.children.cycles-pp.iomap_writepage_map
> 4.74 ± 60% -4.7 0.00 perf-profile.children.cycles-pp.unlink
> 4.69 ± 60% -4.7 0.00 perf-profile.children.cycles-pp.__x64_sys_unlink
> 4.69 ± 60% -4.7 0.00 perf-profile.children.cycles-pp.do_unlinkat
> 4.70 ± 46% -4.6 0.14 ± 97% perf-profile.children.cycles-pp.iomap_writepage_map_blocks
> 4.31 ± 55% -4.2 0.16 ±108% perf-profile.children.cycles-pp.iomap_iter
> 4.13 ± 61% -4.1 0.00 perf-profile.children.cycles-pp.truncate_inode_pages_range
> 4.06 ± 61% -4.1 0.01 ±223% perf-profile.children.cycles-pp.evict
> 3.90 ± 48% -3.8 0.11 ± 82% perf-profile.children.cycles-pp.__iomap_write_begin
> 3.82 ± 56% -3.7 0.08 ± 80% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
> 3.42 ± 47% -3.4 0.07 ±111% perf-profile.children.cycles-pp.zero_user_segments
> 6.97 ± 11% -3.4 3.62 ± 29% perf-profile.children.cycles-pp.worker_thread
> 3.45 ± 46% -3.2 0.24 ± 35% perf-profile.children.cycles-pp.memset_orig
> 3.25 ± 45% -3.1 0.11 ± 88% perf-profile.children.cycles-pp.filemap_read
> 6.29 ± 14% -3.1 3.16 ± 28% perf-profile.children.cycles-pp.process_one_work
> 3.18 ± 54% -2.6 0.59 ± 50% perf-profile.children.cycles-pp.folios_put_refs
> 2.46 ± 67% -2.4 0.10 ± 65% perf-profile.children.cycles-pp.__page_cache_release
> 2.51 ± 50% -2.2 0.28 ± 52% perf-profile.children.cycles-pp.__filemap_get_folio
> 1.58 ± 51% -1.5 0.10 ± 84% perf-profile.children.cycles-pp.filemap_add_folio
> 1.52 ± 45% -1.4 0.10 ±119% perf-profile.children.cycles-pp.copy_page_to_iter
> 1.48 ± 44% -1.4 0.12 ± 75% perf-profile.children.cycles-pp._copy_to_iter
> 2.55 ± 16% -1.1 1.41 ± 36% perf-profile.children.cycles-pp.pick_next_task_fair
> 1.22 ± 24% -1.1 0.10 ± 71% perf-profile.children.cycles-pp.mod_delayed_work_on
> 1.16 ± 50% -1.0 0.13 ± 78% perf-profile.children.cycles-pp.open64
> 0.95 ± 31% -0.9 0.06 ± 84% perf-profile.children.cycles-pp.try_to_grab_pending
> 0.99 ± 45% -0.9 0.10 ± 71% perf-profile.children.cycles-pp.kblockd_mod_delayed_work_on
> 0.88 ± 50% -0.8 0.06 ±111% perf-profile.children.cycles-pp.filemap_get_pages
> 0.81 ± 48% -0.7 0.06 ±113% perf-profile.children.cycles-pp.filemap_get_read_batch
> 1.08 ± 28% -0.6 0.45 ± 39% perf-profile.children.cycles-pp.clear_bhb_loop
> 0.70 ± 51% -0.6 0.08 ±125% perf-profile.children.cycles-pp.folio_alloc_noprof
> 0.66 ± 53% -0.6 0.06 ± 84% perf-profile.children.cycles-pp.__filemap_add_folio
> 0.59 ± 32% -0.6 0.04 ±100% perf-profile.children.cycles-pp.xfs_map_blocks
> 0.88 ± 14% -0.5 0.38 ± 51% perf-profile.children.cycles-pp.__lruvec_stat_mod_folio
> 0.58 ± 38% -0.5 0.13 ± 81% perf-profile.children.cycles-pp.writeback_iter
> 0.71 ± 29% -0.4 0.26 ± 51% perf-profile.children.cycles-pp.entry_SYSCALL_64
> 0.52 ± 43% -0.4 0.12 ± 91% perf-profile.children.cycles-pp.writeback_get_folio
> 0.44 ± 53% -0.4 0.06 ±130% perf-profile.children.cycles-pp.__folio_start_writeback
> 0.49 ± 20% -0.4 0.12 ± 62% perf-profile.children.cycles-pp.percpu_counter_add_batch
> 0.35 ± 38% -0.3 0.08 ± 54% perf-profile.children.cycles-pp.touch_atime
> 0.63 ± 23% -0.3 0.37 ± 66% perf-profile.children.cycles-pp.get_page_from_freelist
> 0.47 ± 33% -0.3 0.21 ± 79% perf-profile.children.cycles-pp.xas_load
> 0.30 ± 35% -0.2 0.09 ± 86% perf-profile.children.cycles-pp.rmqueue
> 0.24 ± 29% -0.2 0.05 ± 71% perf-profile.children.cycles-pp.__mod_node_page_state
> 0.13 ± 22% -0.1 0.04 ±101% perf-profile.children.cycles-pp.lock_timer_base
> 0.16 ± 31% -0.1 0.08 ± 85% perf-profile.children.cycles-pp.xas_find
> 0.16 ± 22% -0.1 0.08 ± 78% perf-profile.children.cycles-pp.xfs_trans_reserve
> 0.05 ± 78% +0.1 0.12 ± 37% perf-profile.children.cycles-pp.hrtimer_try_to_cancel
> 0.00 +0.1 0.08 ± 10% perf-profile.children.cycles-pp.change_protection_range
> 0.13 ± 33% +0.1 0.21 ± 38% perf-profile.children.cycles-pp.wakeup_preempt
> 0.05 ±108% +0.1 0.14 ± 24% perf-profile.children.cycles-pp.perf_event_read
> 0.05 ±108% +0.1 0.14 ± 24% perf-profile.children.cycles-pp.smp_call_function_single
> 0.04 ±118% +0.1 0.14 ± 37% perf-profile.children.cycles-pp.run_ksoftirqd
> 0.02 ±143% +0.1 0.12 ± 60% perf-profile.children.cycles-pp.__poll
> 0.02 ±143% +0.1 0.12 ± 60% perf-profile.children.cycles-pp.__x64_sys_poll
> 0.02 ±143% +0.1 0.12 ± 60% perf-profile.children.cycles-pp.do_sys_poll
> 0.01 ±223% +0.1 0.11 ± 48% perf-profile.children.cycles-pp.lockref_put_return
> 0.02 ±142% +0.1 0.12 ± 60% perf-profile.children.cycles-pp.do_poll
> 0.07 ± 75% +0.1 0.18 ± 34% perf-profile.children.cycles-pp.switch_fpu_return
> 0.04 ±112% +0.1 0.15 ± 66% perf-profile.children.cycles-pp.getenv
> 0.01 ±223% +0.1 0.12 ± 39% perf-profile.children.cycles-pp.folio_putback_lru
> 0.04 ±114% +0.1 0.16 ± 37% perf-profile.children.cycles-pp.generic_exec_single
> 0.02 ±223% +0.1 0.14 ± 38% perf-profile.children.cycles-pp.expand_downwards
> 0.12 ± 44% +0.1 0.24 ± 15% perf-profile.children.cycles-pp.__check_object_size
> 0.14 ± 29% +0.1 0.27 ± 19% perf-profile.children.cycles-pp.vma_alloc_folio_noprof
> 0.01 ±223% +0.1 0.14 ± 37% perf-profile.children.cycles-pp.xfsaild
> 0.01 ±223% +0.1 0.14 ± 37% perf-profile.children.cycles-pp.xfsaild_push
> 0.01 ±223% +0.1 0.14 ± 39% perf-profile.children.cycles-pp.get_cpu_sleep_time_us
> 0.06 ± 83% +0.1 0.19 ± 62% perf-profile.children.cycles-pp.__d_add
> 0.10 ± 92% +0.1 0.24 ± 15% perf-profile.children.cycles-pp.shift_arg_pages
> 0.00 +0.1 0.14 ± 37% perf-profile.children.cycles-pp.get_idle_time
> 0.13 ± 37% +0.1 0.27 ± 19% perf-profile.children.cycles-pp.folio_alloc_mpol_noprof
> 0.11 ± 80% +0.1 0.26 ± 39% perf-profile.children.cycles-pp.mm_init
> 0.07 ± 49% +0.1 0.22 ± 35% perf-profile.children.cycles-pp.pte_alloc_one
> 0.22 ± 47% +0.2 0.37 ± 33% perf-profile.children.cycles-pp.scsi_mq_get_budget
> 0.06 ±103% +0.2 0.21 ± 53% perf-profile.children.cycles-pp.unlink_anon_vmas
> 0.14 ± 79% +0.2 0.30 ± 30% perf-profile.children.cycles-pp.mas_wr_node_store
> 0.04 ±107% +0.2 0.20 ± 70% perf-profile.children.cycles-pp.dyntick_save_progress_counter
> 0.05 ±132% +0.2 0.22 ± 66% perf-profile.children.cycles-pp.sysfs_kf_seq_show
> 0.05 ±136% +0.2 0.22 ± 66% perf-profile.children.cycles-pp.dev_attr_show
> 0.07 ± 89% +0.2 0.24 ± 28% perf-profile.children.cycles-pp.__cmd_record
> 0.07 ± 89% +0.2 0.24 ± 28% perf-profile.children.cycles-pp.cmd_record
> 0.05 ±125% +0.2 0.22 ± 43% perf-profile.children.cycles-pp.move_queued_task
> 0.08 ± 14% +0.2 0.26 ± 51% perf-profile.children.cycles-pp.flush_smp_call_function_queue
> 0.07 ±127% +0.2 0.25 ± 31% perf-profile.children.cycles-pp.__dentry_kill
> 0.04 ± 77% +0.2 0.22 ± 62% perf-profile.children.cycles-pp.mas_split
> 0.08 ± 27% +0.2 0.27 ± 47% perf-profile.children.cycles-pp.mas_alloc_nodes
> 0.14 ± 62% +0.2 0.34 ± 29% perf-profile.children.cycles-pp.create_elf_tables
> 0.01 ±223% +0.2 0.21 ± 60% perf-profile.children.cycles-pp.__put_user_8
> 0.12 ± 64% +0.2 0.32 ± 48% perf-profile.children.cycles-pp.seq_printf
> 0.21 ± 36% +0.2 0.43 ± 37% perf-profile.children.cycles-pp.vfs_statx
> 0.23 ± 73% +0.2 0.45 ± 38% perf-profile.children.cycles-pp.sbitmap_get
> 0.15 ±104% +0.2 0.38 ± 52% perf-profile.children.cycles-pp.get_arg_page
> 0.24 ± 70% +0.2 0.47 ± 42% perf-profile.children.cycles-pp.load_elf_interp
> 0.08 ± 52% +0.2 0.31 ± 44% perf-profile.children.cycles-pp.__get_user_8
> 0.13 ± 48% +0.3 0.38 ± 51% perf-profile.children.cycles-pp.rcu_gp_fqs_loop
> 0.21 ± 66% +0.3 0.46 ± 46% perf-profile.children.cycles-pp.vsnprintf
> 0.12 ± 61% +0.3 0.38 ± 30% perf-profile.children.cycles-pp.slab_show
> 0.15 ± 65% +0.3 0.40 ± 50% perf-profile.children.cycles-pp.rep_stos_alternative
> 0.23 ± 32% +0.3 0.49 ± 32% perf-profile.children.cycles-pp.path_lookupat
> 0.13 ± 82% +0.3 0.40 ± 50% perf-profile.children.cycles-pp.perf_evsel__read
> 0.13 ± 62% +0.3 0.40 ± 24% perf-profile.children.cycles-pp.step_into
> 0.14 ± 70% +0.3 0.42 ± 35% perf-profile.children.cycles-pp.alloc_anon_folio
> 0.23 ± 32% +0.3 0.51 ± 28% perf-profile.children.cycles-pp.filename_lookup
> 0.06 ± 88% +0.3 0.34 ± 48% perf-profile.children.cycles-pp.rseq_ip_fixup
> 0.18 ± 39% +0.3 0.47 ± 38% perf-profile.children.cycles-pp.dput
> 0.16 ± 43% +0.3 0.45 ± 37% perf-profile.children.cycles-pp.rcu_gp_kthread
> 0.28 ± 69% +0.3 0.57 ± 32% perf-profile.children.cycles-pp.__vfork
> 0.24 ± 68% +0.3 0.54 ± 29% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
> 0.09 ± 65% +0.3 0.42 ± 38% perf-profile.children.cycles-pp.__rseq_handle_notify_resume
> 0.36 ± 40% +0.3 0.68 ± 27% perf-profile.children.cycles-pp.__do_sys_newfstatat
> 0.23 ± 66% +0.3 0.58 ± 56% perf-profile.children.cycles-pp.free_pgtables
> 0.32 ± 42% +0.4 0.67 ± 30% perf-profile.children.cycles-pp.tick_irq_enter
> 0.43 ± 39% +0.4 0.78 ± 19% perf-profile.children.cycles-pp.irq_enter_rcu
> 0.21 ± 55% +0.4 0.57 ± 41% perf-profile.children.cycles-pp.readn
> 0.16 ± 33% +0.4 0.52 ± 50% perf-profile.children.cycles-pp.dup_mmap
> 0.15 ± 77% +0.4 0.51 ± 39% perf-profile.children.cycles-pp.show_stat
> 0.34 ± 69% +0.4 0.71 ± 42% perf-profile.children.cycles-pp.zap_present_ptes
> 0.21 ± 67% +0.4 0.60 ± 23% perf-profile.children.cycles-pp.i2c_outb
> 0.23 ± 79% +0.4 0.62 ± 36% perf-profile.children.cycles-pp.seq_read
> 0.18 ± 40% +0.4 0.59 ± 46% perf-profile.children.cycles-pp.dup_mm
> 0.30 ± 67% +0.4 0.72 ± 36% perf-profile.children.cycles-pp.tmigr_handle_remote_up
> 0.20 ± 65% +0.4 0.63 ± 58% perf-profile.children.cycles-pp.sync_regs
> 0.25 ± 66% +0.4 0.68 ± 23% perf-profile.children.cycles-pp.try_address
> 0.27 ± 74% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.output_poll_execute
> 0.28 ± 70% +0.5 0.74 ± 28% perf-profile.children.cycles-pp.pipe_read
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.__i2c_transfer
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.bit_xfer
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.drm_connector_helper_detect_from_ddc
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.drm_do_probe_ddc_edid
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.drm_helper_probe_detect_ctx
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.drm_probe_ddc
> 0.26 ± 70% +0.5 0.72 ± 24% perf-profile.children.cycles-pp.i2c_transfer
> 0.14 ± 61% +0.5 0.61 ± 35% perf-profile.children.cycles-pp.balance_fair
> 0.23 ± 68% +0.5 0.69 ± 34% perf-profile.children.cycles-pp.folio_wait_bit_common
> 0.28 ± 81% +0.5 0.77 ± 29% perf-profile.children.cycles-pp.pipe_write
> 0.35 ± 68% +0.5 0.84 ± 41% perf-profile.children.cycles-pp.tmigr_handle_remote
> 0.24 ± 81% +0.5 0.74 ± 34% perf-profile.children.cycles-pp.copy_strings
> 0.27 ± 72% +0.5 0.78 ± 50% perf-profile.children.cycles-pp.exec_mmap
> 0.19 ± 83% +0.5 0.70 ± 34% perf-profile.children.cycles-pp.folio_wait_writeback
> 0.30 ± 67% +0.5 0.83 ± 31% perf-profile.children.cycles-pp.do_anonymous_page
> 0.44 ± 59% +0.6 1.00 ± 46% perf-profile.children.cycles-pp.zap_pmd_range
> 0.35 ± 86% +0.6 0.93 ± 31% perf-profile.children.cycles-pp.collapse_huge_page
> 0.36 ± 84% +0.6 0.93 ± 30% perf-profile.children.cycles-pp.khugepaged
> 0.35 ± 85% +0.6 0.93 ± 30% perf-profile.children.cycles-pp.hpage_collapse_scan_pmd
> 0.35 ± 85% +0.6 0.93 ± 30% perf-profile.children.cycles-pp.khugepaged_scan_mm_slot
> 0.34 ± 63% +0.6 0.92 ± 37% perf-profile.children.cycles-pp.evlist_cpu_iterator__next
> 0.24 ± 42% +0.6 0.89 ± 47% perf-profile.children.cycles-pp.scsi_dispatch_cmd
> 0.32 ± 62% +0.7 0.99 ± 45% perf-profile.children.cycles-pp.begin_new_exec
> 0.41 ± 58% +0.7 1.10 ± 21% perf-profile.children.cycles-pp.__open64_nocancel
> 0.37 ± 48% +0.7 1.06 ± 34% perf-profile.children.cycles-pp._Fork
> 0.54 ± 66% +0.7 1.27 ± 29% perf-profile.children.cycles-pp.sched_setaffinity
> 0.78 ± 51% +0.8 1.56 ± 28% perf-profile.children.cycles-pp.link_path_walk
> 0.40 ± 35% +0.8 1.19 ± 40% perf-profile.children.cycles-pp.scsi_queue_rq
> 0.51 ± 74% +0.8 1.30 ± 39% perf-profile.children.cycles-pp.elf_load
> 0.34 ± 54% +0.8 1.15 ± 34% perf-profile.children.cycles-pp.smpboot_thread_fn
> 0.72 ± 69% +0.9 1.66 ± 33% perf-profile.children.cycles-pp.read_counters
> 0.77 ± 70% +1.0 1.76 ± 33% perf-profile.children.cycles-pp.cmd_stat
> 0.77 ± 70% +1.0 1.76 ± 33% perf-profile.children.cycles-pp.dispatch_events
> 0.77 ± 70% +1.0 1.76 ± 33% perf-profile.children.cycles-pp.process_interval
> 1.00 ± 19% +1.0 2.01 ± 26% perf-profile.children.cycles-pp.filemap_map_pages
> 1.39 ± 34% +1.1 2.48 ± 22% perf-profile.children.cycles-pp.asm_sysvec_posted_msi_notification
> 0.85 ± 68% +1.1 1.99 ± 32% perf-profile.children.cycles-pp.handle_internal_command
> 0.85 ± 68% +1.1 1.99 ± 32% perf-profile.children.cycles-pp.main
> 0.85 ± 68% +1.1 1.99 ± 32% perf-profile.children.cycles-pp.run_builtin
> 0.88 ± 64% +1.2 2.04 ± 32% perf-profile.children.cycles-pp.do_group_exit
> 0.88 ± 64% +1.2 2.05 ± 32% perf-profile.children.cycles-pp.__x64_sys_exit_group
> 0.64 ± 66% +1.2 1.87 ± 37% perf-profile.children.cycles-pp.seq_read_iter
> 1.24 ± 54% +1.2 2.48 ± 31% perf-profile.children.cycles-pp.x64_sys_call
> 1.03 ± 66% +1.3 2.30 ± 30% perf-profile.children.cycles-pp.do_exit
> 0.88 ± 64% +1.5 2.35 ± 30% perf-profile.children.cycles-pp.setlocale
> 1.08 ± 71% +1.5 2.56 ± 26% perf-profile.children.cycles-pp.mmput
> 1.05 ± 70% +1.5 2.55 ± 27% perf-profile.children.cycles-pp.exit_mmap
> 3.54 ± 34% +1.8 5.33 ± 20% perf-profile.children.cycles-pp.handle_softirqs
> 1.61 ± 55% +1.8 3.40 ± 35% perf-profile.children.cycles-pp.mmap_region
> 1.30 ± 69% +2.0 3.28 ± 32% perf-profile.children.cycles-pp.load_elf_binary
> 1.32 ± 68% +2.0 3.30 ± 31% perf-profile.children.cycles-pp.search_binary_handler
> 1.32 ± 68% +2.0 3.32 ± 31% perf-profile.children.cycles-pp.exec_binprm
> 1.45 ± 70% +2.1 3.52 ± 30% perf-profile.children.cycles-pp.bprm_execve
> 2.01 ± 71% +3.0 5.01 ± 22% perf-profile.children.cycles-pp.__x64_sys_execve
> 2.01 ± 71% +3.0 5.01 ± 23% perf-profile.children.cycles-pp.execve
> 2.00 ± 70% +3.0 5.01 ± 22% perf-profile.children.cycles-pp.do_execveat_common
> 1.18 ± 39% +3.3 4.48 ± 31% perf-profile.children.cycles-pp.dd_dispatch_request
> 2.77 ± 34% +3.6 6.33 ± 28% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
> 2.77 ± 34% +3.6 6.34 ± 28% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
> 2.59 ± 34% +3.8 6.35 ± 27% perf-profile.children.cycles-pp.blk_mq_run_hw_queue
> 1.94 ± 37% +4.3 6.24 ± 29% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
> 3.76 ± 36% +5.0 8.79 ± 30% perf-profile.children.cycles-pp.__blk_flush_plug
> 3.76 ± 36% +5.0 8.79 ± 30% perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
> 3.76 ± 36% +5.0 8.79 ± 30% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
> 4.84 ± 35% +5.1 9.96 ± 29% perf-profile.children.cycles-pp.iomap_submit_ioend
> 4.01 ± 55% +8.1 12.11 ± 30% perf-profile.children.cycles-pp.xfs_file_write_checks
> 3.34 ± 56% +8.3 11.60 ± 31% perf-profile.children.cycles-pp.iomap_zero_range
> 0.00 +11.2 11.22 ± 30% perf-profile.children.cycles-pp.filemap_write_and_wait_range
> 3.43 ± 46% -3.2 0.24 ± 35% perf-profile.self.cycles-pp.memset_orig
> 1.42 ± 47% -1.3 0.08 ±121% perf-profile.self.cycles-pp._copy_to_iter
> 1.07 ± 27% -0.6 0.45 ± 39% perf-profile.self.cycles-pp.clear_bhb_loop
> 0.38 ± 23% -0.3 0.12 ± 62% perf-profile.self.cycles-pp.percpu_counter_add_batch
> 0.30 ± 33% -0.2 0.09 ± 88% perf-profile.self.cycles-pp.do_syscall_64
> 0.22 ± 30% -0.2 0.05 ± 71% perf-profile.self.cycles-pp.__mod_node_page_state
> 0.13 ± 20% -0.1 0.04 ±103% perf-profile.self.cycles-pp.__page_cache_release
> 0.01 ±223% +0.1 0.11 ± 48% perf-profile.self.cycles-pp.lockref_put_return
> 0.02 ±142% +0.1 0.13 ± 65% perf-profile.self.cycles-pp.xfs_ag_block_count
> 0.01 ±223% +0.1 0.13 ± 45% perf-profile.self.cycles-pp.get_cpu_sleep_time_us
> 0.04 ±112% +0.2 0.19 ± 68% perf-profile.self.cycles-pp.dyntick_save_progress_counter
> 0.14 ± 35% +0.2 0.31 ± 35% perf-profile.self.cycles-pp.sched_balance_domains
> 0.08 ± 52% +0.2 0.31 ± 44% perf-profile.self.cycles-pp.__get_user_8
> 0.06 ± 92% +0.3 0.34 ± 60% perf-profile.self.cycles-pp.fold_vm_numa_events
> 0.20 ± 65% +0.4 0.63 ± 58% perf-profile.self.cycles-pp.sync_regs
> 0.26 ± 34% +0.5 0.75 ± 37% perf-profile.self.cycles-pp.filemap_map_pages
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> --
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
>