Re: [linus:master] [NFSD] 39d432fc76: fsmark.files_per_sec -100.0% regression
From: Chuck Lever
Date: Thu Aug 24 2023 - 09:25:32 EST
On Thu, Aug 24, 2023 at 01:59:06PM +0800, kernel test robot wrote:
>
>
> hi, Chuck Lever,
>
> Fengwei (CCed) helped us review this astonishing finding by fsmark tests,
> and doubt below part:
> - nfsd4_end_grace(nn);
> + trace_nfsd_end_grace(netns(file));
>
> and confirmed if adding back:
> nfsd4_end_grace(nn);
>
> the regression is gone and files_per_sec restore to 61.93.
As always, thanks for the report. However, this result is not
plausible. "end_grace" happens only once after a server reboot.
Can you confirm that the NFS server kernel is not crashing
during the test?
> FYI.
>
>
> Hello,
>
> kernel test robot noticed a -100.0% regression of fsmark.files_per_sec on:
>
>
> commit: 39d432fc76301cf0a0c454022117601994ca9397 ("NFSD: trace nfsctl operations")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> testcase: fsmark
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
>
> iterations: 1x
> nr_threads: 32t
> disk: 1SSD
> fs: btrfs
> fs2: nfsv4
> filesize: 16MB
> test_size: 20G
> sync_method: NoSync
> nr_directories: 16d
> nr_files_per_directory: 256fpd
> cpufreq_governor: performance
>
>
> In addition to that, the commit also has significant impact on the following tests:
>
> +------------------+-------------------------------------------------------------------------------------------+
> | testcase: change | fsmark: fsmark.files_per_sec -100.0% regression |
> | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
> | test parameters | cpufreq_governor=performance |
> | | debug-setup=no-monitor |
> | | disk=1SSD |
> | | filesize=16MB |
> | | fs2=nfsv4 |
> | | fs=btrfs |
> | | iterations=1x |
> | | nr_directories=16d |
> | | nr_files_per_directory=256fpd |
> | | nr_threads=32t |
> | | sync_method=NoSync |
> | | test_size=20G |
> +------------------+-------------------------------------------------------------------------------------------+
>
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
> | Closes: https://lore.kernel.org/oe-lkp/202308241229.68396422-oliver.sang@xxxxxxxxx
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20230824/202308241229.68396422-oliver.sang@xxxxxxxxx
>
> =========================================================================================
> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> gcc-12/performance/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark
>
> commit:
> 3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
> 39d432fc76 ("NFSD: trace nfsctl operations")
>
> 3434d7aa77d24c5c 39d432fc76301cf0a0c45402211
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 51.67 ± 25% -90.8% 4.75 ± 45% perf-c2c.DRAM.local
> 35.36 ± 16% +35.2% 47.81 ± 16% boot-time.dhcp
> 1.81 ± 41% +90.9% 3.46 ± 31% boot-time.smp_boot
> 1.455e+09 ± 6% +12542.8% 1.839e+11 ± 20% cpuidle..time
> 3013365 ± 2% +6257.5% 1.916e+08 ± 20% cpuidle..usage
> 81.11 ± 7% +3758.6% 3129 ± 18% uptime.boot
> 4349 ± 8% +4266.2% 189897 ± 19% uptime.idle
> 71.95 +31.7% 94.75 iostat.cpu.idle
> 18.35 ± 3% -84.5% 2.85 ± 56% iostat.cpu.iowait
> 7.98 ± 5% -70.3% 2.37 iostat.cpu.system
> 1.73 ± 3% -98.0% 0.03 ± 15% iostat.cpu.user
> 7176382 ± 6% +37.6% 9873307 ± 6% numa-numastat.node0.local_node
> 7210227 ± 6% +37.5% 9914765 ± 6% numa-numastat.node0.numa_hit
> 6958614 ± 6% +29.1% 8981736 ± 4% numa-numastat.node1.local_node
> 6990967 ± 6% +28.8% 9006618 ± 4% numa-numastat.node1.numa_hit
> 8749533 ± 4% +7050.4% 6.256e+08 ± 18% fsmark.app_overhead
> 61.92 ± 5% -100.0% 0.00 fsmark.files_per_sec
> 22.77 ± 4% +12864.1% 2951 ± 20% fsmark.time.elapsed_time
> 22.77 ± 4% +12864.1% 2951 ± 20% fsmark.time.elapsed_time.max
> 189.33 ± 7% -99.5% 1.00 fsmark.time.percent_of_cpu_this_job_got
> 70.09 +24.7 94.75 mpstat.cpu.all.idle%
> 19.80 ± 4% -16.9 2.86 ± 56% mpstat.cpu.all.iowait%
> 1.18 ± 4% +1.0 2.20 mpstat.cpu.all.irq%
> 0.13 ± 2% -0.1 0.07 ± 3% mpstat.cpu.all.soft%
> 7.10 ± 7% -7.0 0.09 ± 12% mpstat.cpu.all.sys%
> 1.70 ± 5% -1.7 0.03 ± 15% mpstat.cpu.all.usr%
> 71.33 +32.5% 94.50 vmstat.cpu.id
> 18.00 ± 3% -87.5% 2.25 ± 79% vmstat.cpu.wa
> 784537 ± 5% -99.0% 7461 ± 21% vmstat.io.bo
> 10.17 ± 23% -85.2% 1.50 ± 74% vmstat.procs.b
> 6.00 ± 25% -100.0% 0.00 vmstat.procs.r
> 101767 ± 7% -91.5% 8688 ± 8% vmstat.system.cs
> 70688 -9.1% 64225 vmstat.system.in
> 43996 ± 5% +121.0% 97246 ± 9% meminfo.Active
> 4516 ± 14% +1588.6% 76266 ± 6% meminfo.Active(anon)
> 39479 ± 4% -46.9% 20978 ± 24% meminfo.Active(file)
> 34922 ± 23% +551.6% 227542 meminfo.AnonHugePages
> 397478 -24.1% 301733 meminfo.AnonPages
> 653045 -19.5% 525838 meminfo.Committed_AS
> 154701 ± 15% -42.0% 89789 ± 4% meminfo.Dirty
> 401768 -23.5% 307192 meminfo.Inactive(anon)
> 33307 -11.3% 29548 meminfo.Mapped
> 6837 ± 7% -27.6% 4950 meminfo.PageTables
> 9016 ± 7% +814.1% 82421 ± 6% meminfo.Shmem
> 324959 ± 7% +25.1% 406488 ± 2% meminfo.Writeback
> 1940 ± 58% +683.0% 15195 ± 36% numa-meminfo.node0.Active(anon)
> 18757 ± 14% -33.6% 12450 ± 30% numa-meminfo.node0.Active(file)
> 18975 ± 47% +872.5% 184526 ± 25% numa-meminfo.node0.AnonHugePages
> 79966 ± 22% -38.8% 48963 ± 8% numa-meminfo.node0.Dirty
> 5833 ± 21% +235.6% 19573 ± 31% numa-meminfo.node0.Shmem
> 23325 ± 16% +198.4% 69601 ± 9% numa-meminfo.node1.Active
> 2576 ± 43% +2270.7% 61072 ± 5% numa-meminfo.node1.Active(anon)
> 20749 ± 15% -58.9% 8528 ± 46% numa-meminfo.node1.Active(file)
> 183812 ± 19% -67.5% 59692 ± 75% numa-meminfo.node1.AnonPages
> 67153 ± 14% -39.2% 40835 ± 14% numa-meminfo.node1.Dirty
> 184305 ± 19% -66.8% 61274 ± 73% numa-meminfo.node1.Inactive(anon)
> 3143 ± 25% -37.2% 1973 ± 21% numa-meminfo.node1.PageTables
> 3184 ± 36% +1873.9% 62849 ± 5% numa-meminfo.node1.Shmem
> 339.17 ± 5% -91.4% 29.25 turbostat.Avg_MHz
> 11.43 ± 5% -8.2 3.23 turbostat.Busy%
> 3004 -69.7% 909.25 turbostat.Bzy_MHz
> 52538 ± 7% +834.8% 491105 ± 18% turbostat.C1
> 0.41 ± 7% -0.4 0.01 ± 57% turbostat.C1%
> 849439 ± 2% +679.9% 6624934 ± 48% turbostat.C1E
> 23.10 ± 3% -20.1 3.02 ± 53% turbostat.C1E%
> 1067521 ± 8% +17084.8% 1.835e+08 ± 20% turbostat.C6
> 64.56 ± 2% +29.6 94.19 turbostat.C6%
> 87.74 +10.3% 96.76 turbostat.CPU%c1
> 0.83 ± 45% -98.5% 0.01 ± 34% turbostat.CPU%c6
> 128.85 ±152% -99.9% 0.09 turbostat.IPC
> 1858116 ± 5% +10114.2% 1.898e+08 ± 20% turbostat.IRQ
> 0.64 ± 6% -0.6 0.01 ±100% turbostat.POLL%
> 154.16 -23.3% 118.17 turbostat.PkgWatt
> 63.10 -15.5% 53.29 turbostat.RAMWatt
> 484.83 ± 58% +683.5% 3798 ± 36% numa-vmstat.node0.nr_active_anon
> 4690 ± 14% -33.6% 3112 ± 30% numa-vmstat.node0.nr_active_file
> 8.83 ± 51% +916.0% 89.75 ± 25% numa-vmstat.node0.nr_anon_transparent_hugepages
> 19775 ± 24% -38.3% 12194 ± 8% numa-vmstat.node0.nr_dirty
> 1457 ± 21% +235.7% 4893 ± 31% numa-vmstat.node0.nr_shmem
> 484.83 ± 58% +683.5% 3798 ± 36% numa-vmstat.node0.nr_zone_active_anon
> 4690 ± 14% -33.6% 3112 ± 30% numa-vmstat.node0.nr_zone_active_file
> 36457 ± 22% -59.5% 14783 ± 10% numa-vmstat.node0.nr_zone_write_pending
> 7210556 ± 6% +37.5% 9914914 ± 6% numa-vmstat.node0.numa_hit
> 7176711 ± 6% +37.6% 9873455 ± 6% numa-vmstat.node0.numa_local
> 643.83 ± 43% +2271.4% 15268 ± 5% numa-vmstat.node1.nr_active_anon
> 5189 ± 15% -58.9% 2132 ± 46% numa-vmstat.node1.nr_active_file
> 45952 ± 19% -67.5% 14923 ± 75% numa-vmstat.node1.nr_anon_pages
> 16398 ± 15% -37.8% 10204 ± 14% numa-vmstat.node1.nr_dirty
> 46074 ± 19% -66.8% 15318 ± 73% numa-vmstat.node1.nr_inactive_anon
> 785.33 ± 25% -37.3% 492.75 ± 21% numa-vmstat.node1.nr_page_table_pages
> 795.83 ± 36% +1874.3% 15712 ± 5% numa-vmstat.node1.nr_shmem
> 643.83 ± 43% +2271.4% 15268 ± 5% numa-vmstat.node1.nr_zone_active_anon
> 5189 ± 15% -58.9% 2132 ± 46% numa-vmstat.node1.nr_zone_active_file
> 46074 ± 19% -66.8% 15318 ± 73% numa-vmstat.node1.nr_zone_inactive_anon
> 34148 ± 11% -62.3% 12864 ± 19% numa-vmstat.node1.nr_zone_write_pending
> 6991051 ± 6% +28.8% 9006790 ± 4% numa-vmstat.node1.numa_hit
> 6958699 ± 6% +29.1% 8981908 ± 4% numa-vmstat.node1.numa_local
> 1128 ± 14% +1589.1% 19066 ± 6% proc-vmstat.nr_active_anon
> 9870 ± 4% -46.9% 5244 ± 24% proc-vmstat.nr_active_file
> 99377 -24.1% 75431 proc-vmstat.nr_anon_pages
> 16.67 ± 23% +564.5% 110.75 proc-vmstat.nr_anon_transparent_hugepages
> 38368 ± 15% -41.6% 22410 ± 4% proc-vmstat.nr_dirty
> 100443 -23.5% 76796 proc-vmstat.nr_inactive_anon
> 15586 -6.6% 14560 proc-vmstat.nr_kernel_stack
> 8326 -11.3% 7386 proc-vmstat.nr_mapped
> 1709 ± 7% -27.6% 1236 proc-vmstat.nr_page_table_pages
> 2254 ± 7% +814.2% 20605 ± 6% proc-vmstat.nr_shmem
> 81381 ± 7% +24.9% 101605 ± 2% proc-vmstat.nr_writeback
> 1128 ± 14% +1589.1% 19066 ± 6% proc-vmstat.nr_zone_active_anon
> 9870 ± 4% -46.9% 5244 ± 24% proc-vmstat.nr_zone_active_file
> 100443 -23.5% 76796 proc-vmstat.nr_zone_inactive_anon
> 71853 ± 11% -61.5% 27641 ± 12% proc-vmstat.nr_zone_write_pending
> 404.67 ± 6% +1276.3% 5569 ± 13% proc-vmstat.numa_hint_faults
> 69.67 ± 27% +4299.9% 3065 ± 77% proc-vmstat.numa_hint_faults_local
> 14202988 +33.2% 18923999 ± 5% proc-vmstat.numa_hit
> 14136791 +33.4% 18857658 ± 5% proc-vmstat.numa_local
> 335.00 ± 10% +1203.3% 4366 ± 82% proc-vmstat.numa_pages_migrated
> 698.50 ± 8% +5553.1% 39487 ± 15% proc-vmstat.numa_pte_updates
> 11408 ± 2% +84.5% 21048 ± 5% proc-vmstat.pgactivate
> 18935279 +26.2% 23897622 ± 4% proc-vmstat.pgalloc_normal
> 225757 +2641.0% 6188094 ± 19% proc-vmstat.pgfault
> 13477492 +37.7% 18558535 ± 5% proc-vmstat.pgfree
> 335.00 ± 10% +1203.3% 4366 ± 82% proc-vmstat.pgmigrate_success
> 8658 ± 2% +3345.5% 298318 ± 19% proc-vmstat.pgreuse
> 637184 ± 40% +3336.2% 21894720 ± 20% proc-vmstat.unevictable_pgs_scanned
> 0.23 ± 10% -76.3% 0.05 sched_debug.cfs_rq:/.h_nr_running.avg
> 0.43 ± 5% -48.4% 0.22 sched_debug.cfs_rq:/.h_nr_running.stddev
> 13720 ± 57% +127.8% 31261 ± 2% sched_debug.cfs_rq:/.load.avg
> 109.64 ± 26% -72.0% 30.67 ± 14% sched_debug.cfs_rq:/.load_avg.avg
> 1053 ± 2% -20.8% 834.32 ± 3% sched_debug.cfs_rq:/.load_avg.max
> 274.91 ± 14% -48.8% 140.77 ± 7% sched_debug.cfs_rq:/.load_avg.stddev
> 13475 ± 20% +63.8% 22072 ± 17% sched_debug.cfs_rq:/.min_vruntime.avg
> 39896 ± 14% +40.9% 56195 ± 15% sched_debug.cfs_rq:/.min_vruntime.max
> 5378 ± 23% +108.5% 11216 ± 14% sched_debug.cfs_rq:/.min_vruntime.min
> 5503 ± 11% +57.4% 8663 ± 13% sched_debug.cfs_rq:/.min_vruntime.stddev
> 0.23 ± 10% -76.3% 0.05 sched_debug.cfs_rq:/.nr_running.avg
> 0.43 ± 5% -48.4% 0.22 sched_debug.cfs_rq:/.nr_running.stddev
> 65.15 ± 36% -93.0% 4.53 ± 76% sched_debug.cfs_rq:/.removed.load_avg.avg
> 1024 -87.2% 131.45 ±119% sched_debug.cfs_rq:/.removed.load_avg.max
> 242.33 ± 17% -90.8% 22.40 ±101% sched_debug.cfs_rq:/.removed.load_avg.stddev
> 23.97 ± 36% -91.6% 2.00 ± 84% sched_debug.cfs_rq:/.removed.runnable_avg.avg
> 504.33 ± 3% -86.8% 66.39 ±118% sched_debug.cfs_rq:/.removed.runnable_avg.max
> 93.28 ± 17% -88.6% 10.66 ±106% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
> 23.97 ± 36% -91.6% 2.00 ± 84% sched_debug.cfs_rq:/.removed.util_avg.avg
> 504.33 ± 3% -86.8% 66.39 ±118% sched_debug.cfs_rq:/.removed.util_avg.max
> 93.28 ± 17% -88.6% 10.66 ±106% sched_debug.cfs_rq:/.removed.util_avg.stddev
> 447.97 ± 3% -93.2% 30.55 ± 10% sched_debug.cfs_rq:/.runnable_avg.avg
> 1652 ± 21% -65.3% 573.91 sched_debug.cfs_rq:/.runnable_avg.max
> 361.15 ± 10% -72.7% 98.67 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev
> 5515 ± 10% +57.1% 8663 ± 13% sched_debug.cfs_rq:/.spread0.stddev
> 447.20 ± 3% -93.2% 30.54 ± 10% sched_debug.cfs_rq:/.util_avg.avg
> 1651 ± 21% -65.2% 573.90 sched_debug.cfs_rq:/.util_avg.max
> 360.75 ± 10% -72.7% 98.65 ± 4% sched_debug.cfs_rq:/.util_avg.stddev
> 35.87 ± 32% -92.8% 2.57 ± 23% sched_debug.cfs_rq:/.util_est_enqueued.avg
> 627.00 ± 32% -88.2% 73.92 ± 11% sched_debug.cfs_rq:/.util_est_enqueued.max
> 125.31 ± 21% -90.6% 11.82 ± 13% sched_debug.cfs_rq:/.util_est_enqueued.stddev
> 773027 ± 4% +25.1% 966950 sched_debug.cpu.avg_idle.avg
> 3844 ± 16% +9867.6% 383238 ± 36% sched_debug.cpu.avg_idle.min
> 284379 ± 22% -62.7% 106134 ± 15% sched_debug.cpu.avg_idle.stddev
> 56501 ± 11% +2798.3% 1637565 ± 18% sched_debug.cpu.clock.avg
> 56505 ± 11% +2798.1% 1637568 ± 18% sched_debug.cpu.clock.max
> 56493 ± 11% +2798.7% 1637561 ± 18% sched_debug.cpu.clock.min
> 2.84 ± 14% -30.1% 1.98 ± 3% sched_debug.cpu.clock.stddev
> 55933 ± 11% +2766.0% 1603047 ± 18% sched_debug.cpu.clock_task.avg
> 56388 ± 11% +2747.3% 1605532 ± 18% sched_debug.cpu.clock_task.max
> 43272 ± 5% +3548.5% 1578791 ± 18% sched_debug.cpu.clock_task.min
> 1611 ± 45% +102.9% 3269 ± 31% sched_debug.cpu.clock_task.stddev
> 3920 +950.0% 41168 ± 18% sched_debug.cpu.curr->pid.max
> 1512 ± 2% +241.7% 5166 ± 17% sched_debug.cpu.curr->pid.stddev
> 0.00 ± 28% -66.7% 0.00 ± 4% sched_debug.cpu.next_balance.stddev
> 0.22 ± 7% -87.6% 0.03 ± 4% sched_debug.cpu.nr_running.avg
> 0.42 ± 5% -63.9% 0.15 ± 2% sched_debug.cpu.nr_running.stddev
> 6681 +2922.6% 201964 ± 20% sched_debug.cpu.nr_switches.avg
> 203120 +243.0% 696661 ± 25% sched_debug.cpu.nr_switches.max
> 347.50 ± 19% +12362.1% 43305 ± 29% sched_debug.cpu.nr_switches.min
> 26003 ± 2% +367.6% 121580 ± 15% sched_debug.cpu.nr_switches.stddev
> 56496 ± 11% +2798.5% 1637561 ± 18% sched_debug.cpu_clk
> 55771 ± 11% +2834.9% 1636836 ± 18% sched_debug.ktime
> 51473 ± 5% +3047.3% 1620032 ± 18% sched_debug.sched_clk
> 17.02 ± 3% -78.0% 3.74 perf-stat.i.MPKI
> 1.839e+09 ± 6% -94.8% 96458674 ± 5% perf-stat.i.branch-instructions
> 1.20 ± 7% -0.4 0.80 perf-stat.i.branch-miss-rate%
> 22120063 ± 10% -95.7% 961876 ± 9% perf-stat.i.branch-misses
> 59.22 ± 3% -50.6 8.66 perf-stat.i.cache-miss-rate%
> 91195048 ± 7% -99.1% 801389 ± 18% perf-stat.i.cache-misses
> 1.507e+08 ± 7% -98.3% 2538095 ± 10% perf-stat.i.cache-references
> 119037 ± 8% -92.7% 8685 ± 8% perf-stat.i.context-switches
> 2.52 ± 3% -7.7% 2.33 perf-stat.i.cpi
> 2.236e+10 ± 6% -95.2% 1.081e+09 ± 3% perf-stat.i.cpu-cycles
> 264.18 ± 12% -71.9% 74.16 perf-stat.i.cpu-migrations
> 331.37 ± 29% +2939.3% 10071 ± 2% perf-stat.i.cycles-between-cache-misses
> 0.03 ± 14% +0.1 0.09 ± 23% perf-stat.i.dTLB-load-miss-rate%
> 734621 ± 10% -86.2% 101019 ± 22% perf-stat.i.dTLB-load-misses
> 2.249e+09 ± 5% -94.3% 1.281e+08 ± 4% perf-stat.i.dTLB-loads
> 0.01 ± 10% +0.0 0.04 ± 13% perf-stat.i.dTLB-store-miss-rate%
> 122554 ± 8% -80.7% 23688 ± 11% perf-stat.i.dTLB-store-misses
> 9.999e+08 ± 5% -93.2% 67811239 ± 3% perf-stat.i.dTLB-stores
> 9.066e+09 ± 6% -94.6% 4.866e+08 ± 4% perf-stat.i.instructions
> 0.47 ± 64% -97.2% 0.01 ± 42% perf-stat.i.major-faults
> 0.35 ± 6% -95.2% 0.02 ± 3% perf-stat.i.metric.GHz
> 1229 ± 6% -34.7% 802.68 perf-stat.i.metric.K/sec
> 81.72 ± 5% -95.3% 3.81 ± 5% perf-stat.i.metric.M/sec
> 3477 ± 6% -42.5% 1998 perf-stat.i.minor-faults
> 52.18 ± 4% +41.9 94.08 perf-stat.i.node-load-miss-rate%
> 18404682 ± 7% -99.0% 176907 ± 14% perf-stat.i.node-load-misses
> 16742387 ± 9% -99.1% 144408 ± 22% perf-stat.i.node-loads
> 12.77 ± 9% +18.7 31.45 ± 27% perf-stat.i.node-store-miss-rate%
> 4850218 ± 12% -99.0% 49478 ± 9% perf-stat.i.node-store-misses
> 36531812 ± 6% -99.1% 320605 ± 19% perf-stat.i.node-stores
> 3478 ± 6% -42.5% 1998 perf-stat.i.page-faults
> 16.63 ± 3% -68.7% 5.21 ± 5% perf-stat.overall.MPKI
> 1.20 ± 7% -0.2 1.00 ± 4% perf-stat.overall.branch-miss-rate%
> 60.51 ± 2% -29.2 31.34 ± 8% perf-stat.overall.cache-miss-rate%
> 2.47 ± 3% -9.9% 2.22 perf-stat.overall.cpi
> 245.31 +464.9% 1385 ± 15% perf-stat.overall.cycles-between-cache-misses
> 0.03 ± 13% +0.0 0.08 ± 22% perf-stat.overall.dTLB-load-miss-rate%
> 0.01 ± 9% +0.0 0.03 ± 12% perf-stat.overall.dTLB-store-miss-rate%
> 0.41 ± 3% +10.9% 0.45 perf-stat.overall.ipc
> 1.76e+09 ± 5% -94.5% 96442481 ± 5% perf-stat.ps.branch-instructions
> 21170965 ± 10% -95.5% 961854 ± 9% perf-stat.ps.branch-misses
> 87299736 ± 6% -99.1% 801893 ± 18% perf-stat.ps.cache-misses
> 1.443e+08 ± 6% -98.2% 2538635 ± 10% perf-stat.ps.cache-references
> 113951 ± 8% -92.4% 8683 ± 8% perf-stat.ps.context-switches
> 61339 +4.3% 63977 perf-stat.ps.cpu-clock
> 2.141e+10 ± 6% -95.0% 1.081e+09 ± 3% perf-stat.ps.cpu-cycles
> 252.91 ± 12% -70.7% 74.14 perf-stat.ps.cpu-migrations
> 703511 ± 10% -85.6% 100987 ± 22% perf-stat.ps.dTLB-load-misses
> 2.153e+09 ± 5% -94.1% 1.281e+08 ± 4% perf-stat.ps.dTLB-loads
> 117341 ± 8% -79.8% 23681 ± 11% perf-stat.ps.dTLB-store-misses
> 9.573e+08 ± 5% -92.9% 67795911 ± 3% perf-stat.ps.dTLB-stores
> 8.678e+09 ± 5% -94.4% 4.866e+08 ± 4% perf-stat.ps.instructions
> 0.45 ± 64% -97.1% 0.01 ± 42% perf-stat.ps.major-faults
> 3324 ± 5% -39.9% 1998 perf-stat.ps.minor-faults
> 17622482 ± 7% -99.0% 177005 ± 14% perf-stat.ps.node-load-misses
> 16025463 ± 9% -99.1% 144476 ± 22% perf-stat.ps.node-loads
> 4642492 ± 12% -98.9% 49511 ± 10% perf-stat.ps.node-store-misses
> 34969456 ± 6% -99.1% 320794 ± 19% perf-stat.ps.node-stores
> 3324 ± 5% -39.9% 1998 perf-stat.ps.page-faults
> 61339 +4.3% 63977 perf-stat.ps.task-clock
> 2.065e+11 ± 3% +589.5% 1.424e+12 ± 16% perf-stat.total.instructions
> 0.01 ± 60% +113.6% 0.02 ± 20% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 0.00 ±157% +425.0% 0.01 ± 14% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
> 0.03 ± 16% +71.9% 0.05 ± 3% perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork
> 0.01 ± 38% -65.4% 0.00 ± 19% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
> 0.58 ±151% -96.5% 0.02 ± 7% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
> 0.00 ±108% +218.0% 0.01 ± 13% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
> 0.03 ± 21% +92.0% 0.07 perf-sched.sch_delay.max.ms.irq_thread.kthread.ret_from_fork
> 0.00 ±120% +267.2% 0.02 ± 12% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
> 7.44 ±112% -99.6% 0.03 ± 25% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
> 0.04 ± 91% -81.6% 0.01 perf-sched.total_sch_delay.average.ms
> 55.05 ±169% -94.2% 3.20 ± 12% perf-sched.total_sch_delay.max.ms
> 9.41 ± 27% +263.5% 34.20 ± 5% perf-sched.total_wait_and_delay.average.ms
> 2099 ± 13% +875.6% 20478 ± 4% perf-sched.total_wait_and_delay.count.ms
> 309.85 ± 2% +1511.3% 4992 perf-sched.total_wait_and_delay.max.ms
> 9.37 ± 28% +265.1% 34.20 ± 5% perf-sched.total_wait_time.average.ms
> 309.85 ± 2% +1511.3% 4992 perf-sched.total_wait_time.max.ms
> 1.83 ± 24% +1179.9% 23.43 perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
> 68.28 ± 38% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 0.93 ± 30% +26685.0% 250.13 ± 3% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 69.46 +46.7% 101.87 perf-sched.wait_and_delay.avg.ms.irq_thread.kthread.ret_from_fork
> 1.64 ± 6% +2380.5% 40.63 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
> 39.18 ± 14% +863.6% 377.57 perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
> 4.35 ± 6% +576.4% 29.42 ± 6% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 32.32 ± 47% +1966.0% 667.82 perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> 49.09 ± 11% +739.5% 412.11 ± 4% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
> 245.00 ± 39% +56.7% 384.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
> 3.83 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 507.67 ± 9% -75.6% 124.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 449.83 ± 9% -72.7% 123.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
> 3.00 +1500.0% 48.00 perf-sched.wait_and_delay.count.irq_thread.kthread.ret_from_fork
> 343.83 ± 9% +416.7% 1776 ± 2% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
> 73.33 ± 8% +121.9% 162.75 ± 6% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 302.83 ± 30% +71.9% 520.50 perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
> 117.50 ± 18% +212.1% 366.75 ± 4% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
> 299.23 ± 3% +1568.5% 4992 perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
> 236.11 ± 27% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 69.82 ±139% +2069.6% 1514 ± 31% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 238.02 ± 28% +322.4% 1005 perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
> 292.85 ± 3% +70.7% 499.98 perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
> 5.04 +6666.8% 340.75 ± 18% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 305.61 ± 2% +800.7% 2752 ± 29% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
> 303.33 ± 4% +705.6% 2443 ± 13% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
> 1.83 ± 24% +1180.3% 23.43 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
> 68.27 ± 38% -97.3% 1.82 ± 12% perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 0.00 ±223% +7.7e+07% 766.71 ± 7% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
> 0.93 ± 30% +26838.2% 250.12 ± 3% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 0.29 ± 69% -100.0% 0.00 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
> 69.43 +46.6% 101.82 perf-sched.wait_time.avg.ms.irq_thread.kthread.ret_from_fork
> 1.63 ± 6% +2390.3% 40.62 ± 2% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
> 38.67 ± 13% +876.3% 377.56 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
> 0.07 ±223% +576.2% 0.49 ± 7% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
> 4.34 ± 6% +578.4% 29.41 ± 6% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 32.27 ± 47% +1969.2% 667.81 perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
> 48.50 ± 11% +749.6% 412.09 ± 4% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
> 299.06 ± 3% +1569.4% 4992 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
> 236.10 ± 28% -98.5% 3.65 ± 12% perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
> 0.00 ±223% +5.5e+07% 1000 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
> 69.82 ±139% +2069.6% 1514 ± 31% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
> 0.55 ± 77% -100.0% 0.00 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
> 238.01 ± 28% +322.4% 1005 perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
> 292.84 ± 3% +70.7% 499.97 perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
> 0.15 ±223% +647.1% 1.09 ± 12% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
> 5.03 +6675.8% 340.74 ± 18% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
> 305.61 ± 2% +800.7% 2752 ± 29% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
> 303.32 ± 4% +705.6% 2443 ± 13% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
> 37.90 ± 63% -36.6 1.29 ± 9% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 37.90 ± 63% -36.6 1.29 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
> 27.62 ± 86% -27.5 0.16 ±173% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
> 24.63 ± 88% -24.6 0.00 perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
> 24.63 ± 88% -24.6 0.00 perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
> 24.63 ± 88% -24.6 0.00 perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
> 24.63 ± 88% -24.6 0.00 perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
> 23.24 ± 82% -23.2 0.00 perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
> 23.24 ± 82% -23.2 0.00 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
> 23.24 ± 82% -23.2 0.00 perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
> 23.24 ± 82% -23.2 0.00 perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
> 19.58 ±103% -19.6 0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 18.70 ±111% -18.7 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 18.70 ±111% -18.7 0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 18.70 ±111% -18.7 0.00 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
> 18.70 ±111% -18.7 0.00 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
> 10.10 ± 88% -10.1 0.00 perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
> 10.23 ± 79% -10.1 0.15 ±173% perf-profile.calltrace.cycles-pp.__libc_start_main
> 10.23 ± 79% -10.1 0.15 ±173% perf-profile.calltrace.cycles-pp.main.__libc_start_main
> 10.23 ± 79% -10.1 0.15 ±173% perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
> 9.52 ± 97% -9.3 0.18 ±173% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 9.52 ± 97% -9.3 0.19 ±173% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 9.52 ± 97% -9.3 0.19 ±173% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
> 6.39 ± 87% -6.4 0.00 perf-profile.calltrace.cycles-pp.__libc_write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
> 6.15 ±101% -6.1 0.00 perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main.__libc_start_main
> 6.15 ±101% -6.1 0.00 perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main.__libc_start_main
> 6.15 ±101% -6.1 0.00 perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
> 6.08 ± 79% -6.1 0.00 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
> 6.08 ± 79% -6.1 0.00 perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
> 4.63 ± 80% -4.6 0.00 perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
> 3.84 ±104% -3.8 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
> 3.84 ±104% -3.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> 3.84 ±104% -3.8 0.00 perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> 3.84 ±104% -3.8 0.00 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
> 3.84 ±104% -3.8 0.00 perf-profile.calltrace.cycles-pp.__mmap
> 0.00 +0.5 0.52 ± 3% perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
> 0.00 +0.5 0.52 ± 3% perf-profile.calltrace.cycles-pp.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 0.00 +0.6 0.59 ± 9% perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
> 0.00 +0.7 0.71 ± 15% perf-profile.calltrace.cycles-pp._raw_spin_trylock.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
> 0.00 +0.8 0.77 ± 25% perf-profile.calltrace.cycles-pp.update_rq_clock_task.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
> 0.00 +0.8 0.79 ± 12% perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
> 0.00 +0.8 0.80 ± 23% perf-profile.calltrace.cycles-pp.__break_lease.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
> 0.00 +0.8 0.80 ± 24% perf-profile.calltrace.cycles-pp.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
> 0.00 +0.8 0.81 ± 12% perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
> 0.00 +0.8 0.84 ± 12% perf-profile.calltrace.cycles-pp.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 0.00 +0.9 0.86 ± 8% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
> 0.00 +1.1 1.10 ± 20% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
> 0.00 +1.1 1.11 ± 2% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__do_softirq
> 0.00 +1.2 1.18 ± 5% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu
> 0.00 +1.3 1.27 ± 28% perf-profile.calltrace.cycles-pp.tick_sched_do_timer.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
> 0.00 +1.4 1.38 ± 49% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
> 0.00 +1.4 1.44 ± 46% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 0.00 +1.4 1.45 ± 14% perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt
> 0.00 +1.5 1.51 ± 12% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
> 0.00 +1.6 1.56 perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
> 0.00 +1.6 1.59 ± 12% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
> 0.00 +1.6 1.59 ± 5% perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
> 0.00 +1.6 1.61 ± 13% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
> 0.00 +1.6 1.62 ± 12% perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
> 0.00 +1.7 1.72 ± 8% perf-profile.calltrace.cycles-pp.ret_from_fork
> 0.00 +1.7 1.72 ± 8% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
> 0.00 +1.9 1.86 ± 10% perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
> 0.00 +2.0 1.97 ± 11% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
> 0.00 +2.2 2.22 ± 4% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 0.00 +2.6 2.56 ± 5% perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 0.00 +2.8 2.85 ± 7% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
> 0.00 +3.1 3.10 ± 8% perf-profile.calltrace.cycles-pp.arch_scale_freq_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
> 0.00 +4.4 4.40 ± 6% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
> 0.00 +4.6 4.62 ± 5% perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
> 0.88 ±223% +4.7 5.61 ± 11% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
> 0.00 +5.5 5.51 ± 4% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 2.36 ±100% +5.6 7.96 ± 9% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
> 2.36 ±100% +5.9 8.22 ± 9% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
> 2.36 ±100% +8.6 10.93 ± 6% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
> 2.36 ±100% +13.5 15.87 ± 4% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
> 0.88 ±223% +21.2 22.07 ± 4% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
> 0.88 ±223% +21.3 22.22 ± 4% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
> 0.88 ±223% +30.2 31.04 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
> 0.88 ±223% +38.9 39.74 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
> 0.00 +42.2 42.16 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
> 0.88 ±223% +81.5 82.34 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
> 0.88 ±223% +82.4 83.24 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
> 2.48 ±158% +87.7 90.16 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 2.48 ±158% +87.9 90.37 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 2.48 ±158% +87.9 90.38 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
> 0.88 ±223% +88.1 88.95 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
> 2.48 ±158% +89.5 92.00 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
> 74.91 ± 19% -71.0 3.92 ± 2% perf-profile.children.cycles-pp.do_syscall_64
> 74.91 ± 19% -71.0 3.94 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
> 28.80 ± 65% -28.5 0.29 ± 33% perf-profile.children.cycles-pp.zap_pte_range
> 28.80 ± 65% -28.5 0.31 ± 36% perf-profile.children.cycles-pp.unmap_page_range
> 28.80 ± 65% -28.5 0.31 ± 36% perf-profile.children.cycles-pp.zap_pmd_range
> 28.80 ± 65% -28.5 0.33 ± 29% perf-profile.children.cycles-pp.unmap_vmas
> 23.24 ± 82% -23.2 0.00 perf-profile.children.cycles-pp.arch_do_signal_or_restart
> 23.24 ± 82% -23.2 0.00 perf-profile.children.cycles-pp.get_signal
> 7.90 ± 90% -7.9 0.01 ±173% perf-profile.children.cycles-pp.record__mmap_read_evlist
> 6.39 ± 87% -6.4 0.01 ±173% perf-profile.children.cycles-pp.generic_perform_write
> 6.39 ± 87% -6.4 0.01 ±173% perf-profile.children.cycles-pp.perf_mmap__push
> 6.39 ± 87% -6.4 0.01 ±173% perf-profile.children.cycles-pp.record__pushfn
> 6.39 ± 87% -6.4 0.01 ±173% perf-profile.children.cycles-pp.writen
> 6.39 ± 87% -6.4 0.01 ±173% perf-profile.children.cycles-pp.__libc_write
> 6.39 ± 87% -6.4 0.02 ±173% perf-profile.children.cycles-pp.__generic_file_write_iter
> 6.39 ± 87% -6.4 0.02 ±173% perf-profile.children.cycles-pp.generic_file_write_iter
> 0.00 +0.1 0.06 ± 17% perf-profile.children.cycles-pp.unlink_anon_vmas
> 0.00 +0.1 0.06 ± 31% perf-profile.children.cycles-pp.__get_user_pages
> 0.00 +0.1 0.07 ± 28% perf-profile.children.cycles-pp.cpuidle_not_available
> 0.00 +0.1 0.07 ± 28% perf-profile.children.cycles-pp.__lookup_slow
> 0.00 +0.1 0.07 ± 26% perf-profile.children.cycles-pp.check_move_unevictable_folios
> 0.00 +0.1 0.07 ± 26% perf-profile.children.cycles-pp.check_move_unevictable_pages
> 0.00 +0.1 0.07 ± 14% perf-profile.children.cycles-pp.__percpu_counter_init
> 0.00 +0.1 0.07 ± 26% perf-profile.children.cycles-pp.get_arg_page
> 0.00 +0.1 0.07 ± 26% perf-profile.children.cycles-pp.get_user_pages_remote
> 0.00 +0.1 0.08 ± 26% perf-profile.children.cycles-pp.drm_gem_shmem_put_pages_locked
> 0.00 +0.1 0.08 ± 26% perf-profile.children.cycles-pp.drm_gem_put_pages
> 0.00 +0.1 0.08 ± 26% perf-profile.children.cycles-pp.drm_gem_check_release_pagevec
> 0.00 +0.1 0.08 ± 23% perf-profile.children.cycles-pp.hrtimer_run_queues
> 0.00 +0.1 0.08 ± 45% perf-profile.children.cycles-pp.__x64_sys_mprotect
> 0.00 +0.1 0.08 ± 45% perf-profile.children.cycles-pp.do_mprotect_pkey
> 0.00 +0.1 0.09 ± 33% perf-profile.children.cycles-pp.lookup_fast
> 0.00 +0.1 0.09 ± 29% perf-profile.children.cycles-pp.rcu_do_batch
> 0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp.can_stop_idle_tick
> 0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp.__vmalloc_node_range
> 0.00 +0.1 0.09 ± 32% perf-profile.children.cycles-pp.alloc_thread_stack_node
> 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.perf_read
> 0.00 +0.1 0.10 ± 26% perf-profile.children.cycles-pp.perf_evsel__read
> 0.00 +0.1 0.10 ± 21% perf-profile.children.cycles-pp.menu_reflect
> 0.00 +0.1 0.10 ± 21% perf-profile.children.cycles-pp.alloc_bprm
> 0.00 +0.1 0.10 ± 12% perf-profile.children.cycles-pp.vma_interval_tree_insert
> 0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.nohz_balancer_kick
> 0.00 +0.1 0.10 ± 18% perf-profile.children.cycles-pp.mm_init
> 0.00 +0.1 0.10 ± 14% perf-profile.children.cycles-pp.note_gp_changes
> 0.00 +0.1 0.11 ± 21% perf-profile.children.cycles-pp.cpuidle_reflect
> 0.00 +0.1 0.11 ± 9% perf-profile.children.cycles-pp.dup_task_struct
> 0.00 +0.1 0.11 ± 52% perf-profile.children.cycles-pp.find_idlest_cpu
> 0.00 +0.1 0.11 ± 52% perf-profile.children.cycles-pp.find_idlest_group
> 0.00 +0.1 0.12 ± 11% perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
> 0.00 +0.1 0.12 ± 11% perf-profile.children.cycles-pp.drm_gem_vunmap
> 0.00 +0.1 0.12 ± 11% perf-profile.children.cycles-pp.drm_gem_shmem_vunmap
> 0.00 +0.1 0.12 ± 38% perf-profile.children.cycles-pp.irq_work_run_list
> 0.00 +0.1 0.12 ± 13% perf-profile.children.cycles-pp.__switch_to_asm
> 0.00 +0.1 0.13 ± 11% perf-profile.children.cycles-pp.pipe_read
> 0.00 +0.1 0.14 ± 8% perf-profile.children.cycles-pp.exec_mmap
> 0.00 +0.1 0.14 ± 44% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
> 0.00 +0.1 0.14 ± 14% perf-profile.children.cycles-pp.irqentry_exit
> 0.00 +0.1 0.14 ± 14% perf-profile.children.cycles-pp.walk_component
> 0.00 +0.1 0.14 ± 39% perf-profile.children.cycles-pp.pick_next_task_fair
> 0.00 +0.1 0.15 ± 25% perf-profile.children.cycles-pp.smpboot_thread_fn
> 0.00 +0.2 0.15 ± 51% perf-profile.children.cycles-pp.update_wall_time
> 0.00 +0.2 0.15 ± 51% perf-profile.children.cycles-pp.timekeeping_advance
> 0.00 +0.2 0.16 ± 47% perf-profile.children.cycles-pp.copy_strings
> 0.00 +0.2 0.16 ± 16% perf-profile.children.cycles-pp.ct_kernel_exit
> 0.00 +0.2 0.16 ± 40% perf-profile.children.cycles-pp.select_task_rq_fair
> 0.00 +0.2 0.16 ± 33% perf-profile.children.cycles-pp.perf_event_task_tick
> 0.00 +0.2 0.16 ± 4% perf-profile.children.cycles-pp.__libc_read
> 0.00 +0.2 0.16 ± 28% perf-profile.children.cycles-pp.__x64_sys_vfork
> 0.00 +0.2 0.16 ± 19% perf-profile.children.cycles-pp.intel_idle_irq
> 0.00 +0.2 0.16 ± 22% perf-profile.children.cycles-pp.___perf_sw_event
> 0.00 +0.2 0.16 ± 28% perf-profile.children.cycles-pp.update_rt_rq_load_avg
> 0.00 +0.2 0.16 ± 39% perf-profile.children.cycles-pp.arch_cpu_idle_enter
> 0.00 +0.2 0.16 ± 16% perf-profile.children.cycles-pp._find_next_and_bit
> 0.00 +0.2 0.16 ± 24% perf-profile.children.cycles-pp.evsel__read_counter
> 0.00 +0.2 0.17 ± 13% perf-profile.children.cycles-pp.irqtime_account_process_tick
> 0.00 +0.2 0.17 ± 49% perf-profile.children.cycles-pp.shmem_read_folio_gfp
> 0.00 +0.2 0.17 ± 33% perf-profile.children.cycles-pp.rb_erase
> 0.00 +0.2 0.18 ± 31% perf-profile.children.cycles-pp.drm_atomic_commit
> 0.00 +0.2 0.18 ± 31% perf-profile.children.cycles-pp.drm_atomic_helper_commit
> 0.00 +0.2 0.18 ± 16% perf-profile.children.cycles-pp.hrtimer_forward
> 0.00 +0.2 0.18 ± 4% perf-profile.children.cycles-pp.readn
> 0.00 +0.2 0.18 ± 30% perf-profile.children.cycles-pp.sched_clock
> 0.00 +0.2 0.18 ± 27% perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
> 0.00 +0.2 0.19 ± 28% perf-profile.children.cycles-pp.check_cpu_stall
> 0.00 +0.2 0.19 ± 30% perf-profile.children.cycles-pp.begin_new_exec
> 0.00 +0.2 0.19 ± 20% perf-profile.children.cycles-pp.tick_nohz_tick_stopped
> 0.00 +0.2 0.19 ± 19% perf-profile.children.cycles-pp.__vfork
> 0.00 +0.2 0.20 ± 34% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
> 0.00 +0.2 0.20 ± 34% perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
> 0.00 +0.2 0.20 ± 34% perf-profile.children.cycles-pp.drm_gem_get_pages
> 0.00 +0.2 0.20 ± 34% perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
> 0.00 +0.2 0.20 ± 18% perf-profile.children.cycles-pp.rb_next
> 0.00 +0.2 0.20 ± 39% perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
> 0.00 +0.2 0.20 ± 39% perf-profile.children.cycles-pp.drm_gem_vmap
> 0.00 +0.2 0.20 ± 39% perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
> 0.00 +0.2 0.20 ± 39% perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
> 0.00 +0.2 0.21 ± 45% perf-profile.children.cycles-pp.run_posix_cpu_timers
> 0.00 +0.2 0.21 ± 20% perf-profile.children.cycles-pp.tick_program_event
> 0.00 +0.2 0.22 ±102% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
> 0.00 +0.2 0.23 ± 23% perf-profile.children.cycles-pp.ttwu_do_activate
> 0.00 +0.2 0.23 ± 25% perf-profile.children.cycles-pp.arch_cpu_idle_exit
> 0.00 +0.2 0.24 ± 15% perf-profile.children.cycles-pp.rb_insert_color
> 0.00 +0.2 0.24 ± 45% perf-profile.children.cycles-pp.irqentry_enter
> 0.00 +0.3 0.26 ± 7% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
> 0.00 +0.3 0.27 ± 16% perf-profile.children.cycles-pp.rcu_core
> 0.00 +0.3 0.29 ± 9% perf-profile.children.cycles-pp.error_entry
> 0.00 +0.3 0.31 ± 15% perf-profile.children.cycles-pp.call_cpuidle
> 0.00 +0.3 0.33 ± 22% perf-profile.children.cycles-pp.drm_fb_helper_damage_work
> 0.00 +0.3 0.33 ± 22% perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
> 0.00 +0.3 0.33 ± 27% perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
> 0.00 +0.3 0.33 ± 18% perf-profile.children.cycles-pp.ct_kernel_exit_state
> 0.00 +0.3 0.35 ± 47% perf-profile.children.cycles-pp.ct_kernel_enter
> 0.00 +0.3 0.35 ± 17% perf-profile.children.cycles-pp.hrtimer_update_next_event
> 0.00 +0.4 0.37 ± 16% perf-profile.children.cycles-pp.idle_cpu
> 0.00 +0.4 0.40 ± 30% perf-profile.children.cycles-pp.hrtimer_get_next_event
> 0.00 +0.4 0.40 ± 15% perf-profile.children.cycles-pp.irq_work_tick
> 0.00 +0.4 0.40 ± 20% perf-profile.children.cycles-pp.timerqueue_del
> 0.00 +0.4 0.43 ± 10% perf-profile.children.cycles-pp.call_timer_fn
> 0.00 +0.4 0.44 ± 7% perf-profile.children.cycles-pp.read_counters
> 0.00 +0.4 0.44 ± 45% perf-profile.children.cycles-pp.ct_idle_exit
> 0.00 +0.5 0.48 ± 8% perf-profile.children.cycles-pp.cmd_stat
> 0.00 +0.5 0.48 ± 8% perf-profile.children.cycles-pp.dispatch_events
> 0.00 +0.5 0.48 ± 8% perf-profile.children.cycles-pp.process_interval
> 0.00 +0.5 0.50 ± 18% perf-profile.children.cycles-pp.process_one_work
> 0.00 +0.5 0.51 ± 23% perf-profile.children.cycles-pp.__hrtimer_next_event_base
> 0.00 +0.5 0.51 ± 10% perf-profile.children.cycles-pp._raw_spin_lock_irq
> 0.00 +0.5 0.52 ± 3% perf-profile.children.cycles-pp.run_timer_softirq
> 0.00 +0.5 0.52 ± 27% perf-profile.children.cycles-pp.schedule_timeout
> 0.00 +0.5 0.53 ± 16% perf-profile.children.cycles-pp.worker_thread
> 0.00 +0.5 0.53 ± 28% perf-profile.children.cycles-pp.timerqueue_add
> 0.00 +0.6 0.56 ± 18% perf-profile.children.cycles-pp.schedule
> 0.00 +0.6 0.58 ± 3% perf-profile.children.cycles-pp.__run_timers
> 0.00 +0.6 0.58 ± 18% perf-profile.children.cycles-pp.get_cpu_device
> 0.00 +0.6 0.59 ± 14% perf-profile.children.cycles-pp.tick_nohz_irq_exit
> 0.00 +0.6 0.60 ± 16% perf-profile.children.cycles-pp.update_rq_clock
> 0.00 +0.6 0.62 ± 61% perf-profile.children.cycles-pp.calc_global_load_tick
> 0.00 +0.7 0.66 ± 21% perf-profile.children.cycles-pp.enqueue_hrtimer
> 0.00 +0.7 0.66 ± 17% perf-profile.children.cycles-pp.local_clock
> 0.00 +0.7 0.68 ± 7% perf-profile.children.cycles-pp.hrtimer_next_event_without
> 0.00 +0.7 0.70 ± 24% perf-profile.children.cycles-pp.get_next_timer_interrupt
> 0.00 +0.7 0.74 ± 15% perf-profile.children.cycles-pp._raw_spin_trylock
> 0.00 +0.7 0.75 ± 29% perf-profile.children.cycles-pp.update_irq_load_avg
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.__nfsd_open
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.dentry_open
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.__break_lease
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.nfsd
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.svc_process
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.svc_process_common
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.nfsd_dispatch
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.nfsd4_proc_compound
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.nfsd4_commit
> 0.00 +0.8 0.80 ± 24% perf-profile.children.cycles-pp.nfsd_file_do_acquire
> 0.00 +0.8 0.81 ± 13% perf-profile.children.cycles-pp.update_blocked_averages
> 0.00 +0.8 0.82 ± 25% perf-profile.children.cycles-pp.do_dentry_open
> 0.00 +0.8 0.84 ± 12% perf-profile.children.cycles-pp.run_rebalance_domains
> 0.00 +0.9 0.89 ± 15% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
> 0.00 +1.0 0.98 ± 8% perf-profile.children.cycles-pp.update_sg_lb_stats
> 0.00 +1.0 0.98 ± 24% perf-profile.children.cycles-pp.update_rq_clock_task
> 0.00 +1.1 1.08 ± 9% perf-profile.children.cycles-pp.irqtime_account_irq
> 0.00 +1.2 1.16 ± 17% perf-profile.children.cycles-pp.tick_nohz_next_event
> 0.00 +1.2 1.17 ± 14% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
> 0.00 +1.3 1.25 ± 2% perf-profile.children.cycles-pp.update_sd_lb_stats
> 0.00 +1.3 1.31 ± 2% perf-profile.children.cycles-pp.find_busiest_group
> 0.00 +1.4 1.35 ± 28% perf-profile.children.cycles-pp.tick_sched_do_timer
> 0.00 +1.5 1.46 ± 48% perf-profile.children.cycles-pp.tick_irq_enter
> 0.00 +1.5 1.46 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu
> 0.00 +1.5 1.51 ± 12% perf-profile.children.cycles-pp.__intel_pmu_enable_all
> 0.00 +1.5 1.53 ± 42% perf-profile.children.cycles-pp.irq_enter_rcu
> 0.00 +1.6 1.62 ± 12% perf-profile.children.cycles-pp.x86_64_start_kernel
> 0.00 +1.6 1.62 ± 12% perf-profile.children.cycles-pp.x86_64_start_reservations
> 0.00 +1.6 1.62 ± 12% perf-profile.children.cycles-pp.start_kernel
> 0.00 +1.6 1.62 ± 12% perf-profile.children.cycles-pp.arch_call_rest_init
> 0.00 +1.6 1.62 ± 12% perf-profile.children.cycles-pp.rest_init
> 0.00 +1.7 1.70 ± 7% perf-profile.children.cycles-pp.native_sched_clock
> 0.00 +1.7 1.70 ± 3% perf-profile.children.cycles-pp.load_balance
> 0.00 +1.7 1.71 ± 5% perf-profile.children.cycles-pp.lapic_next_deadline
> 0.00 +1.7 1.72 ± 8% perf-profile.children.cycles-pp.kthread
> 0.00 +1.8 1.76 ± 7% perf-profile.children.cycles-pp.ret_from_fork
> 0.00 +1.9 1.95 ± 16% perf-profile.children.cycles-pp.read_tsc
> 0.00 +2.0 1.98 ± 9% perf-profile.children.cycles-pp.perf_rotate_context
> 0.00 +2.0 2.04 ± 11% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
> 0.00 +2.3 2.30 ± 5% perf-profile.children.cycles-pp.clockevents_program_event
> 0.00 +2.6 2.60 ± 6% perf-profile.children.cycles-pp.rebalance_domains
> 0.00 +3.0 2.99 ± 7% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
> 0.00 +3.2 3.18 ± 7% perf-profile.children.cycles-pp.arch_scale_freq_tick
> 0.00 +3.9 3.92 ± 31% perf-profile.children.cycles-pp.ktime_get
> 0.00 +4.5 4.53 ± 6% perf-profile.children.cycles-pp.menu_select
> 0.00 +4.7 4.71 ± 6% perf-profile.children.cycles-pp.__do_softirq
> 0.88 ±223% +5.0 5.86 ± 11% perf-profile.children.cycles-pp.scheduler_tick
> 0.00 +5.6 5.64 ± 4% perf-profile.children.cycles-pp.__irq_exit_rcu
> 2.36 ±100% +5.8 8.15 ± 9% perf-profile.children.cycles-pp.update_process_times
> 2.36 ±100% +6.0 8.35 ± 9% perf-profile.children.cycles-pp.tick_sched_handle
> 2.36 ±100% +8.8 11.21 ± 6% perf-profile.children.cycles-pp.tick_sched_timer
> 2.36 ±100% +13.9 16.30 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues
> 3.24 ±117% +19.3 22.55 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt
> 3.24 ±117% +19.4 22.66 ± 4% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
> 3.24 ±117% +28.0 31.28 ± 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
> 3.24 ±117% +32.7 35.97 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
> 0.00 +42.2 42.16 perf-profile.children.cycles-pp.intel_idle
> 0.88 ±223% +83.5 84.34 perf-profile.children.cycles-pp.cpuidle_enter_state
> 0.88 ±223% +83.9 84.78 perf-profile.children.cycles-pp.cpuidle_enter
> 2.48 ±158% +87.9 90.38 perf-profile.children.cycles-pp.start_secondary
> 2.48 ±158% +89.5 92.00 perf-profile.children.cycles-pp.secondary_startup_64_no_verify
> 2.48 ±158% +89.5 92.00 perf-profile.children.cycles-pp.cpu_startup_entry
> 2.48 ±158% +89.5 92.00 perf-profile.children.cycles-pp.do_idle
> 0.88 ±223% +89.8 90.65 perf-profile.children.cycles-pp.cpuidle_idle_call
> 0.00 +0.1 0.06 ± 20% perf-profile.self.cycles-pp.shmem_get_folio_gfp
> 0.00 +0.1 0.07 ± 19% perf-profile.self.cycles-pp.tick_irq_enter
> 0.00 +0.1 0.07 ± 26% perf-profile.self.cycles-pp.check_move_unevictable_folios
> 0.00 +0.1 0.07 ± 33% perf-profile.self.cycles-pp.perf_event_task_tick
> 0.00 +0.1 0.08 ± 19% perf-profile.self.cycles-pp.hrtimer_run_queues
> 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp.irqentry_exit
> 0.00 +0.1 0.08 ± 21% perf-profile.self.cycles-pp.hrtimer_get_next_event
> 0.00 +0.1 0.08 ± 26% perf-profile.self.cycles-pp.can_stop_idle_tick
> 0.00 +0.1 0.09 ± 29% perf-profile.self.cycles-pp.irqentry_enter
> 0.00 +0.1 0.09 ± 43% perf-profile.self.cycles-pp.update_blocked_averages
> 0.00 +0.1 0.10 ± 12% perf-profile.self.cycles-pp.vma_interval_tree_insert
> 0.00 +0.1 0.10 ± 34% perf-profile.self.cycles-pp.irq_work_run_list
> 0.00 +0.1 0.10 ± 43% perf-profile.self.cycles-pp.ct_kernel_exit
> 0.00 +0.1 0.10 ± 41% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
> 0.00 +0.1 0.11 ± 13% perf-profile.self.cycles-pp.tick_sched_timer
> 0.00 +0.1 0.11 ± 42% perf-profile.self.cycles-pp.filemap_map_pages
> 0.00 +0.1 0.12 ± 26% perf-profile.self.cycles-pp.rb_erase
> 0.00 +0.1 0.12 ± 56% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
> 0.00 +0.1 0.12 ± 13% perf-profile.self.cycles-pp.__switch_to_asm
> 0.00 +0.1 0.13 ± 21% perf-profile.self.cycles-pp.hrtimer_update_next_event
> 0.00 +0.1 0.13 ± 36% perf-profile.self.cycles-pp.hrtimer_forward
> 0.00 +0.1 0.13 ± 25% perf-profile.self.cycles-pp.rb_next
> 0.00 +0.1 0.13 ± 42% perf-profile.self.cycles-pp.update_rq_clock
> 0.00 +0.1 0.14 ± 34% perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
> 0.00 +0.1 0.14 ± 23% perf-profile.self.cycles-pp.enqueue_hrtimer
> 0.00 +0.1 0.14 ± 32% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
> 0.00 +0.1 0.14 ± 15% perf-profile.self.cycles-pp.rcu_sched_clock_irq
> 0.00 +0.1 0.15 ± 40% perf-profile.self.cycles-pp.clockevents_program_event
> 0.00 +0.1 0.15 ± 23% perf-profile.self.cycles-pp.tick_nohz_tick_stopped
> 0.00 +0.2 0.15 ± 21% perf-profile.self.cycles-pp.___perf_sw_event
> 0.00 +0.2 0.16 ± 14% perf-profile.self.cycles-pp.irqtime_account_process_tick
> 0.00 +0.2 0.16 ± 19% perf-profile.self.cycles-pp._find_next_and_bit
> 0.00 +0.2 0.16 ± 23% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
> 0.00 +0.2 0.16 ± 25% perf-profile.self.cycles-pp.update_rt_rq_load_avg
> 0.00 +0.2 0.16 ± 19% perf-profile.self.cycles-pp.intel_idle_irq
> 0.00 +0.2 0.16 ± 17% perf-profile.self.cycles-pp.hrtimer_next_event_without
> 0.00 +0.2 0.18 ± 45% perf-profile.self.cycles-pp.timerqueue_del
> 0.00 +0.2 0.18 ± 27% perf-profile.self.cycles-pp.check_cpu_stall
> 0.00 +0.2 0.18 ± 36% perf-profile.self.cycles-pp.update_sd_lb_stats
> 0.00 +0.2 0.19 ± 44% perf-profile.self.cycles-pp.run_posix_cpu_timers
> 0.00 +0.2 0.19 ± 18% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
> 0.00 +0.2 0.20 ± 47% perf-profile.self.cycles-pp.__do_softirq
> 0.00 +0.2 0.20 ± 23% perf-profile.self.cycles-pp.tick_program_event
> 0.00 +0.2 0.20 ± 26% perf-profile.self.cycles-pp.rb_insert_color
> 0.00 +0.2 0.20 ± 8% perf-profile.self.cycles-pp.sched_clock_cpu
> 0.00 +0.2 0.21 ± 41% perf-profile.self.cycles-pp.get_next_timer_interrupt
> 0.00 +0.2 0.21 ± 60% perf-profile.self.cycles-pp.ct_kernel_enter
> 0.00 +0.2 0.22 ±102% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
> 0.00 +0.2 0.25 ± 26% perf-profile.self.cycles-pp.update_rq_clock_task
> 0.00 +0.2 0.25 ± 26% perf-profile.self.cycles-pp.__irq_exit_rcu
> 0.00 +0.3 0.26 ± 38% perf-profile.self.cycles-pp.rebalance_domains
> 0.00 +0.3 0.26 ± 33% perf-profile.self.cycles-pp.cpuidle_governor_latency_req
> 0.00 +0.3 0.26 ± 25% perf-profile.self.cycles-pp.perf_rotate_context
> 0.00 +0.3 0.28 ± 20% perf-profile.self.cycles-pp.tick_nohz_next_event
> 0.00 +0.3 0.29 ± 9% perf-profile.self.cycles-pp.error_entry
> 0.00 +0.3 0.29 ± 38% perf-profile.self.cycles-pp.hrtimer_interrupt
> 0.00 +0.3 0.30 ± 18% perf-profile.self.cycles-pp.call_cpuidle
> 0.00 +0.3 0.30 ± 16% perf-profile.self.cycles-pp.load_balance
> 0.00 +0.3 0.31 ± 22% perf-profile.self.cycles-pp.ct_kernel_exit_state
> 0.00 +0.3 0.32 ± 20% perf-profile.self.cycles-pp.irqtime_account_irq
> 0.00 +0.3 0.32 ± 31% perf-profile.self.cycles-pp.timerqueue_add
> 0.00 +0.3 0.32 ± 28% perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
> 0.00 +0.4 0.35 ± 20% perf-profile.self.cycles-pp.scheduler_tick
> 0.00 +0.4 0.36 ± 16% perf-profile.self.cycles-pp.idle_cpu
> 0.00 +0.4 0.38 ± 17% perf-profile.self.cycles-pp.irq_work_tick
> 0.00 +0.4 0.39 ± 20% perf-profile.self.cycles-pp.cpuidle_enter
> 0.00 +0.4 0.40 ± 18% perf-profile.self.cycles-pp.__hrtimer_next_event_base
> 0.00 +0.4 0.45 ± 36% perf-profile.self.cycles-pp.do_idle
> 0.00 +0.5 0.50 ± 7% perf-profile.self.cycles-pp._raw_spin_lock_irq
> 0.00 +0.5 0.51 ± 18% perf-profile.self.cycles-pp.__hrtimer_run_queues
> 0.00 +0.5 0.53 ± 17% perf-profile.self.cycles-pp.get_cpu_device
> 0.00 +0.5 0.54 ± 8% perf-profile.self.cycles-pp.update_process_times
> 0.00 +0.6 0.56 ± 27% perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
> 0.00 +0.6 0.62 ± 61% perf-profile.self.cycles-pp.calc_global_load_tick
> 0.00 +0.7 0.69 ± 16% perf-profile.self.cycles-pp.cpuidle_idle_call
> 0.00 +0.7 0.73 ± 14% perf-profile.self.cycles-pp._raw_spin_trylock
> 0.00 +0.7 0.73 ± 29% perf-profile.self.cycles-pp.update_irq_load_avg
> 0.00 +0.7 0.74 ± 9% perf-profile.self.cycles-pp.update_sg_lb_stats
> 0.00 +1.1 1.10 ± 14% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
> 0.00 +1.2 1.18 ± 31% perf-profile.self.cycles-pp.tick_sched_do_timer
> 0.00 +1.4 1.45 ± 9% perf-profile.self.cycles-pp.menu_select
> 0.00 +1.5 1.51 ± 12% perf-profile.self.cycles-pp.__intel_pmu_enable_all
> 0.00 +1.6 1.63 ± 9% perf-profile.self.cycles-pp.native_sched_clock
> 0.00 +1.7 1.70 ± 4% perf-profile.self.cycles-pp.lapic_next_deadline
> 0.00 +1.9 1.93 ± 16% perf-profile.self.cycles-pp.read_tsc
> 0.00 +2.3 2.30 ± 52% perf-profile.self.cycles-pp.ktime_get
> 0.00 +3.2 3.18 ± 7% perf-profile.self.cycles-pp.arch_scale_freq_tick
> 0.00 +5.8 5.84 ± 9% perf-profile.self.cycles-pp.cpuidle_enter_state
> 0.00 +42.2 42.16 perf-profile.self.cycles-pp.intel_idle
>
>
> ***************************************************************************************************
> lkp-icl-2sp7: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> =========================================================================================
> compiler/cpufreq_governor/debug-setup/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
> gcc-12/performance/no-monitor/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark
>
> commit:
> 3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
> 39d432fc76 ("NFSD: trace nfsctl operations")
>
> 3434d7aa77d24c5c 39d432fc76301cf0a0c45402211
> ---------------- ---------------------------
> %stddev %change %stddev
> \ | \
> 7488368 ± 2% +7979.7% 6.05e+08 ± 47% fsmark.app_overhead
> 62.42 ± 2% -100.0% 0.00 fsmark.files_per_sec
> 22.46 ± 2% +15766.1% 3564 ± 5% fsmark.time.elapsed_time
> 22.46 ± 2% +15766.1% 3564 ± 5% fsmark.time.elapsed_time.max
> 192.33 -99.5% 1.00 fsmark.time.percent_of_cpu_this_job_got
> 43.05 +13.7% 48.94 ± 4% fsmark.time.system_time
> 978884 +2.9% 1007574 ± 2% fsmark.time.voluntary_context_switches
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> --
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
>
--
Chuck Lever