Re: [linus:master] [NFSD] 39d432fc76: fsmark.files_per_sec -100.0% regression

[Chuck Lever]

On Thu, Aug 24, 2023 at 01:59:06PM +0800, kernel test robot wrote:
> 
> 
> hi, Chuck Lever,
> 
> Fengwei (CCed) helped us review this astonishing finding by fsmark tests,
> and doubt below part:
> -			nfsd4_end_grace(nn);
> +			trace_nfsd_end_grace(netns(file));
> 
> and confirmed if adding back:
> 			nfsd4_end_grace(nn);
> 
> the regression is gone and files_per_sec restore to 61.93.

Jeff points out that the problem is that nfsd4_end_grace() was
removed, it's not the tracepoint. Agreed, that's a bug. I'll get
this fixed up.


> FYI.
> 
> 
> Hello,
> 
> kernel test robot noticed a -100.0% regression of fsmark.files_per_sec on:
> 
> 
> commit: 39d432fc76301cf0a0c454022117601994ca9397 ("NFSD: trace nfsctl operations")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> testcase: fsmark
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
> 
> 	iterations: 1x
> 	nr_threads: 32t
> 	disk: 1SSD
> 	fs: btrfs
> 	fs2: nfsv4
> 	filesize: 16MB
> 	test_size: 20G
> 	sync_method: NoSync
> 	nr_directories: 16d
> 	nr_files_per_directory: 256fpd
> 	cpufreq_governor: performance
> 
> 
> In addition to that, the commit also has significant impact on the following tests:
> 
> +------------------+-------------------------------------------------------------------------------------------+
> | testcase: change | fsmark: fsmark.files_per_sec -100.0% regression                                           |
> | test machine     | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
> | test parameters  | cpufreq_governor=performance                                                              |
> |                  | debug-setup=no-monitor                                                                    |
> |                  | disk=1SSD                                                                                 |
> |                  | filesize=16MB                                                                             |
> |                  | fs2=nfsv4                                                                                 |
> |                  | fs=btrfs                                                                                  |
> |                  | iterations=1x                                                                             |
> |                  | nr_directories=16d                                                                        |
> |                  | nr_files_per_directory=256fpd                                                             |
> |                  | nr_threads=32t                                                                            |
> |                  | sync_method=NoSync                                                                        |
> |                  | test_size=20G                                                                             |
> +------------------+-------------------------------------------------------------------------------------------+
> 
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
> | Closes: https://lore.kernel.org/oe-lkp/202308241229.68396422-oliver.sang@xxxxxxxxx
> 
> 
> Details are as below:
> -------------------------------------------------------------------------------------------------->
> 
> 
> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20230824/202308241229.68396422-oliver.sang@xxxxxxxxx
> 
> =========================================================================================
> compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>   gcc-12/performance/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark
> 
> commit: 
>   3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
>   39d432fc76 ("NFSD: trace nfsctl operations")
> 
> 3434d7aa77d24c5c 39d432fc76301cf0a0c45402211 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>      51.67 ± 25%     -90.8%       4.75 ± 45%  perf-c2c.DRAM.local
>      35.36 ± 16%     +35.2%      47.81 ± 16%  boot-time.dhcp
>       1.81 ± 41%     +90.9%       3.46 ± 31%  boot-time.smp_boot
>  1.455e+09 ±  6%  +12542.8%  1.839e+11 ± 20%  cpuidle..time
>    3013365 ±  2%   +6257.5%  1.916e+08 ± 20%  cpuidle..usage
>      81.11 ±  7%   +3758.6%       3129 ± 18%  uptime.boot
>       4349 ±  8%   +4266.2%     189897 ± 19%  uptime.idle
>      71.95           +31.7%      94.75        iostat.cpu.idle
>      18.35 ±  3%     -84.5%       2.85 ± 56%  iostat.cpu.iowait
>       7.98 ±  5%     -70.3%       2.37        iostat.cpu.system
>       1.73 ±  3%     -98.0%       0.03 ± 15%  iostat.cpu.user
>    7176382 ±  6%     +37.6%    9873307 ±  6%  numa-numastat.node0.local_node
>    7210227 ±  6%     +37.5%    9914765 ±  6%  numa-numastat.node0.numa_hit
>    6958614 ±  6%     +29.1%    8981736 ±  4%  numa-numastat.node1.local_node
>    6990967 ±  6%     +28.8%    9006618 ±  4%  numa-numastat.node1.numa_hit
>    8749533 ±  4%   +7050.4%  6.256e+08 ± 18%  fsmark.app_overhead
>      61.92 ±  5%    -100.0%       0.00        fsmark.files_per_sec
>      22.77 ±  4%  +12864.1%       2951 ± 20%  fsmark.time.elapsed_time
>      22.77 ±  4%  +12864.1%       2951 ± 20%  fsmark.time.elapsed_time.max
>     189.33 ±  7%     -99.5%       1.00        fsmark.time.percent_of_cpu_this_job_got
>      70.09           +24.7       94.75        mpstat.cpu.all.idle%
>      19.80 ±  4%     -16.9        2.86 ± 56%  mpstat.cpu.all.iowait%
>       1.18 ±  4%      +1.0        2.20        mpstat.cpu.all.irq%
>       0.13 ±  2%      -0.1        0.07 ±  3%  mpstat.cpu.all.soft%
>       7.10 ±  7%      -7.0        0.09 ± 12%  mpstat.cpu.all.sys%
>       1.70 ±  5%      -1.7        0.03 ± 15%  mpstat.cpu.all.usr%
>      71.33           +32.5%      94.50        vmstat.cpu.id
>      18.00 ±  3%     -87.5%       2.25 ± 79%  vmstat.cpu.wa
>     784537 ±  5%     -99.0%       7461 ± 21%  vmstat.io.bo
>      10.17 ± 23%     -85.2%       1.50 ± 74%  vmstat.procs.b
>       6.00 ± 25%    -100.0%       0.00        vmstat.procs.r
>     101767 ±  7%     -91.5%       8688 ±  8%  vmstat.system.cs
>      70688            -9.1%      64225        vmstat.system.in
>      43996 ±  5%    +121.0%      97246 ±  9%  meminfo.Active
>       4516 ± 14%   +1588.6%      76266 ±  6%  meminfo.Active(anon)
>      39479 ±  4%     -46.9%      20978 ± 24%  meminfo.Active(file)
>      34922 ± 23%    +551.6%     227542        meminfo.AnonHugePages
>     397478           -24.1%     301733        meminfo.AnonPages
>     653045           -19.5%     525838        meminfo.Committed_AS
>     154701 ± 15%     -42.0%      89789 ±  4%  meminfo.Dirty
>     401768           -23.5%     307192        meminfo.Inactive(anon)
>      33307           -11.3%      29548        meminfo.Mapped
>       6837 ±  7%     -27.6%       4950        meminfo.PageTables
>       9016 ±  7%    +814.1%      82421 ±  6%  meminfo.Shmem
>     324959 ±  7%     +25.1%     406488 ±  2%  meminfo.Writeback
>       1940 ± 58%    +683.0%      15195 ± 36%  numa-meminfo.node0.Active(anon)
>      18757 ± 14%     -33.6%      12450 ± 30%  numa-meminfo.node0.Active(file)
>      18975 ± 47%    +872.5%     184526 ± 25%  numa-meminfo.node0.AnonHugePages
>      79966 ± 22%     -38.8%      48963 ±  8%  numa-meminfo.node0.Dirty
>       5833 ± 21%    +235.6%      19573 ± 31%  numa-meminfo.node0.Shmem
>      23325 ± 16%    +198.4%      69601 ±  9%  numa-meminfo.node1.Active
>       2576 ± 43%   +2270.7%      61072 ±  5%  numa-meminfo.node1.Active(anon)
>      20749 ± 15%     -58.9%       8528 ± 46%  numa-meminfo.node1.Active(file)
>     183812 ± 19%     -67.5%      59692 ± 75%  numa-meminfo.node1.AnonPages
>      67153 ± 14%     -39.2%      40835 ± 14%  numa-meminfo.node1.Dirty
>     184305 ± 19%     -66.8%      61274 ± 73%  numa-meminfo.node1.Inactive(anon)
>       3143 ± 25%     -37.2%       1973 ± 21%  numa-meminfo.node1.PageTables
>       3184 ± 36%   +1873.9%      62849 ±  5%  numa-meminfo.node1.Shmem
>     339.17 ±  5%     -91.4%      29.25        turbostat.Avg_MHz
>      11.43 ±  5%      -8.2        3.23        turbostat.Busy%
>       3004           -69.7%     909.25        turbostat.Bzy_MHz
>      52538 ±  7%    +834.8%     491105 ± 18%  turbostat.C1
>       0.41 ±  7%      -0.4        0.01 ± 57%  turbostat.C1%
>     849439 ±  2%    +679.9%    6624934 ± 48%  turbostat.C1E
>      23.10 ±  3%     -20.1        3.02 ± 53%  turbostat.C1E%
>    1067521 ±  8%  +17084.8%  1.835e+08 ± 20%  turbostat.C6
>      64.56 ±  2%     +29.6       94.19        turbostat.C6%
>      87.74           +10.3%      96.76        turbostat.CPU%c1
>       0.83 ± 45%     -98.5%       0.01 ± 34%  turbostat.CPU%c6
>     128.85 ±152%     -99.9%       0.09        turbostat.IPC
>    1858116 ±  5%  +10114.2%  1.898e+08 ± 20%  turbostat.IRQ
>       0.64 ±  6%      -0.6        0.01 ±100%  turbostat.POLL%
>     154.16           -23.3%     118.17        turbostat.PkgWatt
>      63.10           -15.5%      53.29        turbostat.RAMWatt
>     484.83 ± 58%    +683.5%       3798 ± 36%  numa-vmstat.node0.nr_active_anon
>       4690 ± 14%     -33.6%       3112 ± 30%  numa-vmstat.node0.nr_active_file
>       8.83 ± 51%    +916.0%      89.75 ± 25%  numa-vmstat.node0.nr_anon_transparent_hugepages
>      19775 ± 24%     -38.3%      12194 ±  8%  numa-vmstat.node0.nr_dirty
>       1457 ± 21%    +235.7%       4893 ± 31%  numa-vmstat.node0.nr_shmem
>     484.83 ± 58%    +683.5%       3798 ± 36%  numa-vmstat.node0.nr_zone_active_anon
>       4690 ± 14%     -33.6%       3112 ± 30%  numa-vmstat.node0.nr_zone_active_file
>      36457 ± 22%     -59.5%      14783 ± 10%  numa-vmstat.node0.nr_zone_write_pending
>    7210556 ±  6%     +37.5%    9914914 ±  6%  numa-vmstat.node0.numa_hit
>    7176711 ±  6%     +37.6%    9873455 ±  6%  numa-vmstat.node0.numa_local
>     643.83 ± 43%   +2271.4%      15268 ±  5%  numa-vmstat.node1.nr_active_anon
>       5189 ± 15%     -58.9%       2132 ± 46%  numa-vmstat.node1.nr_active_file
>      45952 ± 19%     -67.5%      14923 ± 75%  numa-vmstat.node1.nr_anon_pages
>      16398 ± 15%     -37.8%      10204 ± 14%  numa-vmstat.node1.nr_dirty
>      46074 ± 19%     -66.8%      15318 ± 73%  numa-vmstat.node1.nr_inactive_anon
>     785.33 ± 25%     -37.3%     492.75 ± 21%  numa-vmstat.node1.nr_page_table_pages
>     795.83 ± 36%   +1874.3%      15712 ±  5%  numa-vmstat.node1.nr_shmem
>     643.83 ± 43%   +2271.4%      15268 ±  5%  numa-vmstat.node1.nr_zone_active_anon
>       5189 ± 15%     -58.9%       2132 ± 46%  numa-vmstat.node1.nr_zone_active_file
>      46074 ± 19%     -66.8%      15318 ± 73%  numa-vmstat.node1.nr_zone_inactive_anon
>      34148 ± 11%     -62.3%      12864 ± 19%  numa-vmstat.node1.nr_zone_write_pending
>    6991051 ±  6%     +28.8%    9006790 ±  4%  numa-vmstat.node1.numa_hit
>    6958699 ±  6%     +29.1%    8981908 ±  4%  numa-vmstat.node1.numa_local
>       1128 ± 14%   +1589.1%      19066 ±  6%  proc-vmstat.nr_active_anon
>       9870 ±  4%     -46.9%       5244 ± 24%  proc-vmstat.nr_active_file
>      99377           -24.1%      75431        proc-vmstat.nr_anon_pages
>      16.67 ± 23%    +564.5%     110.75        proc-vmstat.nr_anon_transparent_hugepages
>      38368 ± 15%     -41.6%      22410 ±  4%  proc-vmstat.nr_dirty
>     100443           -23.5%      76796        proc-vmstat.nr_inactive_anon
>      15586            -6.6%      14560        proc-vmstat.nr_kernel_stack
>       8326           -11.3%       7386        proc-vmstat.nr_mapped
>       1709 ±  7%     -27.6%       1236        proc-vmstat.nr_page_table_pages
>       2254 ±  7%    +814.2%      20605 ±  6%  proc-vmstat.nr_shmem
>      81381 ±  7%     +24.9%     101605 ±  2%  proc-vmstat.nr_writeback
>       1128 ± 14%   +1589.1%      19066 ±  6%  proc-vmstat.nr_zone_active_anon
>       9870 ±  4%     -46.9%       5244 ± 24%  proc-vmstat.nr_zone_active_file
>     100443           -23.5%      76796        proc-vmstat.nr_zone_inactive_anon
>      71853 ± 11%     -61.5%      27641 ± 12%  proc-vmstat.nr_zone_write_pending
>     404.67 ±  6%   +1276.3%       5569 ± 13%  proc-vmstat.numa_hint_faults
>      69.67 ± 27%   +4299.9%       3065 ± 77%  proc-vmstat.numa_hint_faults_local
>   14202988           +33.2%   18923999 ±  5%  proc-vmstat.numa_hit
>   14136791           +33.4%   18857658 ±  5%  proc-vmstat.numa_local
>     335.00 ± 10%   +1203.3%       4366 ± 82%  proc-vmstat.numa_pages_migrated
>     698.50 ±  8%   +5553.1%      39487 ± 15%  proc-vmstat.numa_pte_updates
>      11408 ±  2%     +84.5%      21048 ±  5%  proc-vmstat.pgactivate
>   18935279           +26.2%   23897622 ±  4%  proc-vmstat.pgalloc_normal
>     225757         +2641.0%    6188094 ± 19%  proc-vmstat.pgfault
>   13477492           +37.7%   18558535 ±  5%  proc-vmstat.pgfree
>     335.00 ± 10%   +1203.3%       4366 ± 82%  proc-vmstat.pgmigrate_success
>       8658 ±  2%   +3345.5%     298318 ± 19%  proc-vmstat.pgreuse
>     637184 ± 40%   +3336.2%   21894720 ± 20%  proc-vmstat.unevictable_pgs_scanned
>       0.23 ± 10%     -76.3%       0.05        sched_debug.cfs_rq:/.h_nr_running.avg
>       0.43 ±  5%     -48.4%       0.22        sched_debug.cfs_rq:/.h_nr_running.stddev
>      13720 ± 57%    +127.8%      31261 ±  2%  sched_debug.cfs_rq:/.load.avg
>     109.64 ± 26%     -72.0%      30.67 ± 14%  sched_debug.cfs_rq:/.load_avg.avg
>       1053 ±  2%     -20.8%     834.32 ±  3%  sched_debug.cfs_rq:/.load_avg.max
>     274.91 ± 14%     -48.8%     140.77 ±  7%  sched_debug.cfs_rq:/.load_avg.stddev
>      13475 ± 20%     +63.8%      22072 ± 17%  sched_debug.cfs_rq:/.min_vruntime.avg
>      39896 ± 14%     +40.9%      56195 ± 15%  sched_debug.cfs_rq:/.min_vruntime.max
>       5378 ± 23%    +108.5%      11216 ± 14%  sched_debug.cfs_rq:/.min_vruntime.min
>       5503 ± 11%     +57.4%       8663 ± 13%  sched_debug.cfs_rq:/.min_vruntime.stddev
>       0.23 ± 10%     -76.3%       0.05        sched_debug.cfs_rq:/.nr_running.avg
>       0.43 ±  5%     -48.4%       0.22        sched_debug.cfs_rq:/.nr_running.stddev
>      65.15 ± 36%     -93.0%       4.53 ± 76%  sched_debug.cfs_rq:/.removed.load_avg.avg
>       1024           -87.2%     131.45 ±119%  sched_debug.cfs_rq:/.removed.load_avg.max
>     242.33 ± 17%     -90.8%      22.40 ±101%  sched_debug.cfs_rq:/.removed.load_avg.stddev
>      23.97 ± 36%     -91.6%       2.00 ± 84%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
>     504.33 ±  3%     -86.8%      66.39 ±118%  sched_debug.cfs_rq:/.removed.runnable_avg.max
>      93.28 ± 17%     -88.6%      10.66 ±106%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
>      23.97 ± 36%     -91.6%       2.00 ± 84%  sched_debug.cfs_rq:/.removed.util_avg.avg
>     504.33 ±  3%     -86.8%      66.39 ±118%  sched_debug.cfs_rq:/.removed.util_avg.max
>      93.28 ± 17%     -88.6%      10.66 ±106%  sched_debug.cfs_rq:/.removed.util_avg.stddev
>     447.97 ±  3%     -93.2%      30.55 ± 10%  sched_debug.cfs_rq:/.runnable_avg.avg
>       1652 ± 21%     -65.3%     573.91        sched_debug.cfs_rq:/.runnable_avg.max
>     361.15 ± 10%     -72.7%      98.67 ±  4%  sched_debug.cfs_rq:/.runnable_avg.stddev
>       5515 ± 10%     +57.1%       8663 ± 13%  sched_debug.cfs_rq:/.spread0.stddev
>     447.20 ±  3%     -93.2%      30.54 ± 10%  sched_debug.cfs_rq:/.util_avg.avg
>       1651 ± 21%     -65.2%     573.90        sched_debug.cfs_rq:/.util_avg.max
>     360.75 ± 10%     -72.7%      98.65 ±  4%  sched_debug.cfs_rq:/.util_avg.stddev
>      35.87 ± 32%     -92.8%       2.57 ± 23%  sched_debug.cfs_rq:/.util_est_enqueued.avg
>     627.00 ± 32%     -88.2%      73.92 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.max
>     125.31 ± 21%     -90.6%      11.82 ± 13%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
>     773027 ±  4%     +25.1%     966950        sched_debug.cpu.avg_idle.avg
>       3844 ± 16%   +9867.6%     383238 ± 36%  sched_debug.cpu.avg_idle.min
>     284379 ± 22%     -62.7%     106134 ± 15%  sched_debug.cpu.avg_idle.stddev
>      56501 ± 11%   +2798.3%    1637565 ± 18%  sched_debug.cpu.clock.avg
>      56505 ± 11%   +2798.1%    1637568 ± 18%  sched_debug.cpu.clock.max
>      56493 ± 11%   +2798.7%    1637561 ± 18%  sched_debug.cpu.clock.min
>       2.84 ± 14%     -30.1%       1.98 ±  3%  sched_debug.cpu.clock.stddev
>      55933 ± 11%   +2766.0%    1603047 ± 18%  sched_debug.cpu.clock_task.avg
>      56388 ± 11%   +2747.3%    1605532 ± 18%  sched_debug.cpu.clock_task.max
>      43272 ±  5%   +3548.5%    1578791 ± 18%  sched_debug.cpu.clock_task.min
>       1611 ± 45%    +102.9%       3269 ± 31%  sched_debug.cpu.clock_task.stddev
>       3920          +950.0%      41168 ± 18%  sched_debug.cpu.curr->pid.max
>       1512 ±  2%    +241.7%       5166 ± 17%  sched_debug.cpu.curr->pid.stddev
>       0.00 ± 28%     -66.7%       0.00 ±  4%  sched_debug.cpu.next_balance.stddev
>       0.22 ±  7%     -87.6%       0.03 ±  4%  sched_debug.cpu.nr_running.avg
>       0.42 ±  5%     -63.9%       0.15 ±  2%  sched_debug.cpu.nr_running.stddev
>       6681         +2922.6%     201964 ± 20%  sched_debug.cpu.nr_switches.avg
>     203120          +243.0%     696661 ± 25%  sched_debug.cpu.nr_switches.max
>     347.50 ± 19%  +12362.1%      43305 ± 29%  sched_debug.cpu.nr_switches.min
>      26003 ±  2%    +367.6%     121580 ± 15%  sched_debug.cpu.nr_switches.stddev
>      56496 ± 11%   +2798.5%    1637561 ± 18%  sched_debug.cpu_clk
>      55771 ± 11%   +2834.9%    1636836 ± 18%  sched_debug.ktime
>      51473 ±  5%   +3047.3%    1620032 ± 18%  sched_debug.sched_clk
>      17.02 ±  3%     -78.0%       3.74        perf-stat.i.MPKI
>  1.839e+09 ±  6%     -94.8%   96458674 ±  5%  perf-stat.i.branch-instructions
>       1.20 ±  7%      -0.4        0.80        perf-stat.i.branch-miss-rate%
>   22120063 ± 10%     -95.7%     961876 ±  9%  perf-stat.i.branch-misses
>      59.22 ±  3%     -50.6        8.66        perf-stat.i.cache-miss-rate%
>   91195048 ±  7%     -99.1%     801389 ± 18%  perf-stat.i.cache-misses
>  1.507e+08 ±  7%     -98.3%    2538095 ± 10%  perf-stat.i.cache-references
>     119037 ±  8%     -92.7%       8685 ±  8%  perf-stat.i.context-switches
>       2.52 ±  3%      -7.7%       2.33        perf-stat.i.cpi
>  2.236e+10 ±  6%     -95.2%  1.081e+09 ±  3%  perf-stat.i.cpu-cycles
>     264.18 ± 12%     -71.9%      74.16        perf-stat.i.cpu-migrations
>     331.37 ± 29%   +2939.3%      10071 ±  2%  perf-stat.i.cycles-between-cache-misses
>       0.03 ± 14%      +0.1        0.09 ± 23%  perf-stat.i.dTLB-load-miss-rate%
>     734621 ± 10%     -86.2%     101019 ± 22%  perf-stat.i.dTLB-load-misses
>  2.249e+09 ±  5%     -94.3%  1.281e+08 ±  4%  perf-stat.i.dTLB-loads
>       0.01 ± 10%      +0.0        0.04 ± 13%  perf-stat.i.dTLB-store-miss-rate%
>     122554 ±  8%     -80.7%      23688 ± 11%  perf-stat.i.dTLB-store-misses
>  9.999e+08 ±  5%     -93.2%   67811239 ±  3%  perf-stat.i.dTLB-stores
>  9.066e+09 ±  6%     -94.6%  4.866e+08 ±  4%  perf-stat.i.instructions
>       0.47 ± 64%     -97.2%       0.01 ± 42%  perf-stat.i.major-faults
>       0.35 ±  6%     -95.2%       0.02 ±  3%  perf-stat.i.metric.GHz
>       1229 ±  6%     -34.7%     802.68        perf-stat.i.metric.K/sec
>      81.72 ±  5%     -95.3%       3.81 ±  5%  perf-stat.i.metric.M/sec
>       3477 ±  6%     -42.5%       1998        perf-stat.i.minor-faults
>      52.18 ±  4%     +41.9       94.08        perf-stat.i.node-load-miss-rate%
>   18404682 ±  7%     -99.0%     176907 ± 14%  perf-stat.i.node-load-misses
>   16742387 ±  9%     -99.1%     144408 ± 22%  perf-stat.i.node-loads
>      12.77 ±  9%     +18.7       31.45 ± 27%  perf-stat.i.node-store-miss-rate%
>    4850218 ± 12%     -99.0%      49478 ±  9%  perf-stat.i.node-store-misses
>   36531812 ±  6%     -99.1%     320605 ± 19%  perf-stat.i.node-stores
>       3478 ±  6%     -42.5%       1998        perf-stat.i.page-faults
>      16.63 ±  3%     -68.7%       5.21 ±  5%  perf-stat.overall.MPKI
>       1.20 ±  7%      -0.2        1.00 ±  4%  perf-stat.overall.branch-miss-rate%
>      60.51 ±  2%     -29.2       31.34 ±  8%  perf-stat.overall.cache-miss-rate%
>       2.47 ±  3%      -9.9%       2.22        perf-stat.overall.cpi
>     245.31          +464.9%       1385 ± 15%  perf-stat.overall.cycles-between-cache-misses
>       0.03 ± 13%      +0.0        0.08 ± 22%  perf-stat.overall.dTLB-load-miss-rate%
>       0.01 ±  9%      +0.0        0.03 ± 12%  perf-stat.overall.dTLB-store-miss-rate%
>       0.41 ±  3%     +10.9%       0.45        perf-stat.overall.ipc
>   1.76e+09 ±  5%     -94.5%   96442481 ±  5%  perf-stat.ps.branch-instructions
>   21170965 ± 10%     -95.5%     961854 ±  9%  perf-stat.ps.branch-misses
>   87299736 ±  6%     -99.1%     801893 ± 18%  perf-stat.ps.cache-misses
>  1.443e+08 ±  6%     -98.2%    2538635 ± 10%  perf-stat.ps.cache-references
>     113951 ±  8%     -92.4%       8683 ±  8%  perf-stat.ps.context-switches
>      61339            +4.3%      63977        perf-stat.ps.cpu-clock
>  2.141e+10 ±  6%     -95.0%  1.081e+09 ±  3%  perf-stat.ps.cpu-cycles
>     252.91 ± 12%     -70.7%      74.14        perf-stat.ps.cpu-migrations
>     703511 ± 10%     -85.6%     100987 ± 22%  perf-stat.ps.dTLB-load-misses
>  2.153e+09 ±  5%     -94.1%  1.281e+08 ±  4%  perf-stat.ps.dTLB-loads
>     117341 ±  8%     -79.8%      23681 ± 11%  perf-stat.ps.dTLB-store-misses
>  9.573e+08 ±  5%     -92.9%   67795911 ±  3%  perf-stat.ps.dTLB-stores
>  8.678e+09 ±  5%     -94.4%  4.866e+08 ±  4%  perf-stat.ps.instructions
>       0.45 ± 64%     -97.1%       0.01 ± 42%  perf-stat.ps.major-faults
>       3324 ±  5%     -39.9%       1998        perf-stat.ps.minor-faults
>   17622482 ±  7%     -99.0%     177005 ± 14%  perf-stat.ps.node-load-misses
>   16025463 ±  9%     -99.1%     144476 ± 22%  perf-stat.ps.node-loads
>    4642492 ± 12%     -98.9%      49511 ± 10%  perf-stat.ps.node-store-misses
>   34969456 ±  6%     -99.1%     320794 ± 19%  perf-stat.ps.node-stores
>       3324 ±  5%     -39.9%       1998        perf-stat.ps.page-faults
>      61339            +4.3%      63977        perf-stat.ps.task-clock
>  2.065e+11 ±  3%    +589.5%  1.424e+12 ± 16%  perf-stat.total.instructions
>       0.01 ± 60%    +113.6%       0.02 ± 20%  perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>       0.00 ±157%    +425.0%       0.01 ± 14%  perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
>       0.03 ± 16%     +71.9%       0.05 ±  3%  perf-sched.sch_delay.avg.ms.irq_thread.kthread.ret_from_fork
>       0.01 ± 38%     -65.4%       0.00 ± 19%  perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
>       0.58 ±151%     -96.5%       0.02 ±  7%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork
>       0.00 ±108%    +218.0%       0.01 ± 13%  perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
>       0.03 ± 21%     +92.0%       0.07        perf-sched.sch_delay.max.ms.irq_thread.kthread.ret_from_fork
>       0.00 ±120%    +267.2%       0.02 ± 12%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll
>       7.44 ±112%     -99.6%       0.03 ± 25%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
>       0.04 ± 91%     -81.6%       0.01        perf-sched.total_sch_delay.average.ms
>      55.05 ±169%     -94.2%       3.20 ± 12%  perf-sched.total_sch_delay.max.ms
>       9.41 ± 27%    +263.5%      34.20 ±  5%  perf-sched.total_wait_and_delay.average.ms
>       2099 ± 13%    +875.6%      20478 ±  4%  perf-sched.total_wait_and_delay.count.ms
>     309.85 ±  2%   +1511.3%       4992        perf-sched.total_wait_and_delay.max.ms
>       9.37 ± 28%    +265.1%      34.20 ±  5%  perf-sched.total_wait_time.average.ms
>     309.85 ±  2%   +1511.3%       4992        perf-sched.total_wait_time.max.ms
>       1.83 ± 24%   +1179.9%      23.43        perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
>      68.28 ± 38%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>       0.93 ± 30%  +26685.0%     250.13 ±  3%  perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
>      69.46           +46.7%     101.87        perf-sched.wait_and_delay.avg.ms.irq_thread.kthread.ret_from_fork
>       1.64 ±  6%   +2380.5%      40.63 ±  2%  perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
>      39.18 ± 14%    +863.6%     377.57        perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
>       4.35 ±  6%    +576.4%      29.42 ±  6%  perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
>      32.32 ± 47%   +1966.0%     667.82        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
>      49.09 ± 11%    +739.5%     412.11 ±  4%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork
>     245.00 ± 39%     +56.7%     384.00        perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
>       3.83 ± 23%    -100.0%       0.00        perf-sched.wait_and_delay.count.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>     507.67 ±  9%     -75.6%     124.00        perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
>     449.83 ±  9%     -72.7%     123.00        perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
>       3.00         +1500.0%      48.00        perf-sched.wait_and_delay.count.irq_thread.kthread.ret_from_fork
>     343.83 ±  9%    +416.7%       1776 ±  2%  perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64
>      73.33 ±  8%    +121.9%     162.75 ±  6%  perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
>     302.83 ± 30%     +71.9%     520.50        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork
>     117.50 ± 18%    +212.1%     366.75 ±  4%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork
>     299.23 ±  3%   +1568.5%       4992        perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
>     236.11 ± 27%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>      69.82 ±139%   +2069.6%       1514 ± 31%  perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
>     238.02 ± 28%    +322.4%       1005        perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
>     292.85 ±  3%     +70.7%     499.98        perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
>       5.04         +6666.8%     340.75 ± 18%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
>     305.61 ±  2%    +800.7%       2752 ± 29%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork
>     303.33 ±  4%    +705.6%       2443 ± 13%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
>       1.83 ± 24%   +1180.3%      23.43        perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
>      68.27 ± 38%     -97.3%       1.82 ± 12%  perf-sched.wait_time.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>       0.00 ±223%  +7.7e+07%     766.71 ±  7%  perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
>       0.93 ± 30%  +26838.2%     250.12 ±  3%  perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
>       0.29 ± 69%    -100.0%       0.00        perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
>      69.43           +46.6%     101.82        perf-sched.wait_time.avg.ms.irq_thread.kthread.ret_from_fork
>       1.63 ±  6%   +2390.3%      40.62 ±  2%  perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
>      38.67 ± 13%    +876.3%     377.56        perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
>       0.07 ±223%    +576.2%       0.49 ±  7%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
>       4.34 ±  6%    +578.4%      29.41 ±  6%  perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
>      32.27 ± 47%   +1969.2%     667.81        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork
>      48.50 ± 11%    +749.6%     412.09 ±  4%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork
>     299.06 ±  3%   +1569.4%       4992        perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
>     236.10 ± 28%     -98.5%       3.65 ± 12%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
>       0.00 ±223%  +5.5e+07%       1000        perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
>      69.82 ±139%   +2069.6%       1514 ± 31%  perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
>       0.55 ± 77%    -100.0%       0.00        perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
>     238.01 ± 28%    +322.4%       1005        perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64
>     292.84 ±  3%     +70.7%     499.97        perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait
>       0.15 ±223%    +647.1%       1.09 ± 12%  perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
>       5.03         +6675.8%     340.74 ± 18%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
>     305.61 ±  2%    +800.7%       2752 ± 29%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork
>     303.32 ±  4%    +705.6%       2443 ± 13%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
>      37.90 ± 63%     -36.6        1.29 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      37.90 ± 63%     -36.6        1.29 ±  8%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
>      27.62 ± 86%     -27.5        0.16 ±173%  perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit
>      24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit
>      24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm
>      24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput
>      24.63 ± 88%     -24.6        0.00        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap
>      23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare
>      23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
>      23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart
>      23.24 ± 82%     -23.2        0.00        perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.get_signal
>      19.58 ±103%     -19.6        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64
>      18.70 ±111%     -18.7        0.00        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode
>      10.10 ± 88%     -10.1        0.00        perf-profile.calltrace.cycles-pp.page_remove_rmap.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas
>      10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.__libc_start_main
>      10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.main.__libc_start_main
>      10.23 ± 79%     -10.1        0.15 ±173%  perf-profile.calltrace.cycles-pp.run_builtin.main.__libc_start_main
>       9.52 ± 97%      -9.3        0.18 ±173%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       9.52 ± 97%      -9.3        0.19 ±173%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       9.52 ± 97%      -9.3        0.19 ±173%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn.perf_mmap__push
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen.record__pushfn
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.writen
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.generic_file_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.vfs_write.ksys_write
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist.__cmd_record
>       6.39 ± 87%      -6.4        0.00        perf-profile.calltrace.cycles-pp.__libc_write.writen.record__pushfn.perf_mmap__push.record__mmap_read_evlist
>       6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.cmd_record.run_builtin.main.__libc_start_main
>       6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.__cmd_record.cmd_record.run_builtin.main.__libc_start_main
>       6.15 ±101%      -6.1        0.00        perf-profile.calltrace.cycles-pp.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin.main
>       6.08 ± 79%      -6.1        0.00        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       6.08 ± 79%      -6.1        0.00        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
>       4.63 ± 80%      -4.6        0.00        perf-profile.calltrace.cycles-pp.perf_mmap__push.record__mmap_read_evlist.__cmd_record.cmd_record.run_builtin
>       3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
>       3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
>       3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
>       3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
>       3.84 ±104%      -3.8        0.00        perf-profile.calltrace.cycles-pp.__mmap
>       0.00            +0.5        0.52 ±  3%  perf-profile.calltrace.cycles-pp.__run_timers.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
>       0.00            +0.5        0.52 ±  3%  perf-profile.calltrace.cycles-pp.run_timer_softirq.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
>       0.00            +0.6        0.59 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_next_event_without.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
>       0.00            +0.7        0.71 ± 15%  perf-profile.calltrace.cycles-pp._raw_spin_trylock.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
>       0.00            +0.8        0.77 ± 25%  perf-profile.calltrace.cycles-pp.update_rq_clock_task.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
>       0.00            +0.8        0.79 ± 12%  perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
>       0.00            +0.8        0.80 ± 23%  perf-profile.calltrace.cycles-pp.__break_lease.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.do_dentry_open.dentry_open.__nfsd_open.nfsd_file_do_acquire.nfsd4_commit
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.svc_process.nfsd.kthread.ret_from_fork
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.nfsd.kthread.ret_from_fork
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.nfsd.kthread
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.nfsd
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
>       0.00            +0.8        0.80 ± 24%  perf-profile.calltrace.cycles-pp.nfsd_file_do_acquire.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
>       0.00            +0.8        0.81 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
>       0.00            +0.8        0.84 ± 12%  perf-profile.calltrace.cycles-pp.run_rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
>       0.00            +0.9        0.86 ±  8%  perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains
>       0.00            +1.1        1.10 ± 20%  perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle
>       0.00            +1.1        1.11 ±  2%  perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains.__do_softirq
>       0.00            +1.2        1.18 ±  5%  perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu
>       0.00            +1.3        1.27 ± 28%  perf-profile.calltrace.cycles-pp.tick_sched_do_timer.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
>       0.00            +1.4        1.38 ± 49%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
>       0.00            +1.4        1.44 ± 46%  perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
>       0.00            +1.4        1.45 ± 14%  perf-profile.calltrace.cycles-pp.__intel_pmu_enable_all.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt
>       0.00            +1.5        1.51 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
>       0.00            +1.6        1.56        perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt
>       0.00            +1.6        1.59 ± 12%  perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init
>       0.00            +1.6        1.59 ±  5%  perf-profile.calltrace.cycles-pp.lapic_next_deadline.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
>       0.00            +1.6        1.61 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.x86_64_start_kernel.secondary_startup_64_no_verify
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.secondary_startup_64_no_verify
>       0.00            +1.6        1.62 ± 12%  perf-profile.calltrace.cycles-pp.rest_init.arch_call_rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
>       0.00            +1.7        1.72 ±  8%  perf-profile.calltrace.cycles-pp.ret_from_fork
>       0.00            +1.7        1.72 ±  8%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
>       0.00            +1.9        1.86 ± 10%  perf-profile.calltrace.cycles-pp.perf_rotate_context.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
>       0.00            +2.0        1.97 ± 11%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
>       0.00            +2.2        2.22 ±  4%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
>       0.00            +2.6        2.56 ±  5%  perf-profile.calltrace.cycles-pp.rebalance_domains.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
>       0.00            +2.8        2.85 ±  7%  perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
>       0.00            +3.1        3.10 ±  8%  perf-profile.calltrace.cycles-pp.arch_scale_freq_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
>       0.00            +4.4        4.40 ±  6%  perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
>       0.00            +4.6        4.62 ±  5%  perf-profile.calltrace.cycles-pp.__do_softirq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
>       0.88 ±223%      +4.7        5.61 ± 11%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
>       0.00            +5.5        5.51 ±  4%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
>       2.36 ±100%      +5.6        7.96 ±  9%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
>       2.36 ±100%      +5.9        8.22 ±  9%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
>       2.36 ±100%      +8.6       10.93 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
>       2.36 ±100%     +13.5       15.87 ±  4%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
>       0.88 ±223%     +21.2       22.07 ±  4%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
>       0.88 ±223%     +21.3       22.22 ±  4%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
>       0.88 ±223%     +30.2       31.04 ±  5%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
>       0.88 ±223%     +38.9       39.74        perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
>       0.00           +42.2       42.16        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
>       0.88 ±223%     +81.5       82.34        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
>       0.88 ±223%     +82.4       83.24        perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
>       2.48 ±158%     +87.7       90.16        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
>       2.48 ±158%     +87.9       90.37        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
>       2.48 ±158%     +87.9       90.38        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
>       0.88 ±223%     +88.1       88.95        perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
>       2.48 ±158%     +89.5       92.00        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
>      74.91 ± 19%     -71.0        3.92 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
>      74.91 ± 19%     -71.0        3.94 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
>      28.80 ± 65%     -28.5        0.29 ± 33%  perf-profile.children.cycles-pp.zap_pte_range
>      28.80 ± 65%     -28.5        0.31 ± 36%  perf-profile.children.cycles-pp.unmap_page_range
>      28.80 ± 65%     -28.5        0.31 ± 36%  perf-profile.children.cycles-pp.zap_pmd_range
>      28.80 ± 65%     -28.5        0.33 ± 29%  perf-profile.children.cycles-pp.unmap_vmas
>      23.24 ± 82%     -23.2        0.00        perf-profile.children.cycles-pp.arch_do_signal_or_restart
>      23.24 ± 82%     -23.2        0.00        perf-profile.children.cycles-pp.get_signal
>       7.90 ± 90%      -7.9        0.01 ±173%  perf-profile.children.cycles-pp.record__mmap_read_evlist
>       6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.generic_perform_write
>       6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.perf_mmap__push
>       6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.record__pushfn
>       6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.writen
>       6.39 ± 87%      -6.4        0.01 ±173%  perf-profile.children.cycles-pp.__libc_write
>       6.39 ± 87%      -6.4        0.02 ±173%  perf-profile.children.cycles-pp.__generic_file_write_iter
>       6.39 ± 87%      -6.4        0.02 ±173%  perf-profile.children.cycles-pp.generic_file_write_iter
>       0.00            +0.1        0.06 ± 17%  perf-profile.children.cycles-pp.unlink_anon_vmas
>       0.00            +0.1        0.06 ± 31%  perf-profile.children.cycles-pp.__get_user_pages
>       0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp.cpuidle_not_available
>       0.00            +0.1        0.07 ± 28%  perf-profile.children.cycles-pp.__lookup_slow
>       0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.check_move_unevictable_folios
>       0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.check_move_unevictable_pages
>       0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.__percpu_counter_init
>       0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.get_arg_page
>       0.00            +0.1        0.07 ± 26%  perf-profile.children.cycles-pp.get_user_pages_remote
>       0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_shmem_put_pages_locked
>       0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_put_pages
>       0.00            +0.1        0.08 ± 26%  perf-profile.children.cycles-pp.drm_gem_check_release_pagevec
>       0.00            +0.1        0.08 ± 23%  perf-profile.children.cycles-pp.hrtimer_run_queues
>       0.00            +0.1        0.08 ± 45%  perf-profile.children.cycles-pp.__x64_sys_mprotect
>       0.00            +0.1        0.08 ± 45%  perf-profile.children.cycles-pp.do_mprotect_pkey
>       0.00            +0.1        0.09 ± 33%  perf-profile.children.cycles-pp.lookup_fast
>       0.00            +0.1        0.09 ± 29%  perf-profile.children.cycles-pp.rcu_do_batch
>       0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.can_stop_idle_tick
>       0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.__vmalloc_node_range
>       0.00            +0.1        0.09 ± 32%  perf-profile.children.cycles-pp.alloc_thread_stack_node
>       0.00            +0.1        0.09 ± 20%  perf-profile.children.cycles-pp.perf_read
>       0.00            +0.1        0.10 ± 26%  perf-profile.children.cycles-pp.perf_evsel__read
>       0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.menu_reflect
>       0.00            +0.1        0.10 ± 21%  perf-profile.children.cycles-pp.alloc_bprm
>       0.00            +0.1        0.10 ± 12%  perf-profile.children.cycles-pp.vma_interval_tree_insert
>       0.00            +0.1        0.10 ± 27%  perf-profile.children.cycles-pp.nohz_balancer_kick
>       0.00            +0.1        0.10 ± 18%  perf-profile.children.cycles-pp.mm_init
>       0.00            +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.note_gp_changes
>       0.00            +0.1        0.11 ± 21%  perf-profile.children.cycles-pp.cpuidle_reflect
>       0.00            +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.dup_task_struct
>       0.00            +0.1        0.11 ± 52%  perf-profile.children.cycles-pp.find_idlest_cpu
>       0.00            +0.1        0.11 ± 52%  perf-profile.children.cycles-pp.find_idlest_group
>       0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_vunmap_unlocked
>       0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_vunmap
>       0.00            +0.1        0.12 ± 11%  perf-profile.children.cycles-pp.drm_gem_shmem_vunmap
>       0.00            +0.1        0.12 ± 38%  perf-profile.children.cycles-pp.irq_work_run_list
>       0.00            +0.1        0.12 ± 13%  perf-profile.children.cycles-pp.__switch_to_asm
>       0.00            +0.1        0.13 ± 11%  perf-profile.children.cycles-pp.pipe_read
>       0.00            +0.1        0.14 ±  8%  perf-profile.children.cycles-pp.exec_mmap
>       0.00            +0.1        0.14 ± 44%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
>       0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.irqentry_exit
>       0.00            +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.walk_component
>       0.00            +0.1        0.14 ± 39%  perf-profile.children.cycles-pp.pick_next_task_fair
>       0.00            +0.1        0.15 ± 25%  perf-profile.children.cycles-pp.smpboot_thread_fn
>       0.00            +0.2        0.15 ± 51%  perf-profile.children.cycles-pp.update_wall_time
>       0.00            +0.2        0.15 ± 51%  perf-profile.children.cycles-pp.timekeeping_advance
>       0.00            +0.2        0.16 ± 47%  perf-profile.children.cycles-pp.copy_strings
>       0.00            +0.2        0.16 ± 16%  perf-profile.children.cycles-pp.ct_kernel_exit
>       0.00            +0.2        0.16 ± 40%  perf-profile.children.cycles-pp.select_task_rq_fair
>       0.00            +0.2        0.16 ± 33%  perf-profile.children.cycles-pp.perf_event_task_tick
>       0.00            +0.2        0.16 ±  4%  perf-profile.children.cycles-pp.__libc_read
>       0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.__x64_sys_vfork
>       0.00            +0.2        0.16 ± 19%  perf-profile.children.cycles-pp.intel_idle_irq
>       0.00            +0.2        0.16 ± 22%  perf-profile.children.cycles-pp.___perf_sw_event
>       0.00            +0.2        0.16 ± 28%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
>       0.00            +0.2        0.16 ± 39%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
>       0.00            +0.2        0.16 ± 16%  perf-profile.children.cycles-pp._find_next_and_bit
>       0.00            +0.2        0.16 ± 24%  perf-profile.children.cycles-pp.evsel__read_counter
>       0.00            +0.2        0.17 ± 13%  perf-profile.children.cycles-pp.irqtime_account_process_tick
>       0.00            +0.2        0.17 ± 49%  perf-profile.children.cycles-pp.shmem_read_folio_gfp
>       0.00            +0.2        0.17 ± 33%  perf-profile.children.cycles-pp.rb_erase
>       0.00            +0.2        0.18 ± 31%  perf-profile.children.cycles-pp.drm_atomic_commit
>       0.00            +0.2        0.18 ± 31%  perf-profile.children.cycles-pp.drm_atomic_helper_commit
>       0.00            +0.2        0.18 ± 16%  perf-profile.children.cycles-pp.hrtimer_forward
>       0.00            +0.2        0.18 ±  4%  perf-profile.children.cycles-pp.readn
>       0.00            +0.2        0.18 ± 30%  perf-profile.children.cycles-pp.sched_clock
>       0.00            +0.2        0.18 ± 27%  perf-profile.children.cycles-pp.drm_atomic_helper_dirtyfb
>       0.00            +0.2        0.19 ± 28%  perf-profile.children.cycles-pp.check_cpu_stall
>       0.00            +0.2        0.19 ± 30%  perf-profile.children.cycles-pp.begin_new_exec
>       0.00            +0.2        0.19 ± 20%  perf-profile.children.cycles-pp.tick_nohz_tick_stopped
>       0.00            +0.2        0.19 ± 19%  perf-profile.children.cycles-pp.__vfork
>       0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages
>       0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_shmem_get_pages_locked
>       0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.drm_gem_get_pages
>       0.00            +0.2        0.20 ± 34%  perf-profile.children.cycles-pp.shmem_read_mapping_page_gfp
>       0.00            +0.2        0.20 ± 18%  perf-profile.children.cycles-pp.rb_next
>       0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_vmap_unlocked
>       0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_vmap
>       0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_shmem_object_vmap
>       0.00            +0.2        0.20 ± 39%  perf-profile.children.cycles-pp.drm_gem_shmem_vmap_locked
>       0.00            +0.2        0.21 ± 45%  perf-profile.children.cycles-pp.run_posix_cpu_timers
>       0.00            +0.2        0.21 ± 20%  perf-profile.children.cycles-pp.tick_program_event
>       0.00            +0.2        0.22 ±102%  perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
>       0.00            +0.2        0.23 ± 23%  perf-profile.children.cycles-pp.ttwu_do_activate
>       0.00            +0.2        0.23 ± 25%  perf-profile.children.cycles-pp.arch_cpu_idle_exit
>       0.00            +0.2        0.24 ± 15%  perf-profile.children.cycles-pp.rb_insert_color
>       0.00            +0.2        0.24 ± 45%  perf-profile.children.cycles-pp.irqentry_enter
>       0.00            +0.3        0.26 ±  7%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>       0.00            +0.3        0.27 ± 16%  perf-profile.children.cycles-pp.rcu_core
>       0.00            +0.3        0.29 ±  9%  perf-profile.children.cycles-pp.error_entry
>       0.00            +0.3        0.31 ± 15%  perf-profile.children.cycles-pp.call_cpuidle
>       0.00            +0.3        0.33 ± 22%  perf-profile.children.cycles-pp.drm_fb_helper_damage_work
>       0.00            +0.3        0.33 ± 22%  perf-profile.children.cycles-pp.drm_fbdev_generic_helper_fb_dirty
>       0.00            +0.3        0.33 ± 27%  perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event
>       0.00            +0.3        0.33 ± 18%  perf-profile.children.cycles-pp.ct_kernel_exit_state
>       0.00            +0.3        0.35 ± 47%  perf-profile.children.cycles-pp.ct_kernel_enter
>       0.00            +0.3        0.35 ± 17%  perf-profile.children.cycles-pp.hrtimer_update_next_event
>       0.00            +0.4        0.37 ± 16%  perf-profile.children.cycles-pp.idle_cpu
>       0.00            +0.4        0.40 ± 30%  perf-profile.children.cycles-pp.hrtimer_get_next_event
>       0.00            +0.4        0.40 ± 15%  perf-profile.children.cycles-pp.irq_work_tick
>       0.00            +0.4        0.40 ± 20%  perf-profile.children.cycles-pp.timerqueue_del
>       0.00            +0.4        0.43 ± 10%  perf-profile.children.cycles-pp.call_timer_fn
>       0.00            +0.4        0.44 ±  7%  perf-profile.children.cycles-pp.read_counters
>       0.00            +0.4        0.44 ± 45%  perf-profile.children.cycles-pp.ct_idle_exit
>       0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.cmd_stat
>       0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.dispatch_events
>       0.00            +0.5        0.48 ±  8%  perf-profile.children.cycles-pp.process_interval
>       0.00            +0.5        0.50 ± 18%  perf-profile.children.cycles-pp.process_one_work
>       0.00            +0.5        0.51 ± 23%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
>       0.00            +0.5        0.51 ± 10%  perf-profile.children.cycles-pp._raw_spin_lock_irq
>       0.00            +0.5        0.52 ±  3%  perf-profile.children.cycles-pp.run_timer_softirq
>       0.00            +0.5        0.52 ± 27%  perf-profile.children.cycles-pp.schedule_timeout
>       0.00            +0.5        0.53 ± 16%  perf-profile.children.cycles-pp.worker_thread
>       0.00            +0.5        0.53 ± 28%  perf-profile.children.cycles-pp.timerqueue_add
>       0.00            +0.6        0.56 ± 18%  perf-profile.children.cycles-pp.schedule
>       0.00            +0.6        0.58 ±  3%  perf-profile.children.cycles-pp.__run_timers
>       0.00            +0.6        0.58 ± 18%  perf-profile.children.cycles-pp.get_cpu_device
>       0.00            +0.6        0.59 ± 14%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
>       0.00            +0.6        0.60 ± 16%  perf-profile.children.cycles-pp.update_rq_clock
>       0.00            +0.6        0.62 ± 61%  perf-profile.children.cycles-pp.calc_global_load_tick
>       0.00            +0.7        0.66 ± 21%  perf-profile.children.cycles-pp.enqueue_hrtimer
>       0.00            +0.7        0.66 ± 17%  perf-profile.children.cycles-pp.local_clock
>       0.00            +0.7        0.68 ±  7%  perf-profile.children.cycles-pp.hrtimer_next_event_without
>       0.00            +0.7        0.70 ± 24%  perf-profile.children.cycles-pp.get_next_timer_interrupt
>       0.00            +0.7        0.74 ± 15%  perf-profile.children.cycles-pp._raw_spin_trylock
>       0.00            +0.7        0.75 ± 29%  perf-profile.children.cycles-pp.update_irq_load_avg
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.__nfsd_open
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.dentry_open
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.__break_lease
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.svc_process
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.svc_process_common
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd_dispatch
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd4_proc_compound
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd4_commit
>       0.00            +0.8        0.80 ± 24%  perf-profile.children.cycles-pp.nfsd_file_do_acquire
>       0.00            +0.8        0.81 ± 13%  perf-profile.children.cycles-pp.update_blocked_averages
>       0.00            +0.8        0.82 ± 25%  perf-profile.children.cycles-pp.do_dentry_open
>       0.00            +0.8        0.84 ± 12%  perf-profile.children.cycles-pp.run_rebalance_domains
>       0.00            +0.9        0.89 ± 15%  perf-profile.children.cycles-pp.cpuidle_governor_latency_req
>       0.00            +1.0        0.98 ±  8%  perf-profile.children.cycles-pp.update_sg_lb_stats
>       0.00            +1.0        0.98 ± 24%  perf-profile.children.cycles-pp.update_rq_clock_task
>       0.00            +1.1        1.08 ±  9%  perf-profile.children.cycles-pp.irqtime_account_irq
>       0.00            +1.2        1.16 ± 17%  perf-profile.children.cycles-pp.tick_nohz_next_event
>       0.00            +1.2        1.17 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>       0.00            +1.3        1.25 ±  2%  perf-profile.children.cycles-pp.update_sd_lb_stats
>       0.00            +1.3        1.31 ±  2%  perf-profile.children.cycles-pp.find_busiest_group
>       0.00            +1.4        1.35 ± 28%  perf-profile.children.cycles-pp.tick_sched_do_timer
>       0.00            +1.5        1.46 ± 48%  perf-profile.children.cycles-pp.tick_irq_enter
>       0.00            +1.5        1.46 ±  3%  perf-profile.children.cycles-pp.sched_clock_cpu
>       0.00            +1.5        1.51 ± 12%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
>       0.00            +1.5        1.53 ± 42%  perf-profile.children.cycles-pp.irq_enter_rcu
>       0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.x86_64_start_kernel
>       0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.x86_64_start_reservations
>       0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.start_kernel
>       0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.arch_call_rest_init
>       0.00            +1.6        1.62 ± 12%  perf-profile.children.cycles-pp.rest_init
>       0.00            +1.7        1.70 ±  7%  perf-profile.children.cycles-pp.native_sched_clock
>       0.00            +1.7        1.70 ±  3%  perf-profile.children.cycles-pp.load_balance
>       0.00            +1.7        1.71 ±  5%  perf-profile.children.cycles-pp.lapic_next_deadline
>       0.00            +1.7        1.72 ±  8%  perf-profile.children.cycles-pp.kthread
>       0.00            +1.8        1.76 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
>       0.00            +1.9        1.95 ± 16%  perf-profile.children.cycles-pp.read_tsc
>       0.00            +2.0        1.98 ±  9%  perf-profile.children.cycles-pp.perf_rotate_context
>       0.00            +2.0        2.04 ± 11%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
>       0.00            +2.3        2.30 ±  5%  perf-profile.children.cycles-pp.clockevents_program_event
>       0.00            +2.6        2.60 ±  6%  perf-profile.children.cycles-pp.rebalance_domains
>       0.00            +3.0        2.99 ±  7%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
>       0.00            +3.2        3.18 ±  7%  perf-profile.children.cycles-pp.arch_scale_freq_tick
>       0.00            +3.9        3.92 ± 31%  perf-profile.children.cycles-pp.ktime_get
>       0.00            +4.5        4.53 ±  6%  perf-profile.children.cycles-pp.menu_select
>       0.00            +4.7        4.71 ±  6%  perf-profile.children.cycles-pp.__do_softirq
>       0.88 ±223%      +5.0        5.86 ± 11%  perf-profile.children.cycles-pp.scheduler_tick
>       0.00            +5.6        5.64 ±  4%  perf-profile.children.cycles-pp.__irq_exit_rcu
>       2.36 ±100%      +5.8        8.15 ±  9%  perf-profile.children.cycles-pp.update_process_times
>       2.36 ±100%      +6.0        8.35 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
>       2.36 ±100%      +8.8       11.21 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
>       2.36 ±100%     +13.9       16.30 ±  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
>       3.24 ±117%     +19.3       22.55 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
>       3.24 ±117%     +19.4       22.66 ±  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
>       3.24 ±117%     +28.0       31.28 ±  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
>       3.24 ±117%     +32.7       35.97 ±  2%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
>       0.00           +42.2       42.16        perf-profile.children.cycles-pp.intel_idle
>       0.88 ±223%     +83.5       84.34        perf-profile.children.cycles-pp.cpuidle_enter_state
>       0.88 ±223%     +83.9       84.78        perf-profile.children.cycles-pp.cpuidle_enter
>       2.48 ±158%     +87.9       90.38        perf-profile.children.cycles-pp.start_secondary
>       2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
>       2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.cpu_startup_entry
>       2.48 ±158%     +89.5       92.00        perf-profile.children.cycles-pp.do_idle
>       0.88 ±223%     +89.8       90.65        perf-profile.children.cycles-pp.cpuidle_idle_call
>       0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.shmem_get_folio_gfp
>       0.00            +0.1        0.07 ± 19%  perf-profile.self.cycles-pp.tick_irq_enter
>       0.00            +0.1        0.07 ± 26%  perf-profile.self.cycles-pp.check_move_unevictable_folios
>       0.00            +0.1        0.07 ± 33%  perf-profile.self.cycles-pp.perf_event_task_tick
>       0.00            +0.1        0.08 ± 19%  perf-profile.self.cycles-pp.hrtimer_run_queues
>       0.00            +0.1        0.08 ± 14%  perf-profile.self.cycles-pp.irqentry_exit
>       0.00            +0.1        0.08 ± 21%  perf-profile.self.cycles-pp.hrtimer_get_next_event
>       0.00            +0.1        0.08 ± 26%  perf-profile.self.cycles-pp.can_stop_idle_tick
>       0.00            +0.1        0.09 ± 29%  perf-profile.self.cycles-pp.irqentry_enter
>       0.00            +0.1        0.09 ± 43%  perf-profile.self.cycles-pp.update_blocked_averages
>       0.00            +0.1        0.10 ± 12%  perf-profile.self.cycles-pp.vma_interval_tree_insert
>       0.00            +0.1        0.10 ± 34%  perf-profile.self.cycles-pp.irq_work_run_list
>       0.00            +0.1        0.10 ± 43%  perf-profile.self.cycles-pp.ct_kernel_exit
>       0.00            +0.1        0.10 ± 41%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
>       0.00            +0.1        0.11 ± 13%  perf-profile.self.cycles-pp.tick_sched_timer
>       0.00            +0.1        0.11 ± 42%  perf-profile.self.cycles-pp.filemap_map_pages
>       0.00            +0.1        0.12 ± 26%  perf-profile.self.cycles-pp.rb_erase
>       0.00            +0.1        0.12 ± 56%  perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
>       0.00            +0.1        0.12 ± 13%  perf-profile.self.cycles-pp.__switch_to_asm
>       0.00            +0.1        0.13 ± 21%  perf-profile.self.cycles-pp.hrtimer_update_next_event
>       0.00            +0.1        0.13 ± 36%  perf-profile.self.cycles-pp.hrtimer_forward
>       0.00            +0.1        0.13 ± 25%  perf-profile.self.cycles-pp.rb_next
>       0.00            +0.1        0.13 ± 42%  perf-profile.self.cycles-pp.update_rq_clock
>       0.00            +0.1        0.14 ± 34%  perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
>       0.00            +0.1        0.14 ± 23%  perf-profile.self.cycles-pp.enqueue_hrtimer
>       0.00            +0.1        0.14 ± 32%  perf-profile.self.cycles-pp.tick_nohz_get_sleep_length
>       0.00            +0.1        0.14 ± 15%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
>       0.00            +0.1        0.15 ± 40%  perf-profile.self.cycles-pp.clockevents_program_event
>       0.00            +0.1        0.15 ± 23%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
>       0.00            +0.2        0.15 ± 21%  perf-profile.self.cycles-pp.___perf_sw_event
>       0.00            +0.2        0.16 ± 14%  perf-profile.self.cycles-pp.irqtime_account_process_tick
>       0.00            +0.2        0.16 ± 19%  perf-profile.self.cycles-pp._find_next_and_bit
>       0.00            +0.2        0.16 ± 23%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
>       0.00            +0.2        0.16 ± 25%  perf-profile.self.cycles-pp.update_rt_rq_load_avg
>       0.00            +0.2        0.16 ± 19%  perf-profile.self.cycles-pp.intel_idle_irq
>       0.00            +0.2        0.16 ± 17%  perf-profile.self.cycles-pp.hrtimer_next_event_without
>       0.00            +0.2        0.18 ± 45%  perf-profile.self.cycles-pp.timerqueue_del
>       0.00            +0.2        0.18 ± 27%  perf-profile.self.cycles-pp.check_cpu_stall
>       0.00            +0.2        0.18 ± 36%  perf-profile.self.cycles-pp.update_sd_lb_stats
>       0.00            +0.2        0.19 ± 44%  perf-profile.self.cycles-pp.run_posix_cpu_timers
>       0.00            +0.2        0.19 ± 18%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
>       0.00            +0.2        0.20 ± 47%  perf-profile.self.cycles-pp.__do_softirq
>       0.00            +0.2        0.20 ± 23%  perf-profile.self.cycles-pp.tick_program_event
>       0.00            +0.2        0.20 ± 26%  perf-profile.self.cycles-pp.rb_insert_color
>       0.00            +0.2        0.20 ±  8%  perf-profile.self.cycles-pp.sched_clock_cpu
>       0.00            +0.2        0.21 ± 41%  perf-profile.self.cycles-pp.get_next_timer_interrupt
>       0.00            +0.2        0.21 ± 60%  perf-profile.self.cycles-pp.ct_kernel_enter
>       0.00            +0.2        0.22 ±102%  perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
>       0.00            +0.2        0.25 ± 26%  perf-profile.self.cycles-pp.update_rq_clock_task
>       0.00            +0.2        0.25 ± 26%  perf-profile.self.cycles-pp.__irq_exit_rcu
>       0.00            +0.3        0.26 ± 38%  perf-profile.self.cycles-pp.rebalance_domains
>       0.00            +0.3        0.26 ± 33%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
>       0.00            +0.3        0.26 ± 25%  perf-profile.self.cycles-pp.perf_rotate_context
>       0.00            +0.3        0.28 ± 20%  perf-profile.self.cycles-pp.tick_nohz_next_event
>       0.00            +0.3        0.29 ±  9%  perf-profile.self.cycles-pp.error_entry
>       0.00            +0.3        0.29 ± 38%  perf-profile.self.cycles-pp.hrtimer_interrupt
>       0.00            +0.3        0.30 ± 18%  perf-profile.self.cycles-pp.call_cpuidle
>       0.00            +0.3        0.30 ± 16%  perf-profile.self.cycles-pp.load_balance
>       0.00            +0.3        0.31 ± 22%  perf-profile.self.cycles-pp.ct_kernel_exit_state
>       0.00            +0.3        0.32 ± 20%  perf-profile.self.cycles-pp.irqtime_account_irq
>       0.00            +0.3        0.32 ± 31%  perf-profile.self.cycles-pp.timerqueue_add
>       0.00            +0.3        0.32 ± 28%  perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
>       0.00            +0.4        0.35 ± 20%  perf-profile.self.cycles-pp.scheduler_tick
>       0.00            +0.4        0.36 ± 16%  perf-profile.self.cycles-pp.idle_cpu
>       0.00            +0.4        0.38 ± 17%  perf-profile.self.cycles-pp.irq_work_tick
>       0.00            +0.4        0.39 ± 20%  perf-profile.self.cycles-pp.cpuidle_enter
>       0.00            +0.4        0.40 ± 18%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
>       0.00            +0.4        0.45 ± 36%  perf-profile.self.cycles-pp.do_idle
>       0.00            +0.5        0.50 ±  7%  perf-profile.self.cycles-pp._raw_spin_lock_irq
>       0.00            +0.5        0.51 ± 18%  perf-profile.self.cycles-pp.__hrtimer_run_queues
>       0.00            +0.5        0.53 ± 17%  perf-profile.self.cycles-pp.get_cpu_device
>       0.00            +0.5        0.54 ±  8%  perf-profile.self.cycles-pp.update_process_times
>       0.00            +0.6        0.56 ± 27%  perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
>       0.00            +0.6        0.62 ± 61%  perf-profile.self.cycles-pp.calc_global_load_tick
>       0.00            +0.7        0.69 ± 16%  perf-profile.self.cycles-pp.cpuidle_idle_call
>       0.00            +0.7        0.73 ± 14%  perf-profile.self.cycles-pp._raw_spin_trylock
>       0.00            +0.7        0.73 ± 29%  perf-profile.self.cycles-pp.update_irq_load_avg
>       0.00            +0.7        0.74 ±  9%  perf-profile.self.cycles-pp.update_sg_lb_stats
>       0.00            +1.1        1.10 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>       0.00            +1.2        1.18 ± 31%  perf-profile.self.cycles-pp.tick_sched_do_timer
>       0.00            +1.4        1.45 ±  9%  perf-profile.self.cycles-pp.menu_select
>       0.00            +1.5        1.51 ± 12%  perf-profile.self.cycles-pp.__intel_pmu_enable_all
>       0.00            +1.6        1.63 ±  9%  perf-profile.self.cycles-pp.native_sched_clock
>       0.00            +1.7        1.70 ±  4%  perf-profile.self.cycles-pp.lapic_next_deadline
>       0.00            +1.9        1.93 ± 16%  perf-profile.self.cycles-pp.read_tsc
>       0.00            +2.3        2.30 ± 52%  perf-profile.self.cycles-pp.ktime_get
>       0.00            +3.2        3.18 ±  7%  perf-profile.self.cycles-pp.arch_scale_freq_tick
>       0.00            +5.8        5.84 ±  9%  perf-profile.self.cycles-pp.cpuidle_enter_state
>       0.00           +42.2       42.16        perf-profile.self.cycles-pp.intel_idle
> 
> 
> ***************************************************************************************************
> lkp-icl-2sp7: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> =========================================================================================
> compiler/cpufreq_governor/debug-setup/disk/filesize/fs2/fs/iterations/kconfig/nr_directories/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
>   gcc-12/performance/no-monitor/1SSD/16MB/nfsv4/btrfs/1x/x86_64-rhel-8.3/16d/256fpd/32t/debian-11.1-x86_64-20220510.cgz/NoSync/lkp-icl-2sp7/20G/fsmark
> 
> commit: 
>   3434d7aa77 ("NFSD: Clean up nfsctl_transaction_write()")
>   39d432fc76 ("NFSD: trace nfsctl operations")
> 
> 3434d7aa77d24c5c 39d432fc76301cf0a0c45402211 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>    7488368 ±  2%   +7979.7%   6.05e+08 ± 47%  fsmark.app_overhead
>      62.42 ±  2%    -100.0%       0.00        fsmark.files_per_sec
>      22.46 ±  2%  +15766.1%       3564 ±  5%  fsmark.time.elapsed_time
>      22.46 ±  2%  +15766.1%       3564 ±  5%  fsmark.time.elapsed_time.max
>     192.33           -99.5%       1.00        fsmark.time.percent_of_cpu_this_job_got
>      43.05           +13.7%      48.94 ±  4%  fsmark.time.system_time
>     978884            +2.9%    1007574 ±  2%  fsmark.time.voluntary_context_switches
> 
> 
> 
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
> 
> 
> -- 
> 0-DAY CI Kernel Test Service
> https://github.com/intel/lkp-tests/wiki
> 

-- 
Chuck Lever