Re: [LKP] [xfs] a5949d3fae: aim7.jobs-per-min -33.6% regression

From: Xing Zhengjun
Date: Wed Jul 01 2020 - 04:33:10 EST




On 6/10/2020 11:07 AM, Xing Zhengjun wrote:
Hi Darrick,

   Do you have time to take a look at this? Thanks.



Ping...


On 6/6/2020 11:48 PM, kernel test robot wrote:
Greeting,

FYI, we noticed a -33.6% regression of aim7.jobs-per-min due to commit:


commit: a5949d3faedf492fa7863b914da408047ab46eb0 ("xfs: force writes to delalloc regions to unwritten")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
with following parameters:

    disk: 1BRD_48G
    fs: xfs
    test: sync_disk_rw
    load: 600
    cpufreq_governor: performance
    ucode: 0x42e

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@xxxxxxxxx>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

         git clone https://github.com/intel/lkp-tests.git
         cd lkp-tests
         bin/lkp install job.yaml  # job file is attached in this email
         bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-7.6/600/debian-x86_64-20191114.cgz/lkp-ivb-2ep1/sync_disk_rw/aim7/0x42e

commit:
   590b16516e ("xfs: refactor xfs_iomap_prealloc_size")
   a5949d3fae ("xfs: force writes to delalloc regions to unwritten")

590b16516ef38e2e a5949d3faedf492fa7863b914da
---------------- ---------------------------
        fail:runs  %reproduction    fail:runs
            |             |             |
            :4           50%           2:4 dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
          %stddev     %change         %stddev
              \          |                \
      35272           -33.6%      23430        aim7.jobs-per-min
     102.13           +50.5%     153.75        aim7.time.elapsed_time
     102.13           +50.5%     153.75        aim7.time.elapsed_time.max
    1388038           +40.2%    1945838 aim7.time.involuntary_context_switches
      43420 ±  2%     +13.4%      49255 ±  2% aim7.time.minor_page_faults
       3123           +44.2%       4504 ±  2%  aim7.time.system_time
      59.31            +6.5%      63.18        aim7.time.user_time
   48595108           +58.6%   77064959 aim7.time.voluntary_context_switches
       1.44           -28.8%       1.02        iostat.cpu.user
       0.07 ±  6%      +0.4        0.44 ±  7%  mpstat.cpu.all.iowait%
       1.44            -0.4        1.02        mpstat.cpu.all.usr%
       8632 ± 50%     +75.6%      15156 ± 34% numa-meminfo.node0.KernelStack
       6583 ±136%    +106.0%      13562 ± 82% numa-meminfo.node0.PageTables
      63325 ± 11%     +14.3%      72352 ± 12% numa-meminfo.node0.SUnreclaim
       8647 ± 50%     +75.3%      15156 ± 34% numa-vmstat.node0.nr_kernel_stack
       1656 ±136%    +104.6%       3389 ± 82% numa-vmstat.node0.nr_page_table_pages
      15831 ± 11%     +14.3%      18087 ± 12% numa-vmstat.node0.nr_slab_unreclaimable
      93640 ±  3%     +41.2%     132211 ±  2%  meminfo.AnonHugePages
      21641           +39.9%      30271 ±  4%  meminfo.KernelStack
     129269           +12.3%     145114        meminfo.SUnreclaim
      28000           -31.2%      19275        meminfo.max_used_kB
    1269307           -26.9%     927657        vmstat.io.bo
     149.75 ±  3%     -17.4%     123.75 ±  4%  vmstat.procs.r
     718992           +13.3%     814567        vmstat.system.cs
     231397            -9.3%     209881 ±  2%  vmstat.system.in
  6.774e+08           +70.0%  1.152e+09        cpuidle.C1.time
   18203372           +60.4%   29198744        cpuidle.C1.usage
  2.569e+08 ± 18%     +81.8%  4.672e+08 ±  5%  cpuidle.C1E.time
    2691402 ± 13%     +98.7%    5346901 ±  3%  cpuidle.C1E.usage
     990350           +95.0%    1931226 ±  2%  cpuidle.POLL.time
     520061           +97.7%    1028004 ±  2%  cpuidle.POLL.usage
      77231            +1.8%      78602        proc-vmstat.nr_active_anon
      19868            +3.8%      20615        proc-vmstat.nr_dirty
     381302            +1.0%     384969        proc-vmstat.nr_file_pages
       4388            -2.7%       4270 proc-vmstat.nr_inactive_anon
      69865            +4.7%      73155 proc-vmstat.nr_inactive_file
      21615           +40.0%      30251 ±  4% proc-vmstat.nr_kernel_stack
       7363            -3.2%       7127        proc-vmstat.nr_mapped
      12595 ±  3%      +5.2%      13255 ±  4%  proc-vmstat.nr_shmem
      19619            +3.2%      20247 proc-vmstat.nr_slab_reclaimable
      32316           +12.3%      36280 proc-vmstat.nr_slab_unreclaimable
      77231            +1.8%      78602 proc-vmstat.nr_zone_active_anon
       4388            -2.7%       4270 proc-vmstat.nr_zone_inactive_anon
      69865            +4.7%      73155 proc-vmstat.nr_zone_inactive_file
      19165            +3.4%      19824 proc-vmstat.nr_zone_write_pending
       9845 ± 27%     +68.0%      16539 ± 37% proc-vmstat.numa_hint_faults_local
    8195135            +2.5%    8402537        proc-vmstat.numa_hit
    8179414            +2.5%    8386209        proc-vmstat.numa_local
      15720            +3.9%      16328        proc-vmstat.numa_other
    8281098            +2.4%    8477157        proc-vmstat.pgalloc_normal
     342925 ±  2%     +39.8%     479279        proc-vmstat.pgfault
    8132789            +2.0%    8298189        proc-vmstat.pgfree
  1.327e+08            +9.7%  1.455e+08        proc-vmstat.pgpgout
     394.00          +151.7%     991.75 ±  5% slabinfo.biovec-max.active_objs
     394.00          +152.1%     993.25 ±  5% slabinfo.biovec-max.num_objs
      21028 ±  6%      -7.8%      19389 ±  5%  slabinfo.filp.active_objs
      21520 ±  5%      -7.6%      19886 ±  5%  slabinfo.filp.num_objs
       4955          +234.9%      16594 ±  2% slabinfo.kmalloc-256.active_objs
      77.25          +238.5%     261.50 ±  2% slabinfo.kmalloc-256.active_slabs
       4955          +238.4%      16768 ±  2% slabinfo.kmalloc-256.num_objs
      77.25          +238.5%     261.50 ±  2% slabinfo.kmalloc-256.num_slabs
       3368 ±  2%     +22.6%       4128 ±  2% slabinfo.kmalloc-2k.active_objs
       3446 ±  2%     +21.9%       4200 ±  2% slabinfo.kmalloc-2k.num_objs
      11815 ±  8%     +29.0%      15239 ±  3% slabinfo.kmalloc-512.active_objs
      11824 ±  9%     +29.0%      15249 ±  3% slabinfo.kmalloc-512.num_objs
       6849 ±  5%      +9.0%       7463 ±  2% slabinfo.kmalloc-96.active_objs
       6906 ±  5%      +8.9%       7521 ±  2% slabinfo.kmalloc-96.num_objs
     765.00 ±  4%     +25.0%     956.25 ±  5% slabinfo.mnt_cache.active_objs
     765.00 ±  4%     +25.0%     956.25 ±  5% slabinfo.mnt_cache.num_objs
      13866           +12.5%      15594 slabinfo.proc_inode_cache.active_objs
      13905           +12.2%      15598 slabinfo.proc_inode_cache.num_objs
       1954           +24.4%       2431 ±  3% slabinfo.sighand_cache.active_objs
       1954           +24.4%       2431 ±  3% slabinfo.sighand_cache.num_objs
       2596           +13.7%       2951 ±  2% slabinfo.signal_cache.active_objs
       2596           +13.7%       2951 ±  2% slabinfo.signal_cache.num_objs
       4368 ±  6%     -11.7%       3856 ±  2% slabinfo.skbuff_head_cache.num_objs
     822.25 ± 11%     +21.4%     998.50 ±  3% slabinfo.task_group.active_objs
     822.25 ± 11%     +21.4%     998.50 ±  3% slabinfo.task_group.num_objs
       1612           +32.1%       2129 ±  3% slabinfo.task_struct.active_objs
     540.50           +31.6%     711.25 ±  3% slabinfo.task_struct.active_slabs
       1622           +31.6%       2134 ±  3% slabinfo.task_struct.num_objs
     540.50           +31.6%     711.25 ±  3% slabinfo.task_struct.num_slabs
       6905           +20.0%       8289 slabinfo.vmap_area.active_objs
       6933           +20.0%       8322 slabinfo.vmap_area.num_objs
     646.25 ±  7%    +195.2%       1907 slabinfo.xfs_buf.active_objs
     646.25 ±  7%    +195.2%       1907        slabinfo.xfs_buf.num_objs
       1.78 ±  2%      -0.3        1.51 ±  2% perf-stat.i.branch-miss-rate%
   64079048            +3.9%   66551536        perf-stat.i.branch-misses
      17.99            -4.2       13.84 perf-stat.i.cache-miss-rate%
   66630506           -16.5%   55647472        perf-stat.i.cache-misses
  3.899e+08            +9.4%  4.265e+08 perf-stat.i.cache-references
     731029           +13.1%     826756 perf-stat.i.context-switches
       3.79            -1.7%       3.73        perf-stat.i.cpi
     113320            +8.3%     122693        perf-stat.i.cpu-migrations
       1498           +20.4%       1803 perf-stat.i.cycles-between-cache-misses
   50450547 ±  4%      -8.0%   46402743 ±  5% perf-stat.i.dTLB-load-misses
       0.20 ±  3%      -0.0        0.16 ±  2% perf-stat.i.dTLB-store-miss-rate%
    4587096 ±  3%     -22.9%    3538422 ±  2% perf-stat.i.dTLB-store-misses
  2.297e+09            -5.6%  2.168e+09        perf-stat.i.dTLB-stores
      86.68            -2.4       84.25 perf-stat.i.iTLB-load-miss-rate%
   19484791            -6.1%   18297125 perf-stat.i.iTLB-load-misses
    3060023           +14.4%    3501587 ±  4%  perf-stat.i.iTLB-loads
       0.29            -1.5%       0.28        perf-stat.i.ipc
       0.21 ±  5%     -20.5%       0.17 ±  6%  perf-stat.i.metric.K/sec
       3120            -5.8%       2938        perf-stat.i.minor-faults
      42.61            +1.2       43.83 perf-stat.i.node-load-miss-rate%
   24887573           -10.1%   22384693 perf-stat.i.node-load-misses
   31814752           -12.5%   27830232        perf-stat.i.node-loads
      36.62            +0.8       37.45 perf-stat.i.node-store-miss-rate%
   22617213           -20.3%   18015607 perf-stat.i.node-store-misses
   35707079           -19.9%   28604167        perf-stat.i.node-stores
       3120            -5.8%       2938        perf-stat.i.page-faults
      13.96            +9.3%      15.25        perf-stat.overall.MPKI
       0.99            +0.0        1.03 perf-stat.overall.branch-miss-rate%
      17.09            -4.0       13.05 perf-stat.overall.cache-miss-rate%
       1555           +19.4%       1856 perf-stat.overall.cycles-between-cache-misses
       0.69 ±  4%      -0.1        0.64 ±  5% perf-stat.overall.dTLB-load-miss-rate%
       0.20 ±  3%      -0.0        0.16 ±  2% perf-stat.overall.dTLB-store-miss-rate%
      86.43            -2.5       83.94 perf-stat.overall.iTLB-load-miss-rate%
       1433            +6.7%       1528 perf-stat.overall.instructions-per-iTLB-miss
      43.89            +0.7       44.58 perf-stat.overall.node-load-miss-rate%
  6.389e+09            +0.8%  6.438e+09 perf-stat.ps.branch-instructions
   63452800            +4.2%   66110454        perf-stat.ps.branch-misses
   65977234           -16.2%   55277568        perf-stat.ps.cache-misses
  3.861e+08            +9.8%  4.237e+08 perf-stat.ps.cache-references
     723870           +13.5%     821265 perf-stat.ps.context-switches
     112209            +8.6%     121881 perf-stat.ps.cpu-migrations
   49955941 ±  4%      -7.7%   46094243 ±  5% perf-stat.ps.dTLB-load-misses
    4542094 ±  3%     -22.6%    3514899 ±  2% perf-stat.ps.dTLB-store-misses
  2.275e+09            -5.3%  2.154e+09        perf-stat.ps.dTLB-stores
   19294019            -5.8%   18175580 perf-stat.ps.iTLB-load-misses
    3030049           +14.8%    3478314 ±  4%  perf-stat.ps.iTLB-loads
       3089            -5.5%       2918        perf-stat.ps.minor-faults
   24643655            -9.8%   22235787 perf-stat.ps.node-load-misses
   31502901           -12.2%   27645143        perf-stat.ps.node-loads
   22395376           -20.1%   17895417 perf-stat.ps.node-store-misses
   35356897           -19.6%   28413366        perf-stat.ps.node-stores
       3089            -5.5%       2918        perf-stat.ps.page-faults
   2.85e+12           +50.6%  4.293e+12 perf-stat.total.instructions
      84193 ± 36%    +135.9%     198575 ± 25% sched_debug.cfs_rq:/.MIN_vruntime.avg
     919053           +98.2%    1821460 ±  2% sched_debug.cfs_rq:/.MIN_vruntime.max
     256686 ± 16%    +114.7%     551162 ± 13% sched_debug.cfs_rq:/.MIN_vruntime.stddev
      19497          +106.7%      40307 sched_debug.cfs_rq:/.exec_clock.avg
      21240          +101.1%      42717 sched_debug.cfs_rq:/.exec_clock.max
      19079          +107.3%      39542 sched_debug.cfs_rq:/.exec_clock.min
     394.52 ±  2%     +54.3%     608.79 ± 13% sched_debug.cfs_rq:/.exec_clock.stddev
     965.25 ± 15%     +59.7%       1541 ±  4% sched_debug.cfs_rq:/.load_avg.avg
       2866 ± 21%     +57.1%       4503 ± 10% sched_debug.cfs_rq:/.load_avg.max
     659.77 ± 15%     +68.9%       1114 ±  9% sched_debug.cfs_rq:/.load_avg.stddev
      84193 ± 36%    +135.9%     198575 ± 25% sched_debug.cfs_rq:/.max_vruntime.avg
     919053           +98.2%    1821462 ±  2% sched_debug.cfs_rq:/.max_vruntime.max
     256686 ± 16%    +114.7%     551163 ± 13% sched_debug.cfs_rq:/.max_vruntime.stddev
     901765           +97.1%    1777741 ±  2% sched_debug.cfs_rq:/.min_vruntime.avg
     941794           +95.8%    1844445 ±  2% sched_debug.cfs_rq:/.min_vruntime.max
     872925           +96.5%    1715658 ±  2% sched_debug.cfs_rq:/.min_vruntime.min
      18358 ± 44%    +146.6%      45281 ± 34% sched_debug.cfs_rq:/.min_vruntime.stddev
       1.22 ±  8%    +159.9%       3.18 ± 20% sched_debug.cfs_rq:/.nr_spread_over.avg
       6.75 ± 23%     +65.4%      11.17 ± 17% sched_debug.cfs_rq:/.nr_spread_over.max
       1.38 ± 11%     +61.4%       2.22 ± 12% sched_debug.cfs_rq:/.nr_spread_over.stddev
      18357 ± 44%    +146.5%      45250 ± 34% sched_debug.cfs_rq:/.spread0.stddev
     773.25 ±  3%     +13.2%     875.02 sched_debug.cfs_rq:/.util_avg.avg
     285624           -30.3%     199111 ±  4% sched_debug.cpu.avg_idle.avg
     576507 ±  2%     -26.2%     425271 ±  7% sched_debug.cpu.avg_idle.max
      51029 ± 68%     -83.4%       8469 ± 29% sched_debug.cpu.avg_idle.min
     135149 ±  7%     -23.4%     103460 ± 10% sched_debug.cpu.avg_idle.stddev
      57492           +51.0%      86835        sched_debug.cpu.clock.avg
      57495           +51.0%      86841        sched_debug.cpu.clock.max
      57487           +51.0%      86828        sched_debug.cpu.clock.min
       1.98 ±  4%     +74.8%       3.46 ± 10% sched_debug.cpu.clock.stddev
      57492           +51.0%      86835 sched_debug.cpu.clock_task.avg
      57495           +51.0%      86841 sched_debug.cpu.clock_task.max
      57487           +51.0%      86828 sched_debug.cpu.clock_task.min
       1.98 ±  4%     +74.8%       3.46 ± 10% sched_debug.cpu.clock_task.stddev
       2831           +43.0%       4049 sched_debug.cpu.curr->pid.max
     417519          +133.3%     974078 sched_debug.cpu.nr_switches.avg
     427854          +134.0%    1001016 sched_debug.cpu.nr_switches.max
     409942          +131.8%     950218 sched_debug.cpu.nr_switches.min
       4770 ± 42%    +238.7%      16156 ± 22% sched_debug.cpu.nr_switches.stddev
       4.70 ± 23%     +37.1%       6.44 ±  4% sched_debug.cpu.nr_uninterruptible.avg
     776.62 ± 19%     +86.1%       1445 ± 12% sched_debug.cpu.nr_uninterruptible.max
    -472.62           +98.3%    -937.33 sched_debug.cpu.nr_uninterruptible.min
     289.80 ±  5%    +109.7%     607.68 ±  3% sched_debug.cpu.nr_uninterruptible.stddev
     415893          +133.8%     972526 sched_debug.cpu.sched_count.avg
     423960          +135.5%     998457 sched_debug.cpu.sched_count.max
     409113          +132.0%     949068 sched_debug.cpu.sched_count.min
       4194 ± 50%    +274.3%      15698 ± 22% sched_debug.cpu.sched_count.stddev
     110094          +125.8%     248571 sched_debug.cpu.sched_goidle.avg
     112236          +125.6%     253163 sched_debug.cpu.sched_goidle.max
     108281          +125.8%     244506 sched_debug.cpu.sched_goidle.min
     772.04 ± 16%    +175.8%       2129 ± 18% sched_debug.cpu.sched_goidle.stddev
     297267          +137.2%     705173 sched_debug.cpu.ttwu_count.avg
     320112          +135.9%     755096 sched_debug.cpu.ttwu_count.max
     272487 ±  2%    +142.2%     660099 sched_debug.cpu.ttwu_count.min
      10662 ± 10%    +110.9%      22491 ± 18% sched_debug.cpu.ttwu_count.stddev
      12295 ±  2%    +834.9%     114954 ±  2% sched_debug.cpu.ttwu_local.avg
      13726 ±  2%    +769.7%     119376 ±  2% sched_debug.cpu.ttwu_local.max
      10620 ±  4%    +942.4%     110710 sched_debug.cpu.ttwu_local.min
     693.29 ± 14%    +228.7%       2278 ± 18% sched_debug.cpu.ttwu_local.stddev
      57488           +51.0%      86829        sched_debug.cpu_clk
      56998           +51.5%      86337        sched_debug.ktime
      57977           +50.6%      87317        sched_debug.sched_clk
      72.50 ± 37%     +55.5%     112.75 ± 26% interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
      53.25 ±  2%     +84.5%      98.25 ± 22% interrupts.37:PCI-MSI.2621443-edge.eth0-TxRx-2
      73.50 ± 26%    +116.3%     159.00 ± 60% interrupts.38:PCI-MSI.2621444-edge.eth0-TxRx-3
     204541           +49.0%     304702 ±  2% interrupts.CPU0.LOC:Local_timer_interrupts
     290268 ±  2%     +29.1%     374641 ±  4% interrupts.CPU0.RES:Rescheduling_interrupts
     204580           +48.9%     304667 ±  2% interrupts.CPU1.LOC:Local_timer_interrupts
     290342 ±  2%     +28.5%     373110 ±  4% interrupts.CPU1.RES:Rescheduling_interrupts
     204537           +48.9%     304619 ±  2% interrupts.CPU10.LOC:Local_timer_interrupts
     291605 ±  2%     +28.3%     374190 ±  4% interrupts.CPU10.RES:Rescheduling_interrupts
     204140           +49.2%     304587 ±  2% interrupts.CPU11.LOC:Local_timer_interrupts
     291988 ±  2%     +28.1%     374024 ±  4% interrupts.CPU11.RES:Rescheduling_interrupts
     204367           +49.0%     304559 ±  2% interrupts.CPU12.LOC:Local_timer_interrupts
     288632           +28.4%     370669 ±  3% interrupts.CPU12.RES:Rescheduling_interrupts
     204555           +48.9%     304669 ±  2% interrupts.CPU13.LOC:Local_timer_interrupts
     289354 ±  2%     +28.5%     371782 ±  3% interrupts.CPU13.RES:Rescheduling_interrupts
     204536           +49.0%     304726 ±  2% interrupts.CPU14.LOC:Local_timer_interrupts
       4937 ± 25%     +41.6%       6991 ±  2% interrupts.CPU14.NMI:Non-maskable_interrupts
       4937 ± 25%     +41.6%       6991 ±  2% interrupts.CPU14.PMI:Performance_monitoring_interrupts
     287759           +28.2%     368986 ±  3% interrupts.CPU14.RES:Rescheduling_interrupts
     204527           +49.0%     304707 ±  2% interrupts.CPU15.LOC:Local_timer_interrupts
     286950           +28.6%     369104 ±  3% interrupts.CPU15.RES:Rescheduling_interrupts
     204556           +49.0%     304718 ±  2% interrupts.CPU16.LOC:Local_timer_interrupts
       5444 ± 32%     -19.9%       4360 ± 32% interrupts.CPU16.NMI:Non-maskable_interrupts
       5444 ± 32%     -19.9%       4360 ± 32% interrupts.CPU16.PMI:Performance_monitoring_interrupts
     287623           +28.1%     368583 ±  3% interrupts.CPU16.RES:Rescheduling_interrupts
     204528           +49.0%     304728 ±  2% interrupts.CPU17.LOC:Local_timer_interrupts
     287734           +28.2%     368920 ±  3% interrupts.CPU17.RES:Rescheduling_interrupts
     204465           +49.0%     304707 ±  2% interrupts.CPU18.LOC:Local_timer_interrupts
       5382 ± 31%     -21.8%       4211 ± 27% interrupts.CPU18.NMI:Non-maskable_interrupts
       5382 ± 31%     -21.8%       4211 ± 27% interrupts.CPU18.PMI:Performance_monitoring_interrupts
     288778           +28.1%     369871 ±  3% interrupts.CPU18.RES:Rescheduling_interrupts
     204302           +49.2%     304727 ±  2% interrupts.CPU19.LOC:Local_timer_interrupts
     289794           +27.8%     370247 ±  3% interrupts.CPU19.RES:Rescheduling_interrupts
     204552           +49.0%     304821 ±  2% interrupts.CPU2.LOC:Local_timer_interrupts
     291056 ±  2%     +28.3%     373290 ±  4% interrupts.CPU2.RES:Rescheduling_interrupts
     204468           +48.9%     304484 ±  2% interrupts.CPU20.LOC:Local_timer_interrupts
     290778           +27.6%     371106 ±  3% interrupts.CPU20.RES:Rescheduling_interrupts
     204546           +48.9%     304619 ±  2% interrupts.CPU21.LOC:Local_timer_interrupts
     289974           +28.0%     371074 ±  3% interrupts.CPU21.RES:Rescheduling_interrupts
     204694           +48.8%     304581 ±  2% interrupts.CPU22.LOC:Local_timer_interrupts
     290312           +28.0%     371734 ±  3% interrupts.CPU22.RES:Rescheduling_interrupts
     204374           +49.0%     304493 ±  2% interrupts.CPU23.LOC:Local_timer_interrupts
     290266           +28.5%     372865 ±  3% interrupts.CPU23.RES:Rescheduling_interrupts
     204518           +49.0%     304653 ±  2% interrupts.CPU24.LOC:Local_timer_interrupts
     288815 ±  2%     +28.7%     371621 ±  4% interrupts.CPU24.RES:Rescheduling_interrupts
     204414           +49.0%     304640 ±  2% interrupts.CPU25.LOC:Local_timer_interrupts
     288779 ±  2%     +28.2%     370202 ±  4% interrupts.CPU25.RES:Rescheduling_interrupts
     204536           +49.0%     304730 ±  2% interrupts.CPU26.LOC:Local_timer_interrupts
     287615 ±  2%     +28.6%     369843 ±  4% interrupts.CPU26.RES:Rescheduling_interrupts
      72.50 ± 37%     +55.5%     112.75 ± 26% interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
     204514           +49.0%     304668 ±  2% interrupts.CPU27.LOC:Local_timer_interrupts
     287014 ±  2%     +28.5%     368825 ±  4% interrupts.CPU27.RES:Rescheduling_interrupts
      53.25 ±  2%     +84.5%      98.25 ± 22% interrupts.CPU28.37:PCI-MSI.2621443-edge.eth0-TxRx-2
     204331           +49.1%     304759 ±  2% interrupts.CPU28.LOC:Local_timer_interrupts
     287278 ±  2%     +28.4%     368876 ±  4% interrupts.CPU28.RES:Rescheduling_interrupts
      73.50 ± 26%    +116.3%     159.00 ± 60% interrupts.CPU29.38:PCI-MSI.2621444-edge.eth0-TxRx-3
     204427           +49.0%     304651 ±  2% interrupts.CPU29.LOC:Local_timer_interrupts
     287382 ±  2%     +28.9%     370434 ±  4% interrupts.CPU29.RES:Rescheduling_interrupts
     204538           +49.0%     304765 ±  2% interrupts.CPU3.LOC:Local_timer_interrupts
       5034 ± 27%     -29.4%       3552 interrupts.CPU3.NMI:Non-maskable_interrupts
       5034 ± 27%     -29.4%       3552 interrupts.CPU3.PMI:Performance_monitoring_interrupts
     288881 ±  2%     +28.3%     370740 ±  4% interrupts.CPU3.RES:Rescheduling_interrupts
     204750           +48.8%     304674 ±  2% interrupts.CPU30.LOC:Local_timer_interrupts
     287139 ±  2%     +28.8%     369825 ±  4% interrupts.CPU30.RES:Rescheduling_interrupts
     204429           +48.9%     304416 ±  2% interrupts.CPU31.LOC:Local_timer_interrupts
     288411 ±  2%     +28.4%     370199 ±  4% interrupts.CPU31.RES:Rescheduling_interrupts
     204395           +49.3%     305106 ±  2% interrupts.CPU32.LOC:Local_timer_interrupts
     289178 ±  2%     +28.6%     371830 ±  4% interrupts.CPU32.RES:Rescheduling_interrupts
     204347           +49.2%     304799 ±  2% interrupts.CPU33.LOC:Local_timer_interrupts
     289689 ±  2%     +28.2%     371280 ±  4% interrupts.CPU33.RES:Rescheduling_interrupts
     204280           +49.1%     304566 ±  2% interrupts.CPU34.LOC:Local_timer_interrupts
       6418 ± 24%     -44.5%       3564 interrupts.CPU34.NMI:Non-maskable_interrupts
       6418 ± 24%     -44.5%       3564 interrupts.CPU34.PMI:Performance_monitoring_interrupts
     290382 ±  2%     +28.3%     372606 ±  4% interrupts.CPU34.RES:Rescheduling_interrupts
     204331           +49.0%     304517 ±  2% interrupts.CPU35.LOC:Local_timer_interrupts
     289736 ±  2%     +28.3%     371869 ±  4% interrupts.CPU35.RES:Rescheduling_interrupts
     204578           +48.8%     304423 ±  2% interrupts.CPU36.LOC:Local_timer_interrupts
       7298           -39.0%       4450 ± 34% interrupts.CPU36.NMI:Non-maskable_interrupts
       7298           -39.0%       4450 ± 34% interrupts.CPU36.PMI:Performance_monitoring_interrupts
     287187           +28.3%     368516 ±  3% interrupts.CPU36.RES:Rescheduling_interrupts
     204538           +49.0%     304729 ±  2% interrupts.CPU37.LOC:Local_timer_interrupts
     288676           +28.1%     369792 ±  3% interrupts.CPU37.RES:Rescheduling_interrupts
     204504           +49.0%     304687 ±  2% interrupts.CPU38.LOC:Local_timer_interrupts
       6301 ± 24%     -29.8%       4424 ± 34% interrupts.CPU38.NMI:Non-maskable_interrupts
       6301 ± 24%     -29.8%       4424 ± 34% interrupts.CPU38.PMI:Performance_monitoring_interrupts
     287510           +28.1%     368349 ±  3% interrupts.CPU38.RES:Rescheduling_interrupts
     204566           +48.9%     304668 ±  2% interrupts.CPU39.LOC:Local_timer_interrupts
     286909           +28.1%     367622 ±  3% interrupts.CPU39.RES:Rescheduling_interrupts
     204445           +49.0%     304686 ±  2% interrupts.CPU4.LOC:Local_timer_interrupts
     288608 ±  2%     +28.7%     371493 ±  3% interrupts.CPU4.RES:Rescheduling_interrupts
     204538           +48.9%     304582 ±  2% interrupts.CPU40.LOC:Local_timer_interrupts
     286219           +27.9%     365983 ±  3% interrupts.CPU40.RES:Rescheduling_interrupts
     204551           +49.0%     304739 ±  2% interrupts.CPU41.LOC:Local_timer_interrupts
       5170 ± 29%     -18.5%       4215 ± 27% interrupts.CPU41.NMI:Non-maskable_interrupts
       5170 ± 29%     -18.5%       4215 ± 27% interrupts.CPU41.PMI:Performance_monitoring_interrupts
     286946           +27.8%     366717 ±  3% interrupts.CPU41.RES:Rescheduling_interrupts
     204492           +48.9%     304442 ±  2% interrupts.CPU42.LOC:Local_timer_interrupts
     287384           +28.0%     367805 ±  3% interrupts.CPU42.RES:Rescheduling_interrupts
     204545           +49.0%     304679 ±  2% interrupts.CPU43.LOC:Local_timer_interrupts
     289100           +27.3%     368059 ±  3% interrupts.CPU43.RES:Rescheduling_interrupts
     204489           +49.0%     304635 ±  2% interrupts.CPU44.LOC:Local_timer_interrupts
       6393 ± 24%     -32.4%       4319 ± 31% interrupts.CPU44.NMI:Non-maskable_interrupts
       6393 ± 24%     -32.4%       4319 ± 31% interrupts.CPU44.PMI:Performance_monitoring_interrupts
     288835           +27.8%     369182 ±  3% interrupts.CPU44.RES:Rescheduling_interrupts
     204473           +49.2%     304983 ±  2% interrupts.CPU45.LOC:Local_timer_interrupts
     289778           +27.4%     369059 ±  3% interrupts.CPU45.RES:Rescheduling_interrupts
     204493           +49.0%     304598 ±  2% interrupts.CPU46.LOC:Local_timer_interrupts
     289485           +27.9%     370172 ±  3% interrupts.CPU46.RES:Rescheduling_interrupts
     204511           +49.2%     305090 ±  2% interrupts.CPU47.LOC:Local_timer_interrupts
     286611           +28.8%     369021 ±  3% interrupts.CPU47.RES:Rescheduling_interrupts
     204441           +49.1%     304794 ±  2% interrupts.CPU5.LOC:Local_timer_interrupts
     289245 ±  2%     +28.4%     371335 ±  4% interrupts.CPU5.RES:Rescheduling_interrupts
     204246           +49.0%     304414 ±  2% interrupts.CPU6.LOC:Local_timer_interrupts
     290006 ±  2%     +28.3%     372180 ±  4% interrupts.CPU6.RES:Rescheduling_interrupts
     204445           +49.0%     304719 ±  2% interrupts.CPU7.LOC:Local_timer_interrupts
     290524 ±  2%     +28.2%     372414 ±  4% interrupts.CPU7.RES:Rescheduling_interrupts
     204178           +49.2%     304663 ±  2% interrupts.CPU8.LOC:Local_timer_interrupts
     291625 ±  2%     +27.8%     372747 ±  4% interrupts.CPU8.RES:Rescheduling_interrupts
     204502           +48.9%     304407 ±  3% interrupts.CPU9.LOC:Local_timer_interrupts
     291615 ±  2%     +27.9%     373024 ±  4% interrupts.CPU9.RES:Rescheduling_interrupts
    9814627           +49.0%   14624016 ±  2% interrupts.LOC:Local_timer_interrupts
   13869791           +28.2%   17785831 ±  3% interrupts.RES:Rescheduling_interrupts
      25728           +47.6%      37969        softirqs.CPU0.RCU
      34944           +41.7%      49503 ±  3%  softirqs.CPU0.SCHED
      42375           +38.5%      58673 ±  2%  softirqs.CPU0.TIMER
      25348           +53.8%      38976 ±  5%  softirqs.CPU1.RCU
      30986           +46.2%      45315 ±  3%  softirqs.CPU1.SCHED
      41265           +44.3%      59538 ±  3%  softirqs.CPU1.TIMER
      25149           +48.3%      37288        softirqs.CPU10.RCU
      29752           +47.1%      43764 ±  2%  softirqs.CPU10.SCHED
      40609           +41.5%      57449        softirqs.CPU10.TIMER
      25218           +48.3%      37399        softirqs.CPU11.RCU
      30294           +44.6%      43800        softirqs.CPU11.SCHED
      41371           +39.2%      57578        softirqs.CPU11.TIMER
      24427           +48.9%      36361        softirqs.CPU12.RCU
      30438           +44.0%      43835        softirqs.CPU12.SCHED
      41045 ±  3%     +43.0%      58687        softirqs.CPU12.TIMER
      24247           +47.0%      35645        softirqs.CPU13.RCU
      30155           +44.5%      43577        softirqs.CPU13.SCHED
      41148 ±  2%     +40.3%      57720        softirqs.CPU13.TIMER
      24506           +46.3%      35856        softirqs.CPU14.RCU
      29991           +44.1%      43229        softirqs.CPU14.SCHED
      41190 ±  2%     +39.7%      57552        softirqs.CPU14.TIMER
      24412           +46.7%      35800        softirqs.CPU15.RCU
      30135           +44.1%      43430        softirqs.CPU15.SCHED
      45081 ± 15%     +28.0%      57684        softirqs.CPU15.TIMER
      25327           +47.0%      37225        softirqs.CPU16.RCU
      30230           +43.8%      43461        softirqs.CPU16.SCHED
      41157           +39.6%      57448        softirqs.CPU16.TIMER
      25136           +47.8%      37142        softirqs.CPU17.RCU
      29894           +44.5%      43212        softirqs.CPU17.SCHED
      41961 ±  5%     +36.5%      57281        softirqs.CPU17.TIMER
      25431           +46.1%      37160        softirqs.CPU18.RCU
      30062           +44.0%      43281        softirqs.CPU18.SCHED
      40956 ±  2%     +42.1%      58217        softirqs.CPU18.TIMER
      25496           +44.8%      36923 ±  2%  softirqs.CPU19.RCU
      29989           +46.5%      43946        softirqs.CPU19.SCHED
      41740 ±  2%     +67.8%      70057 ± 17%  softirqs.CPU19.TIMER
      25414           +48.3%      37678        softirqs.CPU2.RCU
      30905           +43.4%      44309 ±  2%  softirqs.CPU2.SCHED
      41083           +40.6%      57768 ±  2%  softirqs.CPU2.TIMER
      25124           +46.8%      36892        softirqs.CPU20.RCU
      30001           +44.7%      43416        softirqs.CPU20.SCHED
      41060 ±  7%     +39.2%      57151        softirqs.CPU20.TIMER
      24992           +47.6%      36896        softirqs.CPU21.RCU
      30109           +44.4%      43467        softirqs.CPU21.SCHED
      40055 ±  3%     +42.5%      57081 ±  2%  softirqs.CPU21.TIMER
      25072           +46.2%      36656        softirqs.CPU22.RCU
      30296           +43.3%      43425        softirqs.CPU22.SCHED
      24882           +48.8%      37031        softirqs.CPU23.RCU
      29900           +43.7%      42971        softirqs.CPU23.SCHED
      40388 ±  3%     +44.8%      58494 ±  3%  softirqs.CPU23.TIMER
      25007           +48.1%      37036        softirqs.CPU24.RCU
      29755           +45.9%      43399        softirqs.CPU24.SCHED
      40786           +40.6%      57353        softirqs.CPU24.TIMER
      25031           +47.3%      36875        softirqs.CPU25.RCU
      29841 ±  2%     +45.3%      43355        softirqs.CPU25.SCHED
      40971           +39.8%      57266        softirqs.CPU25.TIMER
      25145           +47.6%      37114        softirqs.CPU26.RCU
      29462 ±  2%     +48.5%      43739 ±  2%  softirqs.CPU26.SCHED
      40553           +41.8%      57521        softirqs.CPU26.TIMER
      25184           +47.6%      37169        softirqs.CPU27.RCU
      29867           +45.5%      43450 ±  3%  softirqs.CPU27.SCHED
      41118           +39.3%      57261        softirqs.CPU27.TIMER
      25452           +47.1%      37440        softirqs.CPU28.RCU
      29714           +46.2%      43440 ±  2%  softirqs.CPU28.SCHED
      41209 ±  2%     +37.8%      56783 ±  2%  softirqs.CPU28.TIMER
      25334           +48.8%      37696 ±  2%  softirqs.CPU29.RCU
      29894           +46.3%      43749        softirqs.CPU29.SCHED
      41413 ±  2%     +43.3%      59346 ±  5%  softirqs.CPU29.TIMER
      25514           +47.0%      37495        softirqs.CPU3.RCU
      30016           +46.0%      43831 ±  2%  softirqs.CPU3.SCHED
      41371 ±  2%     +39.3%      57611        softirqs.CPU3.TIMER
      25206           +47.4%      37149        softirqs.CPU30.RCU
      30167           +44.0%      43451 ±  2%  softirqs.CPU30.SCHED
      41166 ±  2%     +58.1%      65092 ± 19%  softirqs.CPU30.TIMER
      25211           +46.8%      37016        softirqs.CPU31.RCU
      30004 ±  2%     +45.2%      43569        softirqs.CPU31.SCHED
      41541 ±  2%     +36.9%      56874        softirqs.CPU31.TIMER
      24432           +46.9%      35885        softirqs.CPU32.RCU
      30206           +43.4%      43322 ±  2%  softirqs.CPU32.SCHED
      41069           +40.7%      57781 ±  2%  softirqs.CPU32.TIMER
      24303           +48.0%      35973        softirqs.CPU33.RCU
      30238           +43.8%      43489        softirqs.CPU33.SCHED
      41161           +38.2%      56872        softirqs.CPU33.TIMER
      24308           +47.6%      35876        softirqs.CPU34.RCU
      29803           +46.5%      43647 ±  2%  softirqs.CPU34.SCHED
      40809           +39.2%      56817        softirqs.CPU34.TIMER
      24291           +47.7%      35877        softirqs.CPU35.RCU
      30092           +44.2%      43386        softirqs.CPU35.SCHED
      40607           +41.5%      57480 ±  2%  softirqs.CPU35.TIMER
      25605           +49.5%      38286        softirqs.CPU36.RCU
      29900           +44.6%      43238        softirqs.CPU36.SCHED
      45207 ± 15%     +27.4%      57602        softirqs.CPU36.TIMER
      25555 ±  2%     +49.0%      38070        softirqs.CPU37.RCU
      29926           +45.2%      43460        softirqs.CPU37.SCHED
      42619 ±  8%     +36.7%      58242        softirqs.CPU37.TIMER
      27683 ±  9%     +50.2%      41577 ±  4%  softirqs.CPU38.RCU
      29700           +45.6%      43230        softirqs.CPU38.SCHED
      41779 ±  4%     +43.5%      59948        softirqs.CPU38.TIMER
      26699 ±  7%     +42.1%      37951        softirqs.CPU39.RCU
      30061           +43.4%      43102        softirqs.CPU39.SCHED
      42312 ±  5%     +36.1%      57581 ±  2%  softirqs.CPU39.TIMER
      25219           +47.7%      37247        softirqs.CPU4.RCU
      30363 ±  3%     +45.9%      44295        softirqs.CPU4.SCHED
      41108 ±  2%     +39.6%      57371 ±  2%  softirqs.CPU4.TIMER
      25413           +50.6%      38268        softirqs.CPU40.RCU
      29910           +46.0%      43665        softirqs.CPU40.SCHED
      39987 ±  2%     +45.9%      58322        softirqs.CPU40.TIMER
      25569           +49.5%      38238        softirqs.CPU41.RCU
      29867           +44.8%      43247        softirqs.CPU41.SCHED
      25492           +49.8%      38185        softirqs.CPU42.RCU
      29871           +45.4%      43440        softirqs.CPU42.SCHED
      40076           +44.8%      58020        softirqs.CPU42.TIMER
      25281           +48.9%      37640        softirqs.CPU43.RCU
      30025           +43.9%      43220        softirqs.CPU43.SCHED
      40222 ±  3%     +44.7%      58195        softirqs.CPU43.TIMER
      25183           +49.3%      37611        softirqs.CPU44.RCU
      30123           +43.6%      43249        softirqs.CPU44.SCHED
      41090 ±  6%     +42.5%      58537        softirqs.CPU44.TIMER
      25314           +49.6%      37863        softirqs.CPU45.RCU
      29998           +43.2%      42957        softirqs.CPU45.SCHED
      39705 ±  3%     +43.6%      57033        softirqs.CPU45.TIMER
      25500           +46.2%      37274        softirqs.CPU46.RCU
      29921           +44.7%      43286        softirqs.CPU46.SCHED
      41088 ±  5%     +39.9%      57463 ±  3%  softirqs.CPU46.TIMER
      25564           +48.4%      37931        softirqs.CPU47.RCU
      30178           +42.4%      42980        softirqs.CPU47.SCHED
      40738 ±  4%     +42.2%      57939 ±  3%  softirqs.CPU47.TIMER
      26467 ±  8%     +40.8%      37278        softirqs.CPU5.RCU
      30334 ±  2%     +44.1%      43714        softirqs.CPU5.SCHED
      42268 ±  4%     +35.8%      57394        softirqs.CPU5.TIMER
      25343           +47.0%      37245        softirqs.CPU6.RCU
      30100           +45.0%      43643 ±  2%  softirqs.CPU6.SCHED
      41270           +43.6%      59249 ±  5%  softirqs.CPU6.TIMER
      25331           +47.7%      37418        softirqs.CPU7.RCU
      30210           +44.7%      43727 ±  2%  softirqs.CPU7.SCHED
      41028           +41.9%      58206 ±  2%  softirqs.CPU7.TIMER
      25483           +47.2%      37514        softirqs.CPU8.RCU
      30324 ±  2%     +43.4%      43483 ±  2%  softirqs.CPU8.SCHED
      41027           +40.4%      57591 ±  2%  softirqs.CPU8.TIMER
      25546           +47.5%      37687        softirqs.CPU9.RCU
      30218           +44.8%      43771 ±  2%  softirqs.CPU9.SCHED
      41193           +40.1%      57716 ±  2%  softirqs.CPU9.TIMER
    1212606           +47.7%    1790808        softirqs.RCU
    1448198           +44.7%    2095500        softirqs.SCHED
    1990064           +40.7%    2800202        softirqs.TIMER
       8.56 ±  3%      -4.3        4.25 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
       8.50 ±  3%      -4.3        4.20 perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
       8.47 ±  3%      -4.3        4.17 perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
      78.55            -4.1       74.44 perf-profile.calltrace.cycles-pp.write
       8.74 ±  3%      -4.0        4.70 perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      77.69            -3.9       73.75 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      77.64            -3.9       73.71 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      77.13            -3.8       73.36 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      77.08            -3.8       73.31 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      76.61            -3.7       72.96 perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      76.43            -3.6       72.80 perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
      69.22            -2.6       66.62 perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
       4.70 ±  2%      -2.2        2.50 perf-profile.calltrace.cycles-pp.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
       4.63 ±  2%      -2.2        2.45 perf-profile.calltrace.cycles-pp.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
       4.60 ±  2%      -2.2        2.43 perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages.do_writepages
       4.32 ±  2%      -2.1        2.22 perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend.xfs_vm_writepages
       3.72 ±  4%      -2.1        1.63 perf-profile.calltrace.cycles-pp.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
       3.71 ±  4%      -2.1        1.62 perf-profile.calltrace.cycles-pp.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
       2.63 ±  4%      -2.0        0.63 ±  3% perf-profile.calltrace.cycles-pp.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
       7.72            -2.0        5.74 ±  2% perf-profile.calltrace.cycles-pp.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
       3.08 ±  5%      -1.9        1.13 perf-profile.calltrace.cycles-pp.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages.do_writepages
       2.49 ±  5%      -1.9        0.55 ±  3% perf-profile.calltrace.cycles-pp.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request.submit_bio
       2.48 ±  5%      -1.9        0.55 ±  3% perf-profile.calltrace.cycles-pp.test_clear_page_writeback.end_page_writeback.iomap_finish_ioend.brd_make_request.generic_make_request
       2.41 ±  6%      -1.8        0.61 ±  2% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.iomap_writepage_map.write_cache_pages.iomap_writepages.xfs_vm_writepages
       4.90            -1.5        3.40 ±  3% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
       4.78            -1.5        3.30 ±  3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
       2.35            -0.6        1.77 ±  2% perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
       2.36            -0.6        1.79 perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
       3.40            -0.5        2.87 ±  6% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write
       1.83            -0.5        1.33 ±  2% perf-profile.calltrace.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
       3.26            -0.5        2.75 ±  6% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
       2.85            -0.4        2.40 perf-profile.calltrace.cycles-pp.complete.process_one_work.worker_thread.kthread.ret_from_fork
       2.79            -0.4        2.34 perf-profile.calltrace.cycles-pp.try_to_wake_up.swake_up_locked.complete.process_one_work.worker_thread
       2.83            -0.4        2.38 perf-profile.calltrace.cycles-pp.swake_up_locked.complete.process_one_work.worker_thread.kthread
       1.48            -0.4        1.05 ±  2% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.iomap_submit_ioend
       0.61 ±  2%      -0.4        0.26 ±100% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock
       1.89            -0.3        1.54 perf-profile.calltrace.cycles-pp.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
       1.71            -0.3        1.38 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn.xfs_log_force_lsn
       1.65            -0.3        1.34 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_for_completion.__flush_work.xlog_cil_force_lsn
       0.69            -0.3        0.38 ± 57% perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
       1.59            -0.3        1.29 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_timeout.wait_for_completion.__flush_work
       0.86            -0.3        0.58 ±  2% perf-profile.calltrace.cycles-pp.rwsem_down_read_slowpath.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
       1.52            -0.3        1.26 ±  3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time.file_update_time
       1.75            -0.2        1.50 ±  2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete.process_one_work
       1.74            -0.2        1.49 ±  2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked
       1.74            -0.2        1.50 ±  2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.swake_up_locked.complete
       2.15            -0.2        1.94 perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
       0.79 ±  3%      -0.2        0.60 ±  4% perf-profile.calltrace.cycles-pp._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit.xfs_vn_update_time
       1.81            -0.2        1.62 perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
       1.87            -0.2        1.68 perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
       0.70 ±  4%      -0.2        0.54 ±  4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_insert_items.xfs_log_commit_cil.__xfs_trans_commit
       0.81            -0.1        0.68 perf-profile.calltrace.cycles-pp.unwind_next_frame.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity
       0.65 ±  2%      -0.1        0.55 ±  2% perf-profile.calltrace.cycles-pp.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn.xfs_log_force_lsn
       0.62 ±  2%      -0.1        0.52 ±  3% perf-profile.calltrace.cycles-pp.scheduler_ipi.reschedule_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.xlog_cil_force_lsn
       1.18            -0.0        1.16 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
       1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
       1.00            -0.0        0.98 perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
       0.96            -0.0        0.93 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
       1.01            -0.0        0.99 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
       0.59 ±  3%      +0.1        0.68 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
       0.58 ±  3%      +0.1        0.67 perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary
       0.53            +0.1        0.63 ±  3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work
       0.91 ±  3%      +0.1        1.02 ±  2% perf-profile.calltrace.cycles-pp.xlog_write.xlog_cil_push_work.process_one_work.worker_thread.kthread
       0.54            +0.1        0.65 ±  3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread
       0.57            +0.1        0.69 ±  3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_cil_push_work.process_one_work.worker_thread.kthread
       1.83            +0.2        1.99 perf-profile.calltrace.cycles-pp.xlog_cil_push_work.process_one_work.worker_thread.kthread.ret_from_fork
       0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
       0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
       0.00            +0.6        0.57 ±  2% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit
       0.00            +0.7        0.70 ±  2% perf-profile.calltrace.cycles-pp.xfs_bmap_add_extent_unwritten_real.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend
       0.00            +0.8        0.75 ±  4% perf-profile.calltrace.cycles-pp.xfs_bmapi_convert_unwritten.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
       0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend
       0.00            +0.8        0.80 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends
       0.00            +0.8        0.80 ± 30% perf-profile.calltrace.cycles-pp.xfs_trans_reserve.xfs_trans_alloc.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks
       0.00            +0.8        0.82 perf-profile.calltrace.cycles-pp.__wake_up_common.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend
       0.00            +0.8        0.85 perf-profile.calltrace.cycles-pp.wake_up_page_bit.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io
       0.00            +0.9        0.89 ±  2% perf-profile.calltrace.cycles-pp.xfs_bmapi_write.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
       0.00            +0.9        0.92 ± 21% perf-profile.calltrace.cycles-pp.xfs_log_commit_cil.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io
       0.00            +1.0        0.97 ± 20% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work
      35.68            +1.0       36.68 perf-profile.calltrace.cycles-pp.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
       0.00            +1.0        1.04 perf-profile.calltrace.cycles-pp.iomap_finish_ioend.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work
       0.00            +1.0        1.05 perf-profile.calltrace.cycles-pp.iomap_finish_ioends.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
       9.35            +1.2       10.54 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
       9.61            +1.2       10.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
       9.63            +1.2       10.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      12.24            +1.3       13.58 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      12.25            +1.3       13.60 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      12.25            +1.3       13.60 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      12.49            +1.4       13.89 perf-profile.calltrace.cycles-pp.secondary_startup_64
      14.38            +1.4       15.81 ±  2% perf-profile.calltrace.cycles-pp.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      13.41            +1.4       14.86 ±  2% perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
      13.37            +1.5       14.82 ±  2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn.xfs_log_force_lsn
      13.34            +1.5       14.80 ±  2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_wait_on_iclog.__xfs_log_force_lsn
      58.91            +1.6       60.56 perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
       0.00            +2.2        2.24 ± 13% perf-profile.calltrace.cycles-pp.xfs_iomap_write_unwritten.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread
       5.63            +2.5        8.15 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn
       5.65            +2.5        8.19 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync
       5.69            +2.5        8.23 perf-profile.calltrace.cycles-pp.remove_wait_queue.xlog_cil_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
       7.04            +2.8        9.80 ±  2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
       7.54            +2.9       10.45 ±  2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
       7.55            +2.9       10.46 ±  2% perf-profile.calltrace.cycles-pp.ret_from_fork
       7.55            +2.9       10.46 ±  2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
       0.00            +3.3        3.30 ±  9% perf-profile.calltrace.cycles-pp.xfs_end_ioend.xfs_end_io.process_one_work.worker_thread.kthread
       0.00            +3.3        3.32 ±  8% perf-profile.calltrace.cycles-pp.xfs_end_io.process_one_work.worker_thread.kthread.ret_from_fork
       8.56 ±  3%      -4.3        4.25 perf-profile.children.cycles-pp.__filemap_fdatawrite_range
       8.47 ±  3%      -4.3        4.18 perf-profile.children.cycles-pp.xfs_vm_writepages
       8.50 ±  3%      -4.3        4.20 perf-profile.children.cycles-pp.do_writepages
      78.64            -4.1       74.51 perf-profile.children.cycles-pp.write
      78.55            -4.1       74.44 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      78.50            -4.1       74.41 perf-profile.children.cycles-pp.do_syscall_64
       8.75 ±  3%      -4.1        4.70 perf-profile.children.cycles-pp.file_write_and_wait_range
      77.14            -3.8       73.37 perf-profile.children.cycles-pp.ksys_write
      77.09            -3.8       73.33 perf-profile.children.cycles-pp.vfs_write
      76.62            -3.7       72.97 perf-profile.children.cycles-pp.new_sync_write
      76.44            -3.6       72.81 perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
      69.23            -2.6       66.62 perf-profile.children.cycles-pp.xfs_file_fsync
       4.70 ±  2%      -2.2        2.50 perf-profile.children.cycles-pp.iomap_submit_ioend
       4.89 ±  2%      -2.1        2.75 perf-profile.children.cycles-pp.submit_bio
       4.85 ±  2%      -2.1        2.71 perf-profile.children.cycles-pp.generic_make_request
       3.73 ±  4%      -2.1        1.63 perf-profile.children.cycles-pp.iomap_writepages
       3.72 ±  4%      -2.1        1.63 perf-profile.children.cycles-pp.write_cache_pages
       4.43 ±  2%      -2.1        2.38 perf-profile.children.cycles-pp.brd_make_request
       7.73            -2.0        5.75 ±  2% perf-profile.children.cycles-pp.__flush_work
       3.08 ±  4%      -1.9        1.14 perf-profile.children.cycles-pp.iomap_writepage_map
       6.00            -1.8        4.20 ±  2% perf-profile.children.cycles-pp._raw_spin_lock_irq
       2.50 ±  4%      -1.8        0.70 ±  3% perf-profile.children.cycles-pp.end_page_writeback
       2.50 ±  5%      -1.8        0.70 ±  4% perf-profile.children.cycles-pp.test_clear_page_writeback
       2.42 ±  6%      -1.8        0.61 ±  2% perf-profile.children.cycles-pp.__test_set_page_writeback
       2.08 ±  7%      -1.6        0.44 ±  4% perf-profile.children.cycles-pp.sb_mark_inode_writeback
       2.08 ±  6%      -1.6        0.44 ±  5% perf-profile.children.cycles-pp.sb_clear_inode_writeback
       2.70 ±  4%      -1.0        1.67 ±  2% perf-profile.children.cycles-pp.iomap_finish_ioend
       2.36            -0.6        1.79 perf-profile.children.cycles-pp.iomap_file_buffered_write
       2.35            -0.6        1.78 perf-profile.children.cycles-pp.iomap_apply
       1.84            -0.5        1.33 ±  2% perf-profile.children.cycles-pp.iomap_write_actor
       2.41 ±  4%      -0.5        1.94 ±  2% perf-profile.children.cycles-pp.reschedule_interrupt
       3.44 ±  2%      -0.5        2.98 perf-profile.children.cycles-pp.sched_ttwu_pending
       2.87            -0.5        2.42 perf-profile.children.cycles-pp.complete
       2.83            -0.4        2.38 perf-profile.children.cycles-pp.swake_up_locked
       2.28 ±  4%      -0.4        1.84 ±  2% perf-profile.children.cycles-pp.scheduler_ipi
       1.56            -0.4        1.17 ±  2% perf-profile.children.cycles-pp.brd_do_bvec
       1.89            -0.4        1.54 perf-profile.children.cycles-pp.wait_for_completion
       1.71            -0.3        1.38 perf-profile.children.cycles-pp.schedule_timeout
       0.88            -0.2        0.68 ±  2% perf-profile.children.cycles-pp.rwsem_down_read_slowpath
       0.69            -0.2        0.51 ±  2% perf-profile.children.cycles-pp.iomap_write_begin
       0.67 ±  2%      -0.2        0.49 perf-profile.children.cycles-pp.lseek64
       1.33 ±  5%      -0.2        1.17 ±  3% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
       0.51            -0.2        0.36 perf-profile.children.cycles-pp.grab_cache_page_write_begin
       0.47            -0.1        0.33 perf-profile.children.cycles-pp.pagecache_get_page
       0.51 ±  3%      -0.1        0.38 ±  2% perf-profile.children.cycles-pp.iomap_write_end
       0.57 ±  2%      -0.1        0.46 perf-profile.children.cycles-pp.blkdev_issue_flush
       0.38            -0.1        0.26 ±  5% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
       0.34            -0.1        0.23 ±  6% perf-profile.children.cycles-pp.copyin
       0.33            -0.1        0.22 ±  6% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
       0.65 ±  2%      -0.1        0.56 ±  2% perf-profile.children.cycles-pp.rwsem_optimistic_spin
       0.37            -0.1        0.28 ±  2% perf-profile.children.cycles-pp.bio_alloc_bioset
       0.31 ±  4%      -0.1        0.22 perf-profile.children.cycles-pp.syscall_return_via_sysret
       0.56 ±  3%      -0.1        0.48 ±  3% perf-profile.children.cycles-pp.osq_lock
       0.26 ±  3%      -0.1        0.18 ±  4% perf-profile.children.cycles-pp.entry_SYSCALL_64
       0.28 ±  2%      -0.1        0.21 ±  5% perf-profile.children.cycles-pp.mempool_alloc
       0.26 ±  6%      -0.1        0.18 ±  8% perf-profile.children.cycles-pp.xlog_state_get_iclog_space
       0.29            -0.1        0.22 ±  3% perf-profile.children.cycles-pp.find_get_entry
       0.34 ±  2%      -0.1        0.27 ±  4% perf-profile.children.cycles-pp.generic_make_request_checks
       0.30 ±  5%      -0.1        0.23 ±  3% perf-profile.children.cycles-pp.iomap_set_page_dirty
       0.32 ±  2%      -0.1        0.26 ±  5% perf-profile.children.cycles-pp.brd_lookup_page
       0.18 ±  2%      -0.1        0.12 ±  6% perf-profile.children.cycles-pp.iomap_set_range_uptodate
       0.15 ±  2%      -0.1        0.09 ±  4% perf-profile.children.cycles-pp.__mod_lruvec_state
       0.15 ±  5%      -0.1        0.10 ±  9% perf-profile.children.cycles-pp.wake_up_q
       0.21 ±  5%      -0.1        0.15 ±  5% perf-profile.children.cycles-pp.xfs_ilock
       0.13 ±  3%      -0.0        0.08 ± 10% perf-profile.children.cycles-pp.rwsem_wake
       0.20 ±  2%      -0.0        0.15 ±  5% perf-profile.children.cycles-pp.ksys_lseek
       0.20 ±  3%      -0.0        0.15 ±  3% perf-profile.children.cycles-pp.xas_load
       0.20 ±  5%      -0.0        0.16 ±  2% perf-profile.children.cycles-pp.__set_page_dirty
       0.19 ±  2%      -0.0        0.15 ±  2% perf-profile.children.cycles-pp.xfs_iunlock
       0.40            -0.0        0.36 ±  4% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin
       0.17 ±  2%      -0.0        0.12 ±  4% perf-profile.children.cycles-pp.clear_page_dirty_for_io
       0.20 ±  3%      -0.0        0.16 ±  4% perf-profile.children.cycles-pp.submit_bio_wait
       0.18 ±  4%      -0.0        0.15 ±  7% perf-profile.children.cycles-pp.security_file_permission
       0.10            -0.0        0.06 ±  6% perf-profile.children.cycles-pp.evict
       0.10            -0.0        0.06 ±  6% perf-profile.children.cycles-pp.truncate_inode_pages_range
       0.06 ±  6%      -0.0        0.03 ±100% perf-profile.children.cycles-pp.vfs_unlink
       0.08 ±  6%      -0.0        0.04 ± 58% perf-profile.children.cycles-pp.add_to_page_cache_lru
       0.26 ±  2%      -0.0        0.23 ±  3% perf-profile.children.cycles-pp.reweight_entity
       0.12 ±  6%      -0.0        0.09 ±  4% perf-profile.children.cycles-pp.bio_endio
       0.10 ±  7%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.do_sys_open
       0.10 ±  7%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.do_sys_openat2
       0.17 ±  2%      -0.0        0.14 perf-profile.children.cycles-pp.__pagevec_release
       0.13 ±  5%      -0.0        0.10 ±  4% perf-profile.children.cycles-pp.down_write
       0.10 ±  5%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.do_filp_open
       0.15 ±  4%      -0.0        0.12 ±  5% perf-profile.children.cycles-pp.tag_pages_for_writeback
       0.11 ±  9%      -0.0        0.08 ±  5% perf-profile.children.cycles-pp.bio_associate_blkg
       0.10 ±  5%      -0.0        0.07 ±  7% perf-profile.children.cycles-pp.open64
       0.09 ±  4%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.path_openat
       0.16 ±  8%      -0.0        0.13 ±  3% perf-profile.children.cycles-pp.pagevec_lookup_range_tag
       0.12 ±  8%      -0.0        0.10 ±  5% perf-profile.children.cycles-pp.account_page_dirtied
       0.10 ±  4%      -0.0        0.08 ±  6% perf-profile.children.cycles-pp.__fdget_pos
       0.14 ±  3%      -0.0        0.11 ±  7% perf-profile.children.cycles-pp.memset_erms
       0.11 ±  3%      -0.0        0.09 ±  4% perf-profile.children.cycles-pp.switch_fpu_return
       0.23            -0.0        0.21 ±  3% perf-profile.children.cycles-pp.__radix_tree_lookup
       0.18 ±  2%      -0.0        0.16 ±  9% perf-profile.children.cycles-pp.insert_work
       0.12 ±  4%      -0.0        0.10 ±  7% perf-profile.children.cycles-pp.common_file_perm
       0.09 ±  4%      -0.0        0.07 ±  6% perf-profile.children.cycles-pp.__fget_light
       0.09 ±  4%      -0.0        0.07 ±  6% perf-profile.children.cycles-pp.bvec_alloc
       0.15 ±  7%      -0.0        0.12 ±  3% perf-profile.children.cycles-pp.find_get_pages_range_tag
       0.08 ±  8%      -0.0        0.06 ±  7% perf-profile.children.cycles-pp.iov_iter_fault_in_readable
       0.09 ±  5%      -0.0        0.06 ± 13% perf-profile.children.cycles-pp.blk_throtl_bio
       0.24 ±  4%      -0.0        0.22 ±  3% perf-profile.children.cycles-pp._cond_resched
       0.13 ±  3%      -0.0        0.11 ±  4% perf-profile.children.cycles-pp.release_pages
       0.10 ±  4%      -0.0        0.08 ±  6% perf-profile.children.cycles-pp.xfs_bmbt_to_iomap
       0.08 ± 10%      -0.0        0.06 perf-profile.children.cycles-pp.unlink
       0.08 ± 10%      -0.0        0.06 ± 11% perf-profile.children.cycles-pp.__sb_start_write
       0.08 ±  5%      -0.0        0.06 ±  7% perf-profile.children.cycles-pp.generic_write_checks
       0.08 ±  8%      -0.0        0.06 perf-profile.children.cycles-pp.do_unlinkat
       0.07 ± 10%      -0.0        0.05 perf-profile.children.cycles-pp.xfs_break_layouts
       0.18 ±  2%      -0.0        0.16 ±  2% perf-profile.children.cycles-pp.__list_add_valid
       0.12 ±  3%      -0.0        0.10 ±  8% perf-profile.children.cycles-pp.__might_sleep
       0.14 ±  3%      -0.0        0.12 ±  7% perf-profile.children.cycles-pp.xas_find_marked
       0.07 ± 10%      -0.0        0.05 ±  8% perf-profile.children.cycles-pp.bio_associate_blkg_from_css
       0.08 ±  5%      -0.0        0.06 ± 11% perf-profile.children.cycles-pp.bio_free
       0.08 ±  6%      -0.0        0.06 perf-profile.children.cycles-pp.xfs_file_llseek
       0.08 ±  5%      -0.0        0.06 ±  6% perf-profile.children.cycles-pp.xfs_file_write_iter
       0.16 ±  2%      +0.0        0.17 ±  2% perf-profile.children.cycles-pp.__switch_to_asm
       0.05            +0.0        0.07 ±  7% perf-profile.children.cycles-pp.cpumask_next_wrap
       0.10 ±  4%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp._find_next_bit
       0.07 ±  7%      +0.0        0.09 ±  5% perf-profile.children.cycles-pp.xlog_state_done_syncing
       0.16 ±  2%      +0.0        0.19 ±  2% perf-profile.children.cycles-pp.check_preempt_curr
       0.19 ±  4%      +0.0        0.22 perf-profile.children.cycles-pp.ttwu_do_wakeup
       0.56 ±  3%      +0.0        0.60 perf-profile.children.cycles-pp.dequeue_entity
       0.20 ±  6%      +0.0        0.23 ±  4% perf-profile.children.cycles-pp.xfs_trans_committed_bulk
       0.07 ±  7%      +0.0        0.10 ±  7% perf-profile.children.cycles-pp.check_preempt_wakeup
       0.82 ±  2%      +0.0        0.86 perf-profile.children.cycles-pp.update_load_avg
       0.31 ±  3%      +0.0        0.35 perf-profile.children.cycles-pp.__unwind_start
       0.08 ±  8%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.destroy_inode
       0.08 ± 10%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.xfs_fs_destroy_inode
       0.08 ± 10%      +0.0        0.12 ±  3% perf-profile.children.cycles-pp.xfs_inactive
       0.08 ± 15%      +0.0        0.12 ±  5% perf-profile.children.cycles-pp.xfs_bmap_validate_extent
       3.96            +0.0        4.01 perf-profile.children.cycles-pp.stack_trace_save_tsk
       0.28            +0.0        0.33 perf-profile.children.cycles-pp.set_next_entity
       0.00            +0.1        0.05 perf-profile.children.cycles-pp.xfs_trans_roll
       0.00            +0.1        0.05 ±  8% perf-profile.children.cycles-pp.xfs_btree_delrec
       0.00            +0.1        0.05 ±  8% perf-profile.children.cycles-pp.xfs_btree_insert
       0.16 ±  4%      +0.1        0.21 ±  5% perf-profile.children.cycles-pp.memcpy_erms
       0.04 ± 57%      +0.1        0.10 ±  4% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags
       0.00            +0.1        0.06 ± 11% perf-profile.children.cycles-pp.xfs_buf_item_format_segment
       0.00            +0.1        0.06 ± 11% perf-profile.children.cycles-pp.xfs_btree_delete
       0.00            +0.1        0.06 perf-profile.children.cycles-pp.xfs_btree_update
       0.00            +0.1        0.07 ±  7% perf-profile.children.cycles-pp.xfs_cil_prepare_item
       0.00            +0.1        0.07 ±  6% perf-profile.children.cycles-pp.xfs_bmbt_update
       0.00            +0.1        0.07 ± 10% perf-profile.children.cycles-pp.xfs_buf_item_format
       0.56            +0.1        0.64 ±  2% perf-profile.children.cycles-pp.pick_next_task_fair
       0.03 ±100%      +0.1        0.10 ±  4% perf-profile.children.cycles-pp.xfs_inactive_truncate
       0.08 ±  5%      +0.1        0.16 ± 11% perf-profile.children.cycles-pp.xfs_iext_lookup_extent
       0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_defer_finish
       0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_bmap_check_leaf_extents
       0.00            +0.1        0.08 ±  5% perf-profile.children.cycles-pp.xfs_verify_agbno
       0.00            +0.1        0.09 ±  9% perf-profile.children.cycles-pp.xfs_defer_finish_noroll
       0.60 ±  3%      +0.1        0.70 perf-profile.children.cycles-pp.schedule_idle
       0.39 ±  2%      +0.1        0.49 perf-profile.children.cycles-pp.xlog_state_release_iclog
       0.00            +0.1        0.10 ±  9% perf-profile.children.cycles-pp.xfs_buf_find
       0.93 ±  3%      +0.1        1.03 ±  2% perf-profile.children.cycles-pp.xlog_write
       2.58            +0.1        2.69 perf-profile.children.cycles-pp.schedule
       1.52            +0.1        1.63 perf-profile.children.cycles-pp.__wake_up_common_lock
       0.00            +0.1        0.11 ±  7% perf-profile.children.cycles-pp.xfs_buf_get_map
       4.62            +0.1        4.73 perf-profile.children.cycles-pp.__account_scheduler_latency
       0.00            +0.1        0.12 ± 10% perf-profile.children.cycles-pp.xfs_buf_read_map
       1.83            +0.2        1.99 perf-profile.children.cycles-pp.xlog_cil_push_work
       5.94            +0.2        6.12 perf-profile.children.cycles-pp.ttwu_do_activate
       0.00            +0.2        0.17 ±  6% perf-profile.children.cycles-pp.xfs_btree_read_buf_block
       5.93            +0.2        6.11 perf-profile.children.cycles-pp.activate_task
       5.88            +0.2        6.06 perf-profile.children.cycles-pp.enqueue_task_fair
       0.00            +0.2        0.18 ±  5% perf-profile.children.cycles-pp.xfs_trans_read_buf_map
       5.41            +0.2        5.59 perf-profile.children.cycles-pp.enqueue_entity
       3.14            +0.2        3.33 perf-profile.children.cycles-pp.__sched_text_start
       0.00            +0.2        0.21 ±  6% perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
       0.00            +0.2        0.23 ± 57% perf-profile.children.cycles-pp.xlog_grant_head_wait
       0.00            +0.2        0.24 ±  6% perf-profile.children.cycles-pp.io_schedule
       0.11 ±  3%      +0.3        0.38 ±  4% perf-profile.children.cycles-pp.__filemap_fdatawait_range
       0.00            +0.3        0.27 ±  8% perf-profile.children.cycles-pp.wait_on_page_bit
       0.00            +0.3        0.34 ±  5% perf-profile.children.cycles-pp.xfs_btree_lookup
       0.08 ± 14%      +0.4        0.43 perf-profile.children.cycles-pp.__queue_work
       0.08 ±  8%      +0.4        0.44 perf-profile.children.cycles-pp.queue_work_on
       0.00            +0.4        0.41 perf-profile.children.cycles-pp.xfs_end_bio
       3.51            +0.4        3.94 ±  9% perf-profile.children.cycles-pp.__xfs_trans_commit
       0.20 ±  9%      +0.5        0.72 ± 54% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant
       0.06 ± 11%      +0.5        0.58 ± 67% perf-profile.children.cycles-pp.xfs_log_space_wake
       0.45 ±  5%      +0.6        1.04 ± 36% perf-profile.children.cycles-pp.xfs_log_reserve
       0.65 ±  3%      +0.6        1.25 ± 29% perf-profile.children.cycles-pp.xfs_trans_alloc
       0.48 ±  5%      +0.6        1.09 ± 34% perf-profile.children.cycles-pp.xfs_trans_reserve
       0.04 ± 57%      +0.6        0.65 ± 60% perf-profile.children.cycles-pp.xlog_grant_head_check
       4.83            +0.7        5.52 perf-profile.children.cycles-pp.try_to_wake_up
       0.00            +0.7        0.70 ±  2% perf-profile.children.cycles-pp.xfs_bmap_add_extent_unwritten_real
       0.00            +0.8        0.75 ±  4% perf-profile.children.cycles-pp.xfs_bmapi_convert_unwritten
       0.04 ± 57%      +0.8        0.80 perf-profile.children.cycles-pp.autoremove_wake_function
       0.05 ±  8%      +0.8        0.85 perf-profile.children.cycles-pp.wake_up_page_bit
       1.50            +0.9        2.36 perf-profile.children.cycles-pp.__wake_up_common
       0.00            +0.9        0.89 ±  2% perf-profile.children.cycles-pp.xfs_bmapi_write
      24.29            +0.9       25.18 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
       0.07 ±  5%      +1.0        1.05 perf-profile.children.cycles-pp.iomap_finish_ioends
      35.68            +1.0       36.69 perf-profile.children.cycles-pp.xlog_cil_force_lsn
       9.53            +1.2       10.76 perf-profile.children.cycles-pp.intel_idle
       9.81            +1.3       11.10 perf-profile.children.cycles-pp.cpuidle_enter_state
       9.82            +1.3       11.11 perf-profile.children.cycles-pp.cpuidle_enter
      12.25            +1.3       13.60 perf-profile.children.cycles-pp.start_secondary
      12.49            +1.4       13.89 perf-profile.children.cycles-pp.secondary_startup_64
      12.49            +1.4       13.89 perf-profile.children.cycles-pp.cpu_startup_entry
      12.48            +1.4       13.88 perf-profile.children.cycles-pp.do_idle
      14.78            +1.4       16.20 ±  2% perf-profile.children.cycles-pp.xlog_wait_on_iclog
      58.91            +1.7       60.57 perf-profile.children.cycles-pp.xfs_log_force_lsn
       0.00            +2.2        2.24 ± 13% perf-profile.children.cycles-pp.xfs_iomap_write_unwritten
       7.05            +2.8        9.81 ±  2% perf-profile.children.cycles-pp.process_one_work
       7.54            +2.9       10.45 ±  2% perf-profile.children.cycles-pp.worker_thread
       7.55            +2.9       10.46 ±  2% perf-profile.children.cycles-pp.ret_from_fork
       7.55            +2.9       10.46 ±  2% perf-profile.children.cycles-pp.kthread
       0.10            +3.2        3.30 ±  9% perf-profile.children.cycles-pp.xfs_end_ioend
       0.10 ±  4%      +3.2        3.32 ±  8% perf-profile.children.cycles-pp.xfs_end_io
      19.57            +4.0       23.62 perf-profile.children.cycles-pp.remove_wait_queue
       1.20            -0.3        0.90 ±  2% perf-profile.self.cycles-pp.brd_do_bvec
       0.55 ±  3%      -0.2        0.39 ±  4% perf-profile.self.cycles-pp.do_syscall_64
       0.31            -0.1        0.22 ±  5% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
       0.31 ±  3%      -0.1        0.22 perf-profile.self.cycles-pp.syscall_return_via_sysret
       0.54 ±  3%      -0.1        0.47 ±  3% perf-profile.self.cycles-pp.osq_lock
       0.23 ±  3%      -0.1        0.16 ±  5% perf-profile.self.cycles-pp.entry_SYSCALL_64
       0.10 ±  4%      -0.1        0.04 ± 57% perf-profile.self.cycles-pp.__mod_lruvec_state
       0.18 ±  4%      -0.1        0.12 ±  5% perf-profile.self.cycles-pp.iomap_set_range_uptodate
       0.60 ±  4%      -0.1        0.54 ±  2% perf-profile.self.cycles-pp._raw_spin_lock
       0.77            -0.0        0.72 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
       0.14            -0.0        0.10 ±  8% perf-profile.self.cycles-pp.find_get_entry
       0.14 ±  3%      -0.0        0.11 ±  4% perf-profile.self.cycles-pp.xas_load
       0.17 ±  4%      -0.0        0.14 ±  3% perf-profile.self.cycles-pp.write
       0.30 ±  2%      -0.0        0.27 ±  6% perf-profile.self.cycles-pp._raw_spin_lock_irq
       0.18 ±  5%      -0.0        0.15 ±  7% perf-profile.self.cycles-pp.__flush_work
       0.08 ±  5%      -0.0        0.05 perf-profile.self.cycles-pp.xfs_inode_item_format_data_fork
       0.12 ±  3%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.brd_lookup_page
       0.12 ±  7%      -0.0        0.09 ±  9% perf-profile.self.cycles-pp.vfs_write
       0.18 ±  4%      -0.0        0.15 ±  7% perf-profile.self.cycles-pp.insert_work
       0.13            -0.0        0.10 ±  8% perf-profile.self.cycles-pp.memset_erms
       0.11 ±  4%      -0.0        0.08 perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
       0.11 ±  6%      -0.0        0.08 ±  5% perf-profile.self.cycles-pp.xfs_buffered_write_iomap_begin
       0.12 ±  3%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.iomap_writepage_map
       0.08 ±  8%      -0.0        0.06 ± 15% perf-profile.self.cycles-pp.bio_alloc_bioset
       0.06 ±  6%      -0.0        0.04 ± 57% perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
       0.09 ±  5%      -0.0        0.06 ±  6% perf-profile.self.cycles-pp.__fget_light
       0.06            -0.0        0.04 ± 57% perf-profile.self.cycles-pp.xfs_map_blocks
       0.18 ±  2%      -0.0        0.15 ±  3% perf-profile.self.cycles-pp.__list_add_valid
       0.08 ±  6%      -0.0        0.06 ±  9% perf-profile.self.cycles-pp.xfs_ilock
       0.08 ±  6%      -0.0        0.06 ±  9% perf-profile.self.cycles-pp.iov_iter_fault_in_readable
       0.11 ±  4%      -0.0        0.09 ±  4% perf-profile.self.cycles-pp.switch_fpu_return
       0.07            -0.0        0.05 perf-profile.self.cycles-pp.iomap_write_actor
       0.23 ±  2%      -0.0        0.21 ±  5% perf-profile.self.cycles-pp.__radix_tree_lookup
       0.18 ±  4%      -0.0        0.16 ±  6% perf-profile.self.cycles-pp.xlog_cil_force_lsn
       0.20 ±  2%      -0.0        0.18 ±  4% perf-profile.self.cycles-pp.kmem_cache_alloc
       0.10 ±  5%      -0.0        0.08 ± 11% perf-profile.self.cycles-pp.xfs_file_fsync
       0.07 ±  5%      -0.0        0.05 ±  9% perf-profile.self.cycles-pp.clear_page_dirty_for_io
       0.07 ±  6%      -0.0        0.05 perf-profile.self.cycles-pp.__test_set_page_writeback
       0.11            -0.0        0.09 ±  8% perf-profile.self.cycles-pp.common_file_perm
       0.23            -0.0        0.21 ±  4% perf-profile.self.cycles-pp.update_rq_clock
       0.08 ±  6%      -0.0        0.06 perf-profile.self.cycles-pp.xfs_file_write_iter
       0.10 ±  5%      -0.0        0.08 ±  8% perf-profile.self.cycles-pp.up_read
       0.08 ±  5%      -0.0        0.07 perf-profile.self.cycles-pp.write_cache_pages
       0.10            -0.0        0.09 perf-profile.self.cycles-pp.reweight_entity
       0.05 ±  8%      +0.0        0.07 ±  7% perf-profile.self.cycles-pp.dequeue_entity
       0.16 ±  4%      +0.0        0.17 ±  2% perf-profile.self.cycles-pp.__switch_to_asm
       0.11 ±  4%      +0.0        0.12 ±  3% perf-profile.self.cycles-pp.set_next_entity
       0.15 ±  3%      +0.0        0.16 ±  2% perf-profile.self.cycles-pp.orc_find
       0.09 ±  4%      +0.0        0.11 ±  6% perf-profile.self.cycles-pp.enqueue_entity
       0.10 ±  4%      +0.0        0.12 ±  3% perf-profile.self.cycles-pp._find_next_bit
       0.01 ±173%      +0.0        0.06 ±  7% perf-profile.self.cycles-pp.check_preempt_wakeup
       0.01 ±173%      +0.0        0.06 ± 11% perf-profile.self.cycles-pp.xfs_trans_committed_bulk
       0.45 ±  2%      +0.1        0.51 ±  2% perf-profile.self.cycles-pp.__sched_text_start
       0.15 ±  6%      +0.1        0.21 ±  3% perf-profile.self.cycles-pp.memcpy_erms
       0.00            +0.1        0.06 ±  6% perf-profile.self.cycles-pp.xfs_verify_agbno
       0.08 ±  8%      +0.1        0.16 ±  9% perf-profile.self.cycles-pp.xfs_iext_lookup_extent
       9.53            +1.2       10.76 perf-profile.self.cycles-pp.intel_idle


                                   aim7.jobs-per-min
   45000 +-------------------------------------------------------------------+
         |        + +                                                       |
   40000 |.+.+    :   + .+.   +. .+.     |
   35000 |-+ :    :    +   +.+ +.+.+.++.+.+.+.++.+.+.+.++.+.+.++.+.+   ++.+.|
         |   : :                                                          |
   30000 |-+ : :                                                           |
   25000 |-O :O O:O OO                                                     |
         |   :   :  O    O O OO O O O O                                      |
   20000 |-+  : :                                                           |
   15000 |-+  : :                                                           |
         |    : :                                                           |
   10000 |-+  : :                                                            |
    5000 |-+  : :                                                            |
         |    : :                                                            |
       0 +-------------------------------------------------------------------+
                                aim7.time.system_time
   5000 +--------------------------------------------------------------------+
   4500 |-+        O   OO   O    O O O                                       |
        | O OO O O   O    O   O O                                            |
   4000 |-+                                                                  |
   3500 |-+                                                                  |
        |             .+ .+.+. .++.+.+.+.+.++.+.+.+.++.+.+.+.+ .+.+.+. .++.+.|
   3000 |.+.+       .+  +     +                               + +      |
   2500 |-+ : +.+                                                         |
   2000 |-+ : :                                                           |
        |   : :                                                            |
   1500 |-+  : :                                                            |
   1000 |-+  : :                                                            |
        |    : :                                                            |
    500 |-+  : :                                                             |
      0 +--------------------------------------------------------------------+
                               aim7.time.elapsed_time
   160 +---------------------------------------------------------------------+
       |   O OO O O O O OO O O O O                                         |
   140 |-O                                                                   |
   120 |-+                                                                   |
|                                                                     |
   100 |-+           .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
       |.+.+    +. .+   +    + +      |
    80 |-+ :    : +                                                          |
       |    : :                                                            |
    60 |-+  : :                                                             |
    40 |-+  : :                                                             |
       |    : :                                                             |
    20 |-+   : :                                                             |
       | ::                                                              |
     0 +---------------------------------------------------------------------+
                             aim7.time.elapsed_time.max
   160 +---------------------------------------------------------------------+
       |   O OO O O O O OO O O O O                                         |
   140 |-O                                                                   |
   120 |-+                                                                   |
|                                                                     |
   100 |-+           .+. +.+. .+.+.++.+.+.+.+.+.++.+.+.+.+.++.+.+.+.+. +.+.+.|
       |.+.+    +. .+   +    + +      |
    80 |-+ :    : +                                                          |
       |    : :                                                            |
    60 |-+  : :                                                             |
    40 |-+  : :                                                             |
       |    : :                                                             |
    20 |-+   : :                                                             |
       | ::                                                              |
     0 +---------------------------------------------------------------------+
                         aim7.time.voluntary_context_switches
   8e+07 +-------------------------------------------------------------------+
         | O          O  O O OO O O                                        |
   7e+07 |-+                                                                 |
   6e+07 |-+                                                                 |
|                                                                   |
   5e+07 |-+           +.+.+.++.+.+.+.++.+.+. .+   .+.+.    .+. .+.   +.+.|
         |.+.+       .+                      +  +.+     ++.+   ++.+ +.+    |
   4e+07 |-+ : +.+                                                        |
         |   : :                                                          |
   3e+07 |-+ : :                                                           |
   2e+07 |-+  : :                                                           |
         |    : :                                                           |
   1e+07 |-+  : :                                                           |
         |    : :                                                            |
       0 +-------------------------------------------------------------------+
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


_______________________________________________
LKP mailing list -- lkp@xxxxxxxxxxxx
To unsubscribe send an email to lkp-leave@xxxxxxxxxxxx



--
Zhengjun Xing