Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Oliver Sang
Date: Sun May 17 2026 - 08:56:28 EST
hi, Shakeel, hi, Qi,
On Fri, May 15, 2026 at 10:09:06AM -0700, Shakeel Butt wrote:
> On Fri, May 15, 2026 at 03:37:22PM +0800, Qi Zheng wrote:
> > Hi Shakeel,
> >
> > On 5/14/26 9:40 PM, Shakeel Butt wrote:
> > > May 14, 2026 at 12:46 AM, "Qi Zheng" <qi.zheng@xxxxxxxxx mailto:qi.zheng@xxxxxxxxx?to=%22Qi%20Zheng%22%20%3Cqi.zheng%40linux.dev%3E > wrote:
> > >
> > >
> > > >
> > > > On 5/13/26 10:27 PM, Shakeel Butt wrote:
> > > >
> > > > >
> > > > > On Wed, May 13, 2026 at 06:49:45AM -0700, Shakeel Butt wrote:
> > > > >
> > > > > >
> > > > > > On Wed, May 13, 2026 at 10:10:34AM +0800, Qi Zheng wrote:
> > > > > >
> > > > > On 5/13/26 12:03 AM, Shakeel Butt wrote:
> > > > > On Tue, May 12, 2026 at 08:56:52PM +0800, kernel test robot wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
> > > > >
> > > > > commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> > > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > > >
> > > > > This is most probably due to shuffling of struct mem_cgroup and struct
> > > > > mem_cgroup_per_node members.
> > > > >
> > > > > Another possibility is that after objcg was split into per-node, the
> > > > > slab accounting fast path is still designed assuming only one current
> > > > > objcg per CPU:
> > > > >
> > > > > struct obj_stock_pcp {
> > > > > struct obj_cgroup *cached_objcg;
> > > > > };
> > > > >
> > > > > So it's may cause the following thrashing:
> > > > >
> > > > > CPU stock cached = memcg/node0 objcg
> > > > > free object tagged = memcg/node1 objcg
> > > > > => __refill_obj_stock --> objcg mismatch
> > > > > => drain_obj_stock()
> > > > > => cache switches to node1 objcg
> > > > >
> > > > > next local allocation tagged = node0 objcg
> > > > > => mismatch again
> > > > > => drain_obj_stock()
> > > > >
> > > > > >
> > > > > > Actually I think this is the issue, we have ping pong threads running on
> > > > > > different nodes where though theu are in same cgroup but their current->obcg is
> > > > > > for local node and thus this ping pong is thrashing the per-cpu objcg stock.
> > > > > >
> > > > > > The easier fix would be to compare objcg->memcg instead of just objcg during
> > > > > > draining and caching. In addition we can add support for multiple objcg per-cpu
> > > > > > stock caching.
> > > > > >
> > > > > Something like the following:
> > > > > From d756abe831a905d6fe32bad9a984fc619dafb7e0 Mon Sep 17 00:00:00 2001
> > > > > From: Shakeel Butt <shakeel.butt@xxxxxxxxx>
> > > > > Date: Wed, 13 May 2026 07:24:55 -0700
> > > > > Subject: [PATCH] mm/memcontrol: skip obj_stock drain when refilled objcg
> > > > > shares memcg
> > > > > Signed-off-by: Shakeel Butt <shakeel.butt@xxxxxxxxx>
> > > > > ---
> > > > > mm/memcontrol.c | 14 +++++++++++++-
> > > > > 1 file changed, 13 insertions(+), 1 deletion(-)
> > > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > > index d978e18b9b2d..01ed7a8e18ac 100644
> > > > > --- a/mm/memcontrol.c
> > > > > +++ b/mm/memcontrol.c
> > > > > @@ -3318,6 +3318,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > unsigned int nr_bytes,
> > > > > bool allow_uncharge)
> > > > > {
> > > > > + struct obj_cgroup *cached;
> > > > > unsigned int nr_pages = 0;
> > > > > > if (!stock) {
> > > > > @@ -3327,7 +3328,18 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > goto out;
> > > > > }
> > > > > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > > > + cached = READ_ONCE(stock->cached_objcg);
> > > > > + if (cached != objcg &&
> > > > > + (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
> > > > > drain_obj_stock(stock);
> > > > > obj_cgroup_get(objcg);
> > > > > stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > > > >
> > > > This change looks like it should be able to fix the ping-pong issue, but
> > > > I stiil haven't reproduced the performance regression locally. I'll
> > > > continue testing it.
> > >
> > > Same here, couldn't reproduce locally. It seems like we had to craft a scenario
> > > where the pair pingpong threads get their current->objcg from different nodes.
> > > I will try that.
> >
> > I still haven't been able to reproduce the LKP results locally, but I
> > used an AI bot to generate a pingpong test case (pasted at the end) and
> > automatically ran the test on a physical machine. The results are as
> > follows:
> >
> > parent: 8285917d6f
> > bad: 01b9da291c
> > fix: 01b9da291c + stock patch
> >
> > | kernel | mq_ops/sec mean | vs parent | drain_obj_stock / round |
> > |--------|-----------------|-----------|-------------------------|
> > | parent | 9.743M | baseline | ~0 |
> > | bad | 7.821M | -19.73% | ~11.16M |
> > | fix | 9.274M | -4.81% | ~0 |
> >
> > Probing the drain_obj_stock() calls confirms that the fix restores the
> > frequency to the parent's baseline.
> >
> > And it seems that besides __refill_obj_stock(), we should also modify
> > __consume_obj_stock()?
> >
>
> Thanks a lot Qi. I will send the formal patch and will add your Debugged-by if
> you don't mind.
>
Tested-by: kernel test robot <oliver.sang@xxxxxxxxx>
we tested above patch, and it recovers the regression:
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
682fd4e9ff <--- above patch from Shakeel
8285917d6f383aef 01b9da291c4969354807b52956f 682fd4e9ffd4009805f81dd25ed
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
5849 +210.2% 18145 ± 3% +1.5% 5935 stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% -1.4% 2.263e+09 stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% -1.4% 37739220 stress-ng.switch.ops_per_sec
full compasison is as below [3]
but there are two notes.
#1 is that we noticed there is a fomal patch later from Shakeel in [1] which has
more changes. not sure if this test is enough? do you want us to test [1]
further?
#2: when we test above patch, we found the server easy to crash while running
tests. we try to run up to 20 times, only 2 of them run successfully (above
37739220 is just the average data from these 2 runs, since the data is stable,
we think maybe it's ok to report to you with this data).
we also noticed for [1] there is a [syzbot ci] report in [2]. since we don't
have serial output for our test server in this report which is for performance
tests, we cannot say if other 18 runs failed due to similar reason. just FYI.
[1] https://lore.kernel.org/all/20260515171953.2224503-1-shakeel.butt@xxxxxxxxx/
[2] https://lore.kernel.org/all/6a081599.170a0220.4530d.0003.GAE@xxxxxxxxxx/
[3]
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
682fd4e9ff <--- above patch from Shakeel
8285917d6f383aef 01b9da291c4969354807b52956f 682fd4e9ffd4009805f81dd25ed
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
5849 +210.2% 18145 ± 3% +1.5% 5935 stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% -1.4% 2.263e+09 stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% -1.4% 37739220 stress-ng.switch.ops_per_sec
93416932 -68.6% 29310048 ± 3% +0.1% 93506345 stress-ng.time.involuntary_context_switches
15845 +11.0% 17584 +0.3% 15894 stress-ng.time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.5% 8597 stress-ng.time.system_time
963.36 -53.5% 447.72 ± 3% -1.3% 950.84 stress-ng.time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.5% 1.496e+09 stress-ng.time.voluntary_context_switches
1124 ± 17% +34.3% 1509 ± 8% -7.7% 1037 ± 14% perf-c2c.HITM.remote
2.55e+09 -12.3% 2.236e+09 ± 3% -2.3% 2.491e+09 cpuidle..time
8.29e+08 -71.8% 2.337e+08 ± 2% -1.8% 8.143e+08 cpuidle..usage
14184409 ± 2% -16.4% 11860068 +0.5% 14255389 ± 2% vmstat.memory.cache
39204964 -69.7% 11868752 ± 2% -1.2% 38715052 vmstat.system.cs
1808848 -38.5% 1111830 -0.8% 1793867 vmstat.system.in
115757 ± 52% +8.5% 125625 ± 44% -62.0% 44014 ± 32% numa-numastat.node0.other_node
4102960 ± 5% -19.0% 3324393 ± 4% -1.0% 4063575 ± 2% numa-numastat.node1.local_node
4218983 ± 3% -18.7% 3430325 ± 3% +7.3% 4526769 ± 3% numa-numastat.node1.numa_hit
116023 ± 52% -8.7% 105932 ± 52% +62.0% 187903 ± 7% numa-numastat.node1.other_node
93416932 -68.6% 29310048 ± 3% +0.1% 93506345 time.involuntary_context_switches
15845 +11.0% 17584 +0.3% 15894 time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.5% 8597 time.system_time
963.36 -53.5% 447.72 ± 3% -1.3% 950.84 time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.5% 1.496e+09 time.voluntary_context_switches
22.48 -4.4 18.08 -0.2 22.29 mpstat.cpu.all.idle%
1.13 -0.4 0.73 -0.0 1.12 mpstat.cpu.all.irq%
0.10 -0.0 0.09 -0.0 0.10 mpstat.cpu.all.soft%
67.98 +9.1 77.06 +0.3 68.29 mpstat.cpu.all.sys%
8.32 -4.3 4.04 ± 2% -0.1 8.21 mpstat.cpu.all.usr%
17.33 ± 2% +15.4% 20.00 ± 4% +3.8% 18.00 mpstat.max_utilization.seconds
10552401 ± 4% -23.3% 8092823 +0.3% 10586331 ± 6% numa-meminfo.node1.Active
10552392 ± 4% -23.3% 8092820 +0.3% 10586323 ± 6% numa-meminfo.node1.Active(anon)
12454155 ± 15% -34.9% 8106052 -3.6% 12008629 ± 16% numa-meminfo.node1.FilePages
559046 ± 8% -19.2% 451929 ± 2% -2.4% 545736 ± 10% numa-meminfo.node1.Mapped
14688311 ± 13% -30.0% 10285394 ± 2% -4.7% 14004338 ± 15% numa-meminfo.node1.MemUsed
10028979 ± 3% -22.4% 7783864 +0.8% 10109957 ± 3% numa-meminfo.node1.Shmem
10939677 ± 2% -21.0% 8641166 +0.5% 10995134 ± 2% meminfo.Active
10939661 ± 2% -21.0% 8641149 +0.5% 10995117 ± 2% meminfo.Active(anon)
13917673 ± 2% -16.4% 11633722 +0.4% 13973077 ± 2% meminfo.Cached
14400924 ± 2% -16.0% 12102150 +0.6% 14482134 ± 2% meminfo.Committed_AS
8394752 ± 5% +16.7% 9796949 ± 8% +10.5% 9274368 ± 16% meminfo.DirectMap2M
617671 -12.0% 543559 -1.5% 608569 meminfo.Mapped
18364992 -12.5% 16065468 +0.3% 18426986 meminfo.Memused
10124702 ± 2% -22.6% 7839682 +0.6% 10181038 ± 3% meminfo.Shmem
18393665 -12.5% 16100473 +0.4% 18458236 meminfo.max_used_kB
115757 ± 52% +8.5% 125625 ± 44% -62.0% 44014 ± 32% numa-vmstat.node0.numa_other
2638537 ± 4% -23.3% 2022639 +0.3% 2647116 ± 6% numa-vmstat.node1.nr_active_anon
3113944 ± 15% -34.9% 2025946 -3.6% 3002667 ± 16% numa-vmstat.node1.nr_file_pages
139848 ± 9% -19.3% 112912 ± 2% -2.4% 136548 ± 10% numa-vmstat.node1.nr_mapped
2507650 ± 3% -22.4% 1945399 +0.8% 2527999 ± 3% numa-vmstat.node1.nr_shmem
2638531 ± 4% -23.3% 2022634 +0.3% 2647111 ± 6% numa-vmstat.node1.nr_zone_active_anon
4219206 ± 3% -18.7% 3430093 ± 3% +7.3% 4527044 ± 3% numa-vmstat.node1.numa_hit
4103183 ± 4% -19.0% 3324161 ± 4% -1.0% 4063850 ± 2% numa-vmstat.node1.numa_local
116023 ± 52% -8.7% 105932 ± 52% +62.0% 187903 ± 7% numa-vmstat.node1.numa_other
10.59 -7.6 2.97 ± 8% -0.0 10.58 turbostat.C1%
0.85 ± 3% +9.1 9.96 ± 2% +0.1 0.90 turbostat.C1E%
1.29 ± 6% +19.4% 1.54 ± 2% +7.8% 1.39 turbostat.CPU%c1
48.67 ± 2% -15.1% 41.33 ± 3% -14.7% 41.50 turbostat.CoreTmp
0.56 -60.7% 0.22 ± 3% +1.8% 0.57 turbostat.IPC
1.153e+08 -38.7% 70680365 -1.1% 1.14e+08 turbostat.IRQ
10242404 -14.8% 8723704 -0.2% 10218332 turbostat.NMI
88.65 -84.0 4.67 ± 33% -2.4 86.25 ± 2% turbostat.PKG_%
3.82 -3.8 0.04 ± 10% -0.1 3.68 turbostat.POLL%
48.67 ± 2% -13.7% 42.00 ± 3% -12.7% 42.50 turbostat.PkgTmp
683.77 -13.1% 594.00 -0.1% 683.24 turbostat.PkgWatt
18.74 -3.3% 18.13 -1.0% 18.54 turbostat.RAMWatt
2735312 ± 2% -21.0% 2160742 +0.5% 2749182 ± 2% proc-vmstat.nr_active_anon
204708 -1.6% 201435 -0.1% 204401 proc-vmstat.nr_anon_pages
3479812 ± 2% -16.4% 2908863 +0.4% 3493660 ± 2% proc-vmstat.nr_file_pages
154477 -12.0% 135959 -1.5% 152200 proc-vmstat.nr_mapped
2531568 ± 2% -22.6% 1960353 +0.6% 2545651 ± 3% proc-vmstat.nr_shmem
42010 -3.5% 40543 +0.0% 42030 proc-vmstat.nr_slab_reclaimable
2735312 ± 2% -21.0% 2160742 +0.5% 2749182 ± 2% proc-vmstat.nr_zone_active_anon
210167 ± 5% -11.5% 185950 ± 11% +7.6% 226044 ± 3% proc-vmstat.numa_hint_faults
198174 ± 6% -7.8% 182781 ± 11% +12.5% 222958 ± 4% proc-vmstat.numa_hint_faults_local
4730338 ± 2% -18.2% 3871343 +6.3% 5030657 ± 3% proc-vmstat.numa_hit
4498551 ± 2% -19.1% 3639783 +0.6% 4523449 ± 2% proc-vmstat.numa_local
22013 ± 74% -11.9% 19394 ± 79% -88.0% 2632 ± 2% proc-vmstat.numa_pages_migrated
4808959 ± 2% -17.8% 3954157 +0.3% 4821123 ± 2% proc-vmstat.pgalloc_normal
806619 -5.1% 765525 ± 2% +1.1% 815270 proc-vmstat.pgfault
467932 ± 3% -0.3% 466368 ± 2% -5.2% 443691 proc-vmstat.pgfree
22013 ± 74% -11.9% 19394 ± 79% -88.0% 2632 ± 2% proc-vmstat.pgmigrate_success
34098 ± 3% -14.8% 29054 -8.8% 31100 ± 4% proc-vmstat.pgreuse
0.11 +59.9% 0.17 ± 3% -3.8% 0.10 ± 2% perf-stat.i.MPKI
6.653e+10 -61.7% 2.546e+10 ± 2% +1.7% 6.767e+10 perf-stat.i.branch-instructions
0.76 +0.1 0.89 +0.0 0.78 perf-stat.i.branch-miss-rate%
4.685e+08 -59.7% 1.888e+08 ± 2% +4.8% 4.911e+08 perf-stat.i.branch-misses
1.12 +0.6 1.76 ± 3% +0.0 1.12 perf-stat.i.cache-miss-rate%
35553724 ± 3% -40.4% 21188697 -0.9% 35218701 perf-stat.i.cache-misses
4.194e+09 -68.3% 1.331e+09 ± 2% -1.3% 4.139e+09 perf-stat.i.cache-references
40710745 -69.6% 12395879 ± 2% -1.5% 40082554 perf-stat.i.context-switches
1.84 +189.1% 5.31 ± 2% -1.7% 1.81 perf-stat.i.cpi
5.965e+11 -2.0% 5.848e+11 -0.1% 5.962e+11 perf-stat.i.cpu-cycles
8813175 -64.5% 3125097 ± 2% -2.0% 8632551 perf-stat.i.cpu-migrations
24447 ± 3% +68.5% 41184 ± 2% +2.6% 25087 perf-stat.i.cycles-between-cache-misses
3.374e+11 -61.8% 1.287e+11 ± 2% +1.5% 3.426e+11 perf-stat.i.instructions
0.57 -60.8% 0.22 ± 2% +1.6% 0.58 perf-stat.i.ipc
0.01 ±141% +185.0% 0.04 ± 46% +280.6% 0.05 ± 20% perf-stat.i.major-faults
221.10 -68.6% 69.32 ± 2% -1.6% 217.48 perf-stat.i.metric.K/sec
11782 ± 3% -6.1% 11068 ± 3% -1.0% 11660 perf-stat.i.minor-faults
11782 ± 3% -6.1% 11068 ± 3% -1.0% 11660 perf-stat.i.page-faults
0.10 ± 2% +59.2% 0.17 ± 3% -3.0% 0.10 perf-stat.overall.MPKI
0.71 +0.0 0.75 +0.0 0.73 perf-stat.overall.branch-miss-rate%
0.83 ± 3% +0.7 1.56 ± 3% -0.0 0.82 perf-stat.overall.cache-miss-rate%
1.78 +162.2% 4.67 ± 2% -1.4% 1.76 perf-stat.overall.cpi
17181 ± 3% +64.6% 28283 +1.6% 17452 perf-stat.overall.cycles-between-cache-misses
0.56 -61.8% 0.21 ± 2% +1.4% 0.57 perf-stat.overall.ipc
6.388e+10 -62.3% 2.409e+10 ± 2% +1.6% 6.492e+10 perf-stat.ps.branch-instructions
4.538e+08 -60.0% 1.817e+08 ± 2% +5.0% 4.764e+08 perf-stat.ps.branch-misses
33674051 ± 3% -40.1% 20155290 -1.6% 33143601 perf-stat.ps.cache-misses
4.077e+09 -68.2% 1.296e+09 ± 2% -1.1% 4.032e+09 perf-stat.ps.cache-references
39570629 -69.5% 12072702 ± 2% -1.4% 39036286 perf-stat.ps.context-switches
5.78e+11 -1.4% 5.7e+11 +0.1% 5.784e+11 perf-stat.ps.cpu-cycles
8584979 -64.5% 3051930 ± 2% -1.8% 8430431 perf-stat.ps.cpu-migrations
3.243e+11 -62.4% 1.22e+11 ± 2% +1.5% 3.291e+11 perf-stat.ps.instructions
0.01 ±141% +195.1% 0.03 ± 41% +270.3% 0.04 ± 20% perf-stat.ps.major-faults
11022 ± 4% -6.5% 10300 ± 3% -0.1% 11015 perf-stat.ps.minor-faults
11022 ± 4% -6.5% 10300 ± 3% -0.1% 11015 perf-stat.ps.page-faults
1.941e+13 -61.9% 7.405e+12 ± 3% +2.5% 1.989e+13 perf-stat.total.instructions
18451 +9.9% 20272 +2.2% 18858 ± 4% sched_debug.cfs_rq:/.avg_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% +10.3% 6472 ± 19% sched_debug.cfs_rq:/.avg_vruntime.stddev
0.68 ± 2% -12.9% 0.59 ± 3% +0.6% 0.68 sched_debug.cfs_rq:/.h_nr_queued.stddev
0.62 ± 6% -12.9% 0.54 ± 2% +0.4% 0.62 sched_debug.cfs_rq:/.h_nr_runnable.stddev
8469 +12.7% 9544 ± 2% +3.0% 8727 ± 4% sched_debug.cfs_rq:/.left_deadline.stddev
8467 +12.7% 9542 ± 2% +3.1% 8727 ± 4% sched_debug.cfs_rq:/.left_vruntime.stddev
3513124 ± 25% -30.0% 2459550 ± 10% -2.7% 3419965 ± 22% sched_debug.cfs_rq:/.load.max
588329 ± 5% -11.2% 522578 ± 5% +4.1% 612170 sched_debug.cfs_rq:/.load.stddev
50699 ± 17% -19.8% 40655 ± 7% +20.0% 60854 ± 36% sched_debug.cfs_rq:/.load_avg.max
0.68 ± 2% -12.9% 0.59 ± 3% +0.9% 0.68 sched_debug.cfs_rq:/.nr_queued.stddev
38.80 ± 32% +108.5% 80.90 ± 16% +276.0% 145.88 ± 64% sched_debug.cfs_rq:/.removed.load_avg.avg
857.83 ± 12% +61.0% 1381 ± 12% +2561.3% 22829 ± 96% sched_debug.cfs_rq:/.removed.load_avg.max
152.02 ± 18% +57.2% 239.02 ± 11% +955.1% 1604 ± 88% sched_debug.cfs_rq:/.removed.load_avg.stddev
26.08 ± 28% +143.0% 63.37 ± 14% +23.5% 32.20 ± 9% sched_debug.cfs_rq:/.removed.runnable_avg.avg
547.00 ± 13% +88.7% 1032 ± 12% +6.4% 582.00 sched_debug.cfs_rq:/.removed.runnable_avg.max
94.86 ± 17% +84.3% 174.82 ± 9% +19.2% 113.04 ± 2% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
9.09 ± 52% +253.3% 32.11 ± 17% +53.1% 13.91 ± 6% sched_debug.cfs_rq:/.removed.util_avg.avg
275.17 ± 3% +130.3% 633.67 ± 9% +0.0% 275.25 sched_debug.cfs_rq:/.removed.util_avg.max
44.90 ± 30% +126.4% 101.66 ± 11% +30.9% 58.78 ± 2% sched_debug.cfs_rq:/.removed.util_avg.stddev
8467 +12.7% 9542 ± 2% +3.1% 8727 ± 4% sched_debug.cfs_rq:/.right_vruntime.stddev
659.63 ± 3% +13.0% 745.47 +0.6% 663.51 sched_debug.cfs_rq:/.runnable_avg.avg
271.34 ± 2% +31.2% 355.98 ± 3% +0.8% 273.61 sched_debug.cfs_rq:/.runnable_avg.stddev
0.00 ± 26% +110.4% 0.00 ± 45% +3.3% 0.00 ± 21% sched_debug.cfs_rq:/.spread.avg
0.01 ± 13% +174.3% 0.02 ± 25% -36.7% 0.00 sched_debug.cfs_rq:/.spread.max
0.00 ± 7% +146.2% 0.00 ± 27% -13.2% 0.00 ± 8% sched_debug.cfs_rq:/.spread.stddev
431.00 +14.5% 493.62 -1.9% 422.73 sched_debug.cfs_rq:/.util_avg.avg
1061 ± 3% +26.4% 1341 ± 3% -3.5% 1024 sched_debug.cfs_rq:/.util_avg.max
151.53 ± 5% +50.1% 227.46 ± 2% -8.0% 139.45 sched_debug.cfs_rq:/.util_avg.stddev
206.96 +17.5% 243.18 ± 3% +10.4% 228.53 sched_debug.cfs_rq:/.util_est.avg
18451 +9.9% 20272 +2.2% 18858 ± 4% sched_debug.cfs_rq:/.zero_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% +10.3% 6472 ± 19% sched_debug.cfs_rq:/.zero_vruntime.stddev
460133 ± 2% +2.0% 469231 +5.0% 483144 ± 5% sched_debug.cpu.avg_idle.avg
2345 +33.6% 3133 ± 5% +887.0% 23149 ± 88% sched_debug.cpu.avg_idle.min
13.18 ± 2% +39.8% 18.42 ± 6% +5.1% 13.85 ± 5% sched_debug.cpu.clock.stddev
3961 +14.6% 4541 +0.7% 3989 sched_debug.cpu.curr->pid.avg
3213 -15.4% 2718 -5.7% 3030 sched_debug.cpu.curr->pid.stddev
0.00 ± 29% +157.3% 0.00 ± 35% -9.3% 0.00 ± 26% sched_debug.cpu.next_balance.stddev
0.70 -15.8% 0.59 ± 3% -2.7% 0.68 sched_debug.cpu.nr_running.stddev
5474800 -69.7% 1660250 ± 2% -1.5% 5392540 sched_debug.cpu.nr_switches.avg
5648642 -65.5% 1946319 ± 5% -1.3% 5576589 sched_debug.cpu.nr_switches.max
2229198 ± 8% -67.1% 734011 ± 20% +11.1% 2476939 ± 5% sched_debug.cpu.nr_switches.min
297592 ± 6% -25.9% 220513 ± 18% -5.6% 281029 ± 8% sched_debug.cpu.nr_switches.stddev
27.83 ± 30% -24.0% 21.17 ± 17% +33.8% 37.25 ± 14% sched_debug.cpu.nr_uninterruptible.max
23.75 -10.9 12.88 -0.3 23.42 perf-profile.calltrace.cycles-pp.common_startup_64
23.65 -10.8 12.82 -0.3 23.32 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
23.62 -10.8 12.81 -0.3 23.28 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
23.51 -10.8 12.76 -0.3 23.18 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
12.93 -7.0 5.94 ± 4% -0.1 12.82 perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.78 -6.9 5.89 ± 4% -0.1 12.66 perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
11.30 -5.2 6.07 ± 3% -0.1 11.22 perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.17 -5.2 6.02 ± 3% -0.1 11.10 perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
9.89 -4.8 5.08 ± 4% -0.1 9.76 perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.52 -4.5 0.00 -0.1 4.39 perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
12.41 -4.4 7.96 -0.2 12.20 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
9.19 -4.4 4.77 ± 4% -0.1 9.06 perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up
11.29 -4.0 7.29 -0.2 11.11 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
11.38 -4.0 7.40 -0.2 11.20 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
8.04 -3.9 4.13 ± 4% -0.1 7.93 perf-profile.calltrace.cycles-pp.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq
6.91 -3.8 3.08 ± 4% -0.1 6.80 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
6.76 -3.8 3.00 ± 4% -0.1 6.66 perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive
8.71 -3.7 5.00 -0.0 8.67 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.71 -3.4 2.26 ± 4% -0.1 5.64 perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
8.12 -3.4 4.72 -0.0 8.08 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
7.76 -3.2 4.55 -0.0 7.72 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend
8.39 -3.1 5.26 -0.0 8.35 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.47 -3.1 4.35 -0.0 7.44 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend
4.92 -2.9 2.00 ± 4% -0.1 4.86 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
7.79 -2.8 4.97 -0.0 7.74 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
7.48 -2.7 4.79 -0.0 7.44 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive
7.17 -2.6 4.60 -0.0 7.13 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive
5.54 -2.3 3.24 ± 4% -0.1 5.48 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
5.41 -2.2 3.16 ± 4% -0.1 5.35 perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend
3.88 -2.2 1.67 ± 4% -0.1 3.82 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
4.04 -2.2 1.88 -0.0 3.99 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
3.44 -2.2 1.28 ± 3% -0.1 3.38 perf-profile.calltrace.cycles-pp.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.84 -2.0 1.80 -0.0 3.79 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
4.83 -2.0 2.85 -0.0 4.80 perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
4.71 -1.9 2.78 -0.0 4.68 perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock
1.84 -1.8 0.00 -0.0 1.81 perf-profile.calltrace.cycles-pp.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.39 -1.8 2.58 -0.0 4.36 perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
2.37 -1.5 0.84 ± 3% -0.0 2.34 perf-profile.calltrace.cycles-pp._copy_to_user.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.66 -1.4 1.26 ± 4% -0.0 2.62 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
2.73 -1.4 1.33 ± 4% -0.0 2.70 perf-profile.calltrace.cycles-pp.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.50 -1.3 2.17 -0.0 3.49 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
2.37 -1.3 1.06 ± 2% -0.0 2.32 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.23 -1.2 0.00 -0.0 1.20 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule_idle.do_idle.cpu_startup_entry
1.15 -1.2 0.00 -0.0 1.13 perf-profile.calltrace.cycles-pp.task_h_load.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up
1.15 -1.1 0.00 -0.0 1.12 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule_idle.do_idle
4.10 -1.1 3.03 +0.0 4.10 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
2.20 -1.1 1.13 ± 5% -0.0 2.16 perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
2.20 -1.1 1.15 ± 4% -0.0 2.18 perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.05 -1.0 0.00 +0.0 1.06 perf-profile.calltrace.cycles-pp.set_next_task_idle.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
1.21 -1.0 0.18 ±141% -0.0 1.20 perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
1.20 -1.0 0.17 ±141% -0.0 1.19 perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedsend
1.74 -1.0 0.73 ± 3% +0.0 1.77 perf-profile.calltrace.cycles-pp.msg_get.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 -1.0 0.00 +0.0 1.00 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.98 -1.0 0.00 +0.0 1.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.32 -1.0 0.37 ± 70% -0.0 1.32 perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive
1.93 -0.9 0.99 ± 4% -0.0 1.89 perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
1.34 -0.8 0.54 ± 5% -0.0 1.34 perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
1.52 -0.8 0.73 ± 4% -0.0 1.50 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending
0.74 -0.7 0.00 -0.0 0.73 perf-profile.calltrace.cycles-pp.__smp_call_single_queue.ttwu_queue_wakelist.try_to_wake_up.wake_up_q.do_mq_timedsend
1.05 -0.7 0.34 ± 70% -0.0 1.05 perf-profile.calltrace.cycles-pp.__switch_to
0.71 -0.7 0.00 -0.0 0.68 perf-profile.calltrace.cycles-pp.msg_insert.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.57 -0.7 0.90 ± 6% -0.0 1.56 perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.67 -0.7 0.00 -0.0 0.64 perf-profile.calltrace.cycles-pp.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.66 -0.7 0.00 -0.0 0.65 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate
0.65 -0.6 0.00 -0.0 0.64 perf-profile.calltrace.cycles-pp.___task_rq_lock.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
0.60 -0.6 0.00 -0.0 0.58 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
0.56 -0.6 0.00 -0.0 0.55 perf-profile.calltrace.cycles-pp._raw_spin_lock.raw_spin_rq_lock_nested.___task_rq_lock.try_to_wake_up.wake_up_q
0.54 -0.5 0.00 -0.0 0.53 perf-profile.calltrace.cycles-pp.os_xsave
0.52 -0.5 0.00 -0.0 0.51 perf-profile.calltrace.cycles-pp.__switch_to_asm
2.56 -0.3 2.28 -0.0 2.54 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
0.00 +0.0 0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.__check_object_size.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
5.87 +0.9 6.78 -0.0 5.86 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +0.9 0.91 ± 30% +0.0 0.00 perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
35.80 +3.5 39.29 +0.1 35.92 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
35.59 +3.6 39.21 +0.1 35.70 perf-profile.calltrace.cycles-pp.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.4 4.37 ± 3% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg
0.00 +4.4 4.39 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg
0.00 +8.0 8.01 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend
0.00 +8.3 8.29 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive
28.51 +13.5 41.99 +0.3 28.81 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
28.23 +13.5 41.71 +0.3 28.53 perf-profile.calltrace.cycles-pp.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
70.69 +13.7 84.35 +0.3 71.02 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
70.44 +13.8 84.26 +0.3 70.77 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.99 +20.2 23.24 ± 2% +0.4 3.35 perf-profile.calltrace.cycles-pp.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.79 +20.4 23.15 ± 2% +0.4 3.16 perf-profile.calltrace.cycles-pp.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.43 +20.6 22.98 ± 2% +0.4 2.80 perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive
2.26 +26.0 28.23 ± 2% +0.6 2.90 perf-profile.calltrace.cycles-pp.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 +26.8 27.80 ± 2% +0.6 1.60 perf-profile.calltrace.cycles-pp.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
0.65 +27.0 27.62 ± 2% +0.6 1.24 perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend
24.25 -12.2 12.02 ± 4% -0.2 24.05 perf-profile.children.cycles-pp.wake_up_q
24.00 -12.1 11.93 ± 4% -0.2 23.81 perf-profile.children.cycles-pp.try_to_wake_up
23.75 -10.9 12.88 -0.3 23.42 perf-profile.children.cycles-pp.common_startup_64
23.75 -10.9 12.88 -0.3 23.42 perf-profile.children.cycles-pp.cpu_startup_entry
23.68 -10.8 12.85 -0.3 23.34 perf-profile.children.cycles-pp.do_idle
23.65 -10.8 12.82 -0.3 23.32 perf-profile.children.cycles-pp.start_secondary
19.65 -8.3 11.33 -0.1 19.54 perf-profile.children.cycles-pp.__schedule
17.14 -6.9 10.28 -0.1 17.06 perf-profile.children.cycles-pp.wq_sleep
16.24 -6.4 9.82 -0.1 16.18 perf-profile.children.cycles-pp.schedule
15.92 -6.2 9.69 -0.1 15.84 perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
12.46 -6.1 6.32 ± 4% -0.2 12.30 perf-profile.children.cycles-pp.select_task_rq
12.19 -6.0 6.17 ± 4% -0.2 12.03 perf-profile.children.cycles-pp.select_task_rq_fair
9.91 -4.8 5.09 ± 4% -0.1 9.78 perf-profile.children.cycles-pp.select_idle_sibling
4.57 -4.6 0.02 ±141% -0.1 4.44 perf-profile.children.cycles-pp.poll_idle
9.27 -4.5 4.80 ± 4% -0.1 9.14 perf-profile.children.cycles-pp.select_idle_cpu
12.49 -4.5 8.03 -0.2 12.28 perf-profile.children.cycles-pp.cpuidle_idle_call
11.41 -4.0 7.39 -0.2 11.23 perf-profile.children.cycles-pp.cpuidle_enter_state
11.44 -4.0 7.43 -0.2 11.26 perf-profile.children.cycles-pp.cpuidle_enter
8.17 -4.0 4.18 ± 4% -0.1 8.06 perf-profile.children.cycles-pp.select_idle_core
5.76 -3.5 2.28 ± 4% -0.1 5.70 perf-profile.children.cycles-pp.flush_smp_call_function_queue
5.47 -3.1 2.35 ± 4% -0.1 5.40 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
4.30 -2.4 1.94 ± 4% -0.1 4.24 perf-profile.children.cycles-pp.sched_ttwu_pending
4.08 -2.2 1.90 -0.0 4.03 perf-profile.children.cycles-pp.schedule_idle
3.47 -2.2 1.30 ± 2% -0.1 3.42 perf-profile.children.cycles-pp.store_msg
5.87 -2.1 3.78 -0.0 5.86 perf-profile.children.cycles-pp.__pick_next_task
4.84 -2.0 2.86 -0.0 4.81 perf-profile.children.cycles-pp.try_to_block_task
4.73 -1.9 2.79 -0.0 4.69 perf-profile.children.cycles-pp.dequeue_entities
4.73 -1.9 2.82 -0.0 4.70 perf-profile.children.cycles-pp.dequeue_task_fair
4.04 -1.8 2.26 -0.0 4.00 perf-profile.children.cycles-pp._raw_spin_lock
3.72 -1.7 2.03 ± 4% -0.0 3.70 perf-profile.children.cycles-pp.enqueue_task
2.40 -1.6 0.85 ± 3% -0.0 2.37 perf-profile.children.cycles-pp._copy_to_user
3.39 -1.5 1.86 ± 3% -0.0 3.36 perf-profile.children.cycles-pp.ttwu_do_activate
2.56 -1.5 1.04 ± 5% -0.0 2.54 perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
2.54 -1.5 1.03 ± 5% -0.0 2.52 perf-profile.children.cycles-pp.do_perf_trace_sched_wakeup_template
2.76 -1.4 1.34 ± 4% -0.0 2.73 perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare
3.75 -1.4 2.34 -0.0 3.74 perf-profile.children.cycles-pp.dequeue_entity
2.32 -1.4 0.95 ± 4% -0.0 2.30 perf-profile.children.cycles-pp.ttwu_queue_wakelist
2.72 -1.3 1.38 ± 2% -0.0 2.72 perf-profile.children.cycles-pp.update_curr
2.38 -1.3 1.06 ± 2% -0.1 2.32 perf-profile.children.cycles-pp.exit_to_user_mode_loop
4.24 -1.2 2.99 -0.0 4.21 perf-profile.children.cycles-pp.pick_next_task_fair
2.21 -1.1 1.15 ± 5% -0.0 2.19 perf-profile.children.cycles-pp.switch_fpu_return
1.76 -1.0 0.73 ± 3% +0.0 1.80 perf-profile.children.cycles-pp.msg_get
1.67 -1.0 0.65 ± 3% -0.1 1.54 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.47 -1.0 1.47 ± 3% -0.0 2.44 perf-profile.children.cycles-pp.enqueue_task_fair
2.29 -1.0 1.32 -0.0 2.25 perf-profile.children.cycles-pp.update_load_avg
2.39 -1.0 1.42 -0.0 2.37 perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
1.60 -1.0 0.64 -0.0 1.59 perf-profile.children.cycles-pp.switch_mm_irqs_off
1.51 -0.9 0.57 ± 5% -0.0 1.49 perf-profile.children.cycles-pp.__smp_call_single_queue
1.84 -0.9 0.93 ± 6% -0.0 1.82 perf-profile.children.cycles-pp.wake_affine
1.49 -0.9 0.59 +0.0 1.49 perf-profile.children.cycles-pp.__check_object_size
1.61 -0.9 0.72 ± 5% +0.0 1.62 perf-profile.children.cycles-pp.update_rq_clock_task
1.73 -0.9 0.87 ± 2% -0.0 1.73 perf-profile.children.cycles-pp.update_se
1.51 -0.9 0.65 ± 2% +0.0 1.53 perf-profile.children.cycles-pp.wakeup_preempt
2.00 -0.8 1.15 ± 3% -0.0 1.97 perf-profile.children.cycles-pp.enqueue_entity
1.26 -0.8 0.42 ± 3% -0.1 1.14 perf-profile.children.cycles-pp.__wake_up
1.31 -0.7 0.58 ± 5% -0.0 1.30 perf-profile.children.cycles-pp.set_task_cpu
1.15 -0.7 0.45 ± 3% -0.0 1.10 perf-profile.children.cycles-pp.msg_insert
1.04 -0.7 0.35 ± 5% -0.0 1.04 perf-profile.children.cycles-pp.perf_trace_buf_alloc
1.57 -0.7 0.90 ± 5% -0.0 1.56 perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
1.00 -0.7 0.34 ± 7% -0.0 1.00 perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.95 -0.7 0.29 ± 4% -0.0 0.94 perf-profile.children.cycles-pp.llist_reverse_order
1.14 -0.6 0.50 ± 5% -0.0 1.13 perf-profile.children.cycles-pp.migrate_task_rq_fair
1.11 -0.6 0.51 -0.0 1.10 perf-profile.children.cycles-pp.set_next_entity
0.95 -0.6 0.39 ± 2% +0.0 0.96 perf-profile.children.cycles-pp.__update_idle_core
1.04 -0.6 0.49 ± 2% -0.0 1.03 perf-profile.children.cycles-pp.pick_task_fair
1.15 -0.6 0.60 ± 7% -0.0 1.14 perf-profile.children.cycles-pp.task_h_load
0.84 -0.5 0.30 ± 5% -0.0 0.84 perf-profile.children.cycles-pp.call_function_single_prep_ipi
1.06 -0.5 0.52 +0.0 1.07 perf-profile.children.cycles-pp.set_next_task_idle
1.04 -0.5 0.51 -0.0 1.03 perf-profile.children.cycles-pp._find_next_bit
1.38 -0.5 0.85 -0.0 1.36 perf-profile.children.cycles-pp.update_cfs_rq_load_avg
1.28 -0.5 0.75 -0.0 1.27 perf-profile.children.cycles-pp.__switch_to
0.85 -0.5 0.34 ± 3% +0.0 0.86 perf-profile.children.cycles-pp.cpuacct_charge
0.77 ± 4% -0.5 0.25 -0.0 0.73 perf-profile.children.cycles-pp.__bitmap_andnot
0.88 -0.5 0.41 ± 6% +0.0 0.89 perf-profile.children.cycles-pp.update_entity_lag
0.94 -0.5 0.48 -0.0 0.92 perf-profile.children.cycles-pp.prepare_task_switch
0.74 -0.4 0.31 ± 2% +0.0 0.76 perf-profile.children.cycles-pp.check_heap_object
0.75 -0.4 0.32 ± 5% +0.0 0.75 perf-profile.children.cycles-pp.requeue_delayed_entity
0.80 -0.4 0.40 -0.0 0.80 perf-profile.children.cycles-pp.wakeup_preempt_fair
0.55 -0.4 0.16 ± 2% -0.0 0.54 perf-profile.children.cycles-pp.native_sched_clock
0.57 -0.4 0.18 ± 4% -0.0 0.56 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.55 -0.4 0.17 -0.0 0.54 perf-profile.children.cycles-pp.os_xsave
0.58 -0.4 0.20 -0.0 0.57 perf-profile.children.cycles-pp.sched_clock_cpu
1.39 -0.4 1.01 -0.0 1.36 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.18 -0.4 0.81 -0.0 1.18 perf-profile.children.cycles-pp.___task_rq_lock
0.51 ± 3% -0.4 0.14 ± 3% -0.0 0.50 perf-profile.children.cycles-pp._copy_from_user
0.56 -0.4 0.19 ± 2% -0.0 0.54 perf-profile.children.cycles-pp.update_rq_clock
0.51 -0.3 0.17 ± 2% -0.0 0.50 perf-profile.children.cycles-pp.sched_clock
0.56 -0.3 0.23 ± 3% +0.0 0.58 perf-profile.children.cycles-pp.simple_inode_init_ts
0.89 -0.3 0.56 +0.0 0.89 perf-profile.children.cycles-pp.put_prev_entity
0.53 ± 3% -0.3 0.21 ± 2% +0.0 0.53 perf-profile.children.cycles-pp.__put_user_4
0.68 ± 12% -0.3 0.36 ± 4% +0.1 0.76 perf-profile.children.cycles-pp.stress_switch_mq
0.52 -0.3 0.20 -0.0 0.50 perf-profile.children.cycles-pp.set_next_task_fair
0.55 -0.3 0.24 ± 5% -0.0 0.54 perf-profile.children.cycles-pp.__switch_to_asm
0.51 -0.3 0.21 ± 3% +0.0 0.52 perf-profile.children.cycles-pp.inode_set_ctime_current
0.56 -0.3 0.26 ± 3% +0.0 0.57 perf-profile.children.cycles-pp.fdget
0.52 -0.3 0.23 ± 3% -0.0 0.51 perf-profile.children.cycles-pp.mm_cid_switch_to
0.54 -0.3 0.25 ± 3% -0.0 0.54 perf-profile.children.cycles-pp.remove_entity_load_avg
0.79 -0.3 0.51 -0.0 0.78 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.37 -0.3 0.11 ± 7% +0.0 0.37 perf-profile.children.cycles-pp.__resched_curr
0.57 -0.2 0.32 -0.0 0.56 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.35 -0.2 0.12 ± 4% -0.0 0.35 perf-profile.children.cycles-pp.avg_vruntime
0.62 -0.2 0.39 ± 2% -0.0 0.62 perf-profile.children.cycles-pp.sysvec_call_function_single
0.34 -0.2 0.11 -0.0 0.33 perf-profile.children.cycles-pp.entry_SYSCALL_64
0.58 -0.2 0.36 +0.0 0.59 perf-profile.children.cycles-pp.__enqueue_entity
0.46 -0.2 0.24 -0.0 0.45 perf-profile.children.cycles-pp.__pick_eevdf
0.52 -0.2 0.30 ± 3% +0.0 0.52 perf-profile.children.cycles-pp.perf_tp_event
0.35 -0.2 0.14 ± 3% -0.0 0.35 perf-profile.children.cycles-pp.__wrgsbase_inactive
0.56 -0.2 0.36 -0.0 0.56 perf-profile.children.cycles-pp.__sysvec_call_function_single
0.33 -0.2 0.13 ± 3% -0.0 0.32 perf-profile.children.cycles-pp.__update_load_avg_se
0.31 ± 2% -0.2 0.12 ± 4% +0.0 0.33 perf-profile.children.cycles-pp.__virt_addr_valid
0.58 -0.2 0.40 -0.0 0.58 perf-profile.children.cycles-pp.menu_select
0.32 ± 5% -0.2 0.13 -0.0 0.29 ± 3% perf-profile.children.cycles-pp.__check_heap_object
0.31 -0.2 0.13 ± 3% -0.0 0.30 perf-profile.children.cycles-pp.place_entity
0.32 -0.2 0.14 -0.0 0.32 perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.36 -0.2 0.18 ± 2% -0.0 0.36 perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.29 -0.2 0.11 -0.0 0.28 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.33 -0.2 0.16 ± 3% -0.0 0.32 perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
0.35 -0.2 0.18 ± 2% +0.0 0.36 perf-profile.children.cycles-pp.ktime_get
0.25 -0.2 0.08 ± 5% -0.0 0.25 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.37 -0.2 0.20 ± 4% -0.0 0.36 perf-profile.children.cycles-pp.___perf_sw_event
0.23 -0.2 0.07 ± 6% -0.0 0.22 ± 2% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.22 -0.2 0.07 -0.0 0.22 ± 2% perf-profile.children.cycles-pp.read_tsc
0.21 -0.1 0.06 ± 7% -0.0 0.20 ± 2% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.25 -0.1 0.11 ± 7% -0.0 0.25 perf-profile.children.cycles-pp.strnlen
0.16 -0.1 0.02 ±141% -0.0 0.15 perf-profile.children.cycles-pp.ct_idle_exit
0.32 -0.1 0.18 -0.0 0.31 perf-profile.children.cycles-pp.__dequeue_entity
0.23 ± 2% -0.1 0.10 ± 4% +0.0 0.24 ± 2% perf-profile.children.cycles-pp.check_stack_object
0.53 -0.1 0.40 ± 2% -0.0 0.52 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.23 ± 2% -0.1 0.09 ± 5% -0.0 0.22 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.30 ± 3% -0.1 0.17 ± 4% -0.0 0.30 perf-profile.children.cycles-pp.tick_nohz_handler
0.20 -0.1 0.07 +0.4 0.57 ± 3% perf-profile.children.cycles-pp.__account_obj_stock
0.13 -0.1 0.00 -0.0 0.12 perf-profile.children.cycles-pp.ct_kernel_enter
0.13 -0.1 0.00 -0.0 0.12 perf-profile.children.cycles-pp.ct_kernel_exit_state
0.41 -0.1 0.28 ± 2% -0.0 0.40 perf-profile.children.cycles-pp.hrtimer_interrupt
0.31 ± 2% -0.1 0.18 ± 2% -0.0 0.30 perf-profile.children.cycles-pp.__hrtimer_run_queues
0.41 ± 2% -0.1 0.29 ± 3% -0.0 0.41 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.28 ± 2% -0.1 0.16 ± 6% -0.0 0.27 perf-profile.children.cycles-pp.update_process_times
0.33 -0.1 0.21 ± 6% +0.0 0.33 perf-profile.children.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.0 0.08 perf-profile.children.cycles-pp.__do_notify
0.19 ± 2% -0.1 0.07 ± 7% -0.0 0.18 perf-profile.children.cycles-pp.wake_q_add_safe
0.23 ± 2% -0.1 0.11 -0.0 0.22 perf-profile.children.cycles-pp.__kmalloc_cache_noprof
0.18 -0.1 0.07 ± 6% -0.0 0.18 perf-profile.children.cycles-pp.nohz_run_idle_balance
0.56 -0.1 0.46 -0.0 0.56 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.15 -0.1 0.05 +0.0 0.15 perf-profile.children.cycles-pp._raw_spin_unlock
0.17 -0.1 0.08 -0.0 0.16 perf-profile.children.cycles-pp.security_msg_msg_free
0.15 -0.1 0.06 +0.0 0.16 perf-profile.children.cycles-pp.inode_set_ctime_to_ts
0.08 ± 5% -0.1 0.00 +0.0 0.11 perf-profile.children.cycles-pp.trylock_stock
0.16 ± 2% -0.1 0.08 ± 5% -0.0 0.16 perf-profile.children.cycles-pp.dl_server_update
0.13 -0.1 0.06 ± 8% +0.0 0.13 perf-profile.children.cycles-pp.timestamp_truncate
0.15 ± 3% -0.1 0.08 +0.0 0.17 perf-profile.children.cycles-pp.perf_trace_buf_update
0.13 ± 3% -0.1 0.06 +0.0 0.13 perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
0.13 ± 3% -0.1 0.06 +0.0 0.13 perf-profile.children.cycles-pp.migrate_disable_switch
0.12 ± 6% -0.1 0.05 ± 8% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.11 ± 4% -0.1 0.05 -0.0 0.11 perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.06 -0.1 0.00 +0.0 0.07 perf-profile.children.cycles-pp.raw_spin_rq_unlock
0.14 -0.1 0.08 ± 5% +0.0 0.14 perf-profile.children.cycles-pp.update_curr_dl_se
0.06 ± 16% -0.1 0.00 +0.1 0.12 ± 4% perf-profile.children.cycles-pp.css_rstat_updated
0.10 -0.1 0.05 -0.0 0.10 perf-profile.children.cycles-pp.ct_kernel_exit
0.11 -0.1 0.06 +0.0 0.11 perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.10 ± 4% -0.0 0.05 +0.0 0.10 perf-profile.children.cycles-pp.__rb_insert_augmented
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.rest_init
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.x86_64_start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.x86_64_start_reservations
0.08 ± 10% -0.0 0.04 ± 71% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.mq_timedreceive
0.15 -0.0 0.11 ± 4% +0.0 0.15 perf-profile.children.cycles-pp.vruntime_eligible
0.13 -0.0 0.09 +0.0 0.13 perf-profile.children.cycles-pp.put_prev_task_fair
0.09 -0.0 0.05 ± 8% -0.0 0.09 perf-profile.children.cycles-pp.native_irq_return_iret
0.08 -0.0 0.05 +0.0 0.08 perf-profile.children.cycles-pp.choose_new_asid
0.13 -0.0 0.11 +0.0 0.13 perf-profile.children.cycles-pp.__irq_exit_rcu
0.07 -0.0 0.05 +0.0 0.07 perf-profile.children.cycles-pp.__set_next_task_fair
0.76 -0.0 0.74 -0.0 0.74 perf-profile.children.cycles-pp.finish_task_switch
0.09 -0.0 0.07 ± 6% -0.0 0.09 perf-profile.children.cycles-pp.propagate_entity_load_avg
0.10 -0.0 0.09 -0.0 0.10 perf-profile.children.cycles-pp.handle_softirqs
0.07 -0.0 0.06 +0.0 0.08 perf-profile.children.cycles-pp.clockevents_program_event
0.00 +0.0 0.00 +0.1 0.14 ± 3% perf-profile.children.cycles-pp.__mod_memcg_state
0.00 +0.0 0.00 +0.1 0.14 ± 3% perf-profile.children.cycles-pp.try_charge_memcg
0.00 +0.0 0.00 +0.3 0.32 ± 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.07 +0.0 0.08 +0.0 0.07 perf-profile.children.cycles-pp.perf_swevent_event
0.48 +0.0 0.49 +0.0 0.48 perf-profile.children.cycles-pp.process_simple
0.05 +0.0 0.06 ± 7% -0.0 0.05 perf-profile.children.cycles-pp.sched_update_worker
0.05 +0.0 0.07 -0.0 0.05 perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.07 ± 11% +0.0 0.09 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.mq_timedsend
0.15 ± 3% +0.0 0.20 ± 4% -0.0 0.15 perf-profile.children.cycles-pp.x64_sys_call
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.__sched_balance_update_blocked_averages
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ± 23% +0.0 0.00 perf-profile.children.cycles-pp.generic_perform_write
0.00 +0.1 0.06 ± 7% +0.0 0.00 perf-profile.children.cycles-pp.detach_tasks
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.shmem_file_write_iter
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.vfs_write
0.00 +0.1 0.07 ± 25% +0.0 0.00 perf-profile.children.cycles-pp.ksys_write
0.00 +0.1 0.08 ± 30% +0.0 0.00 perf-profile.children.cycles-pp.record__pushfn
0.04 ± 71% +0.1 0.12 ± 35% -0.0 0.03 ±100% perf-profile.children.cycles-pp.perf_mmap__push
0.54 ± 2% +0.1 0.62 ± 7% -0.0 0.54 ± 2% perf-profile.children.cycles-pp.cmd_record
0.04 ± 70% +0.1 0.13 ± 32% -0.0 0.03 ±100% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.handle_internal_command
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.main
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.run_builtin
0.10 ± 4% +0.1 0.20 ± 4% -0.0 0.10 perf-profile.children.cycles-pp.do_perf_trace_sched_switch
0.00 +0.1 0.10 ± 4% +0.0 0.00 perf-profile.children.cycles-pp.ct_idle_enter
0.13 ± 3% +0.2 0.29 ± 4% -0.0 0.13 perf-profile.children.cycles-pp.perf_trace_sched_switch
0.21 ± 6% +0.6 0.78 ± 3% -0.0 0.20 perf-profile.children.cycles-pp.update_sg_lb_stats
0.22 ± 6% +0.6 0.81 ± 4% -0.0 0.21 perf-profile.children.cycles-pp.update_sd_lb_stats
0.22 ± 6% +0.6 0.81 ± 3% -0.0 0.21 perf-profile.children.cycles-pp.sched_balance_find_src_group
0.40 ± 3% +0.7 1.08 ± 4% -0.0 0.40 perf-profile.children.cycles-pp.sched_balance_newidle
0.27 ± 6% +0.7 0.97 ± 4% -0.0 0.26 perf-profile.children.cycles-pp.sched_balance_rq
5.90 +0.9 6.81 -0.0 5.88 perf-profile.children.cycles-pp.intel_idle
35.82 +3.5 39.30 +0.1 35.93 perf-profile.children.cycles-pp.__x64_sys_mq_timedreceive
35.70 +3.5 39.25 +0.1 35.82 perf-profile.children.cycles-pp.do_mq_timedreceive
0.00 +8.8 8.78 ± 3% +0.0 0.00 perf-profile.children.cycles-pp.drain_obj_stock
28.34 +13.4 41.76 +0.3 28.64 perf-profile.children.cycles-pp.do_mq_timedsend
28.52 +13.5 41.99 +0.3 28.82 perf-profile.children.cycles-pp.__x64_sys_mq_timedsend
70.76 +13.7 84.45 +0.3 71.10 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
70.56 +13.8 84.39 +0.3 70.90 perf-profile.children.cycles-pp.do_syscall_64
0.08 +16.2 16.31 ± 4% +0.2 0.27 ± 3% perf-profile.children.cycles-pp.__refill_obj_stock
3.22 +20.1 23.33 ± 2% +0.4 3.62 perf-profile.children.cycles-pp.kfree
3.01 +20.2 23.25 ± 2% +0.4 3.38 perf-profile.children.cycles-pp.free_msg
2.46 +20.5 23.00 ± 2% +0.4 2.83 perf-profile.children.cycles-pp.__memcg_slab_free_hook
2.31 +25.9 28.25 ± 2% +0.6 2.94 perf-profile.children.cycles-pp.load_msg
1.00 +26.8 27.81 ± 2% +0.6 1.62 perf-profile.children.cycles-pp.__kmalloc_node_noprof
0.68 +27.0 27.65 ± 2% +0.6 1.29 perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
4.41 -4.4 0.02 ±141% -0.1 4.28 perf-profile.self.cycles-pp.poll_idle
6.79 -3.2 3.63 ± 5% -0.1 6.70 perf-profile.self.cycles-pp.select_idle_core
3.07 -1.7 1.33 ± 4% -0.0 3.06 perf-profile.self.cycles-pp.do_mq_timedreceive
2.90 -1.6 1.31 ± 3% -0.0 2.88 perf-profile.self.cycles-pp.__schedule
2.37 -1.5 0.84 ± 3% -0.0 2.34 perf-profile.self.cycles-pp._copy_to_user
2.73 -1.5 1.22 ± 3% -0.0 2.70 perf-profile.self.cycles-pp.do_mq_timedsend
2.63 -1.4 1.25 ± 3% -0.0 2.59 perf-profile.self.cycles-pp._raw_spin_lock
1.64 -1.0 0.64 ± 3% -0.1 1.52 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.46 -0.9 0.55 ± 2% -0.0 1.45 perf-profile.self.cycles-pp.switch_mm_irqs_off
1.54 -0.9 0.67 ± 6% +0.0 1.54 perf-profile.self.cycles-pp.update_rq_clock_task
1.48 -0.9 0.62 ± 3% +0.0 1.49 perf-profile.self.cycles-pp.msg_get
1.36 -0.8 0.58 ± 3% -0.1 1.30 perf-profile.self.cycles-pp.exit_to_user_mode_loop
1.11 -0.7 0.44 ± 4% -0.0 1.08 perf-profile.self.cycles-pp.msg_insert
1.56 -0.7 0.90 ± 6% -0.0 1.56 perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.95 -0.7 0.29 ± 5% -0.0 0.94 perf-profile.self.cycles-pp.llist_reverse_order
1.00 -0.7 0.34 ± 7% -0.0 0.99 perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
1.19 -0.6 0.59 ± 3% -0.0 1.18 perf-profile.self.cycles-pp.wq_sleep
0.92 -0.6 0.36 ± 6% -0.0 0.92 perf-profile.self.cycles-pp.do_perf_trace_sched_wakeup_template
0.90 -0.6 0.34 ± 2% +0.0 0.92 perf-profile.self.cycles-pp.__update_idle_core
1.15 -0.5 0.60 ± 7% -0.0 1.14 perf-profile.self.cycles-pp.task_h_load
0.83 -0.5 0.30 ± 5% -0.0 0.83 perf-profile.self.cycles-pp.call_function_single_prep_ipi
0.97 -0.5 0.44 ± 2% -0.0 0.94 perf-profile.self.cycles-pp.dequeue_entities
0.84 -0.5 0.34 ± 3% +0.0 0.86 perf-profile.self.cycles-pp.cpuacct_charge
0.96 -0.5 0.47 ± 4% +0.0 0.96 perf-profile.self.cycles-pp.do_syscall_64
1.23 -0.5 0.74 -0.0 1.22 perf-profile.self.cycles-pp.__switch_to
0.73 ± 4% -0.5 0.24 ± 3% -0.0 0.70 perf-profile.self.cycles-pp.__bitmap_andnot
0.69 -0.5 0.20 ± 4% +0.0 0.69 perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.94 -0.5 0.47 +0.0 0.94 ± 2% perf-profile.self.cycles-pp._find_next_bit
0.77 -0.4 0.36 ± 6% +0.0 0.77 perf-profile.self.cycles-pp.update_entity_lag
0.76 -0.4 0.36 ± 3% -0.0 0.74 perf-profile.self.cycles-pp.update_load_avg
0.67 -0.4 0.27 ± 5% -0.0 0.66 perf-profile.self.cycles-pp.__smp_call_single_queue
0.70 -0.4 0.31 ± 2% +0.0 0.70 perf-profile.self.cycles-pp.pick_next_task_fair
0.55 -0.4 0.17 ± 2% -0.0 0.54 perf-profile.self.cycles-pp.os_xsave
0.56 -0.4 0.18 ± 4% -0.0 0.54 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.63 -0.4 0.25 -0.0 0.62 perf-profile.self.cycles-pp.switch_fpu_return
1.38 -0.4 1.01 -0.0 1.36 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.69 -0.4 0.33 ± 5% -0.0 0.68 perf-profile.self.cycles-pp.wake_affine
0.53 -0.4 0.16 -0.0 0.51 perf-profile.self.cycles-pp.native_sched_clock
0.67 -0.4 0.30 ± 2% +0.0 0.70 perf-profile.self.cycles-pp.kfree
0.75 -0.4 0.39 ± 3% +0.0 0.76 perf-profile.self.cycles-pp.update_curr
0.67 -0.4 0.31 ± 5% -0.0 0.66 perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.53 -0.4 0.18 ± 2% -0.0 0.52 perf-profile.self.cycles-pp.arch_exit_to_user_mode_prepare
0.49 ± 2% -0.4 0.14 ± 3% -0.0 0.48 perf-profile.self.cycles-pp._copy_from_user
0.63 -0.3 0.28 ± 3% -0.0 0.62 perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
0.58 -0.3 0.23 ± 4% -0.0 0.57 perf-profile.self.cycles-pp.select_idle_sibling
0.58 -0.3 0.24 ± 7% -0.0 0.56 perf-profile.self.cycles-pp.migrate_task_rq_fair
0.64 -0.3 0.31 -0.0 0.63 perf-profile.self.cycles-pp.prepare_task_switch
0.82 -0.3 0.49 ± 3% -0.0 0.82 perf-profile.self.cycles-pp.select_idle_cpu
0.52 ± 3% -0.3 0.21 ± 2% +0.0 0.52 perf-profile.self.cycles-pp.__put_user_4
0.63 ± 11% -0.3 0.32 ± 5% +0.0 0.64 perf-profile.self.cycles-pp.stress_switch_mq
0.54 -0.3 0.24 ± 5% -0.0 0.53 perf-profile.self.cycles-pp.__switch_to_asm
0.54 -0.3 0.25 ± 3% +0.0 0.55 perf-profile.self.cycles-pp.fdget
0.51 -0.3 0.22 ± 4% -0.0 0.50 perf-profile.self.cycles-pp.mm_cid_switch_to
0.44 -0.3 0.15 ± 6% -0.0 0.44 perf-profile.self.cycles-pp.select_task_rq_fair
0.43 -0.3 0.15 ± 5% -0.0 0.42 perf-profile.self.cycles-pp.sched_ttwu_pending
0.49 -0.3 0.23 ± 6% +0.0 0.50 perf-profile.self.cycles-pp.enqueue_task
0.59 -0.2 0.34 ± 2% -0.0 0.58 perf-profile.self.cycles-pp.try_to_wake_up
0.73 -0.2 0.48 -0.0 0.72 perf-profile.self.cycles-pp.update_cfs_rq_load_avg
0.35 -0.2 0.11 ± 4% +0.0 0.35 perf-profile.self.cycles-pp.__resched_curr
0.40 -0.2 0.16 ± 2% -0.0 0.40 perf-profile.self.cycles-pp.__pick_next_task
0.36 -0.2 0.12 ± 3% -0.0 0.35 perf-profile.self.cycles-pp.cpuidle_idle_call
0.34 -0.2 0.11 -0.0 0.33 perf-profile.self.cycles-pp.entry_SYSCALL_64
0.57 -0.2 0.36 +0.0 0.58 perf-profile.self.cycles-pp.__enqueue_entity
0.51 -0.2 0.31 -0.0 0.50 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.34 -0.2 0.14 ± 5% +0.0 0.36 perf-profile.self.cycles-pp.wakeup_preempt
0.34 -0.2 0.14 ± 3% -0.0 0.34 perf-profile.self.cycles-pp.__wrgsbase_inactive
0.39 -0.2 0.20 ± 2% -0.0 0.39 perf-profile.self.cycles-pp.do_idle
0.31 ± 5% -0.2 0.11 -0.0 0.28 ± 5% perf-profile.self.cycles-pp.__check_heap_object
0.37 -0.2 0.17 ± 2% +0.0 0.37 perf-profile.self.cycles-pp.check_heap_object
0.51 -0.2 0.32 +0.0 0.51 perf-profile.self.cycles-pp.schedule
0.36 -0.2 0.18 ± 2% -0.0 0.36 perf-profile.self.cycles-pp.__pick_eevdf
0.61 -0.2 0.43 +0.0 0.63 perf-profile.self.cycles-pp.dequeue_entity
0.29 ± 2% -0.2 0.11 ± 4% +0.0 0.31 perf-profile.self.cycles-pp.__virt_addr_valid
0.30 -0.2 0.12 -0.0 0.29 perf-profile.self.cycles-pp.__update_load_avg_se
0.27 -0.2 0.10 ± 4% -0.0 0.27 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.25 -0.2 0.08 ± 5% -0.0 0.24 ± 2% perf-profile.self.cycles-pp.__check_object_size
0.26 -0.2 0.11 ± 4% -0.0 0.25 perf-profile.self.cycles-pp.place_entity
0.45 -0.2 0.30 ± 3% -0.0 0.44 perf-profile.self.cycles-pp.enqueue_entity
0.44 -0.2 0.29 ± 5% -0.0 0.44 perf-profile.self.cycles-pp.enqueue_task_fair
0.24 -0.2 0.09 +0.0 0.24 perf-profile.self.cycles-pp.wake_up_q
0.22 ± 2% -0.1 0.07 -0.0 0.21 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.24 -0.1 0.09 ± 5% +0.0 0.24 perf-profile.self.cycles-pp.___perf_sw_event
0.32 -0.1 0.18 ± 2% +0.0 0.32 perf-profile.self.cycles-pp.dequeue_task_fair
0.21 ± 2% -0.1 0.06 ± 7% -0.0 0.20 perf-profile.self.cycles-pp.read_tsc
0.20 -0.1 0.06 -0.0 0.20 ± 2% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.36 -0.1 0.22 ± 4% -0.0 0.35 perf-profile.self.cycles-pp.perf_tp_event
0.24 -0.1 0.11 ± 4% -0.0 0.24 perf-profile.self.cycles-pp.strnlen
0.28 -0.1 0.14 +0.0 0.28 perf-profile.self.cycles-pp.__kmalloc_node_noprof
0.21 -0.1 0.08 ± 6% +0.0 0.23 perf-profile.self.cycles-pp.load_msg
0.33 -0.1 0.20 ± 4% -0.0 0.32 perf-profile.self.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.0 0.08 ± 6% perf-profile.self.cycles-pp.__do_notify
0.41 -0.1 0.29 -0.0 0.40 perf-profile.self.cycles-pp.update_se
0.44 -0.1 0.33 -0.0 0.44 perf-profile.self.cycles-pp.menu_select
0.27 -0.1 0.15 ± 6% -0.0 0.26 perf-profile.self.cycles-pp.select_task_rq
0.17 ± 2% -0.1 0.06 ± 8% -0.0 0.17 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.23 ± 2% -0.1 0.11 ± 4% -0.0 0.22 perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.19 -0.1 0.08 ± 6% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.update_rq_clock
0.18 ± 2% -0.1 0.06 ± 7% -0.0 0.17 perf-profile.self.cycles-pp.wake_q_add_safe
0.20 ± 2% -0.1 0.09 +0.0 0.21 perf-profile.self.cycles-pp.check_stack_object
0.18 -0.1 0.07 +0.1 0.26 perf-profile.self.cycles-pp.__account_obj_stock
0.21 ± 2% -0.1 0.10 ± 4% -0.0 0.20 perf-profile.self.cycles-pp.__kmalloc_cache_noprof
0.24 -0.1 0.14 -0.0 0.24 ± 2% perf-profile.self.cycles-pp.__dequeue_entity
0.19 -0.1 0.10 ± 4% -0.0 0.19 perf-profile.self.cycles-pp.pick_task_fair
0.14 ± 3% -0.1 0.05 +0.0 0.14 ± 3% perf-profile.self.cycles-pp.inode_set_ctime_current
0.16 ± 3% -0.1 0.06 ± 7% +0.0 0.16 perf-profile.self.cycles-pp.nohz_run_idle_balance
0.15 -0.1 0.06 +0.0 0.15 perf-profile.self.cycles-pp.avg_vruntime
0.16 -0.1 0.07 +0.0 0.16 perf-profile.self.cycles-pp.schedule_idle
0.08 -0.1 0.00 -0.0 0.07 perf-profile.self.cycles-pp.security_msg_msg_free
0.07 -0.1 0.00 -0.0 0.06 perf-profile.self.cycles-pp.ct_kernel_enter
0.07 -0.1 0.00 +0.0 0.08 ± 5% perf-profile.self.cycles-pp.trylock_stock
0.15 -0.1 0.08 ± 5% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.dl_server_update
0.15 -0.1 0.08 ± 5% +0.0 0.15 perf-profile.self.cycles-pp.raw_spin_rq_lock_nested
0.12 -0.1 0.05 ± 8% +0.0 0.12 perf-profile.self.cycles-pp.migrate_disable_switch
0.12 ± 4% -0.1 0.05 +0.0 0.12 perf-profile.self.cycles-pp.__x64_sys_mq_timedreceive
0.13 -0.1 0.07 ± 7% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.___task_rq_lock
0.09 ± 5% -0.1 0.03 ± 70% +0.0 0.11 perf-profile.self.cycles-pp.inode_set_ctime_to_ts
0.22 -0.1 0.16 -0.0 0.21 perf-profile.self.cycles-pp.cpuidle_enter_state
0.12 -0.1 0.06 -0.0 0.12 ± 4% perf-profile.self.cycles-pp.store_msg
0.12 -0.1 0.06 -0.0 0.12 ± 4% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.11 -0.1 0.05 +0.0 0.11 perf-profile.self.cycles-pp.timestamp_truncate
0.11 ± 4% -0.1 0.05 ± 8% +0.0 0.12 perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
0.12 ± 4% -0.1 0.06 ± 7% -0.0 0.11 perf-profile.self.cycles-pp.do_perf_trace_sched_stat_runtime
0.11 ± 4% -0.1 0.06 ± 8% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.07 ± 11% -0.1 0.02 ±141% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.mq_timedreceive
0.13 ± 3% -0.0 0.08 +0.0 0.13 perf-profile.self.cycles-pp.update_curr_dl_se
0.15 -0.0 0.11 ± 4% +0.0 0.16 ± 3% perf-profile.self.cycles-pp.sched_balance_newidle
0.09 -0.0 0.05 ± 8% -0.0 0.09 perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 3% -0.0 0.10 ± 4% -0.0 0.13 perf-profile.self.cycles-pp.vruntime_eligible
0.14 ± 3% -0.0 0.11 +0.0 0.14 ± 3% perf-profile.self.cycles-pp.ktime_get
0.07 ± 7% -0.0 0.03 ± 70% -0.0 0.06 perf-profile.self.cycles-pp.__set_next_task_fair
0.15 ± 3% -0.0 0.13 +0.0 0.16 perf-profile.self.cycles-pp.put_prev_entity
0.02 ±141% -0.0 0.00 +0.1 0.09 perf-profile.self.cycles-pp.css_rstat_updated
0.02 ±141% -0.0 0.00 +0.0 0.06 ± 9% perf-profile.self.cycles-pp.perf_trace_buf_update
0.19 -0.0 0.18 ± 2% -0.0 0.18 perf-profile.self.cycles-pp.__x64_sys_mq_timedsend
0.00 +0.0 0.00 +0.1 0.11 perf-profile.self.cycles-pp.__mod_memcg_state
0.00 +0.0 0.00 +0.1 0.12 ± 4% perf-profile.self.cycles-pp.try_charge_memcg
0.00 +0.0 0.00 +0.3 0.26 ± 3% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.11 ± 4% +0.0 0.12 ± 3% +0.0 0.11 perf-profile.self.cycles-pp.set_next_task_idle
0.06 +0.0 0.08 ± 6% +0.0 0.06 perf-profile.self.cycles-pp.perf_swevent_event
0.07 ± 7% +0.0 0.09 ± 10% +0.0 0.11 perf-profile.self.cycles-pp.mq_timedsend
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.ct_idle_enter
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 +0.0 0.00 perf-profile.self.cycles-pp.sched_update_worker
0.12 +0.1 0.18 ± 6% +0.0 0.12 perf-profile.self.cycles-pp.x64_sys_call
0.38 ± 2% +0.1 0.46 -0.0 0.37 perf-profile.self.cycles-pp.finish_task_switch
0.08 ± 5% +0.1 0.20 ± 6% -0.0 0.08 perf-profile.self.cycles-pp.do_perf_trace_sched_switch
0.18 ± 5% +0.5 0.71 ± 3% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.update_sg_lb_stats
5.89 +0.9 6.81 -0.0 5.88 perf-profile.self.cycles-pp.intel_idle
0.07 +7.4 7.48 ± 4% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.__refill_obj_stock
0.00 +8.7 8.70 ± 3% +0.0 0.00 perf-profile.self.cycles-pp.drain_obj_stock
2.25 +12.4 14.61 +0.0 2.26 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.53 +19.0 19.48 +0.1 0.67 perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook