Re: [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression

[Oliver Sang]

hi, Peter Zijlstra,

On Fri, Apr 24, 2026 at 11:55:23AM +0200, Peter Zijlstra wrote:
> On Fri, Apr 24, 2026 at 03:10:07PM +0800, kernel test robot wrote:
> > 
> > 
> > Hello,
> > 
> > kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
> > 
> > 
> > commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > 
> > [still regression on linus/master      1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
> > [still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
> > [still regression on fix commit        eef9f648fb0e92618041f019d4bdcf7ae17cb743]
> > 
> > testcase: will-it-scale
> > config: x86_64-rhel-9.4
> > compiler: gcc-14
> > test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
> > parameters:
> 
> So this commit has lived in private trees (tglx and mine), has been in
> tip, in next etc. *WHY* do we only get this report now?
> 
> This isn't the first time I see a regression report after things have
> hit Linus' tree. This is starting to get really annoying. The point of
> having this robot was to catch issues *before* they hit Linus, no?

there are some difference between kbuild/boot tests and functional/performance
tests by kernel test bot.
kbuild/boot tests could be triggered directly by repo/branch changes.
functional/performance depends on so-called hourly kernel merge to trigger tests
and bisection (due to resource constraints).

for this commit and related branches, we did generate kbuild/boot reports.
including e.g. kbuild report [1] and boot report [2].
(
for [2], the reported fbc is acefed487add3, the commit "sched: Optimize
hrtimer handling" is the tip commit of tglx-devel/sched/hrtick branch at that
time

* 1f963bc350a6f sched: Optimize hrtimer handling    <----
* 8bc0da68c41a3 sched: Default enable HRTICK
...
* acefed487add3 x86/apic: Enable TSC coupled programming mode
* c903d7114698b x86/apic: Avoid the PVOPS indirection for the TSC deadline timer
)

after reviewing the history, we discovered that both reports originated from
direct repository/branch testing, meaning these branches were not merged into
hourly kernels, causing functional/performance tests to be skipped.

most likely, merge conflicts prevented this change from going through the
functional/performance pipeline until it reached mainline.

since mainline serves as the base for the hourly kernels mentioned above,
the commit will have more opportunity to go through functional/performance
testing after merging into mainline.

Thank you for bringing this to our attention. We are now considering
improvements to address this gap in our testing coverage.


> 
> That said, looking back at will-it-scale reports, 2.8 seems to be pretty
> much in the noise range, no?

this test supplies stable results. for parent commit:

c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json:  "will-it-scale.per_process_ops": [
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    238895,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    237238,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    237558,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    238616,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    238299,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-    237793
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json-  ],

for this commit:

96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json:  "will-it-scale.per_process_ops": [
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    231051,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    232241,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    231346,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    231058,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    230174,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-    232143
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json-  ],

We will conduct additional verification.



[1] https://lore.kernel.org/oe-kbuild-all/202602171013.VyR8FIZ2-lkp@xxxxxxxxx/
[2] https://lore.kernel.org/all/202602181118.929e4694-lkp@xxxxxxxxx/