[SCHEDULER] Performance drop in 4.19 compared to 4.18 kernel

[Jirka Hladky]

Hi Srikar,

I work at Red Hat in the Kernel Performance Team. I would like to ask
you for help.

We have detected a significant performance drop (20% and more) with
4.19rc1 relatively to 4.18 vanilla. We see the regression on different
2 NUMA and 4 NUMA boxes with pretty much all the benchmarks we use -
NAS, Stream, SPECjbb2005, SPECjvm2008.

Mel Gorman has suggested checking
2d4056fafa196e1ab4e7161bae4df76f9602d56d commit - with it reverted we
got some performance back but not entirely:

 * Compared to 4.18, there is still performance regression -
especially with NAS (sp_C_x subtest) and SPECjvm2008. On 4 NUMA
systems, regression is around 10-15%
  * Compared to 4.19rc1 there is a clear gain across all benchmarks, up to 20%.

We are investigating the issue further, Mel has suggested to check
305c1fac3225dfa7eeb89bfe91b7335a6edd5172 as next.

Do you have any further recommendations, which commits have possibly
caused the performance degradation?

I want to discuss with you how can we collaborate on performance
testing for the upstream kernel. Does your testing show as well
performance drop in 4.19? If so, do you have any plans for the fix? If
no, can we send you some more information about our tests so that you
can try to reproduce it?

We would also be more than happy to test the new patches for the
performance - please let us know if you are interested.  We have a
pool of 1 NUMA up to 8 NUMA boxes for that, both AMD and Intel,
covering different CPU generations from Sandy Bridge till Skylake.

I'm looking forward to hearing from you.
Jirka