Re: Kernel 4.7rc3 - Performance drop 30-40% for SPECjbb2005 and SPECjvm2008 benchmarks against 4.6 kernel

[Peter Zijlstra]

On Thu, Jun 16, 2016 at 06:38:50PM +0200, Jirka Hladky wrote:
> Hello,
> 
> we see performance drop 30-40% for SPECjbb2005 and SPECjvm2008

Blergh, of course I don't have those.. :/

> benchmarks starting from 4.7.0-0.rc0 kernel compared to 4.6 kernel.
> 
> We have tested kernels 4.7.0-0.rc1 and 4.7.0-0.rc3 and these are as
> well affected.
> 
> We have observed the drop on variety of different x86_64 servers with
> different configuration (different CPU models, RAM sizes, both with
> Hyper Threading ON and OFF, different NUMA configurations (2 and 4
> NUMA nodes)

What kind of config and userspace setup? Do you run this cruft in a
cgroup of sorts?

If so, does it change anything if you run it in the root cgroup?

> Linpack and Stream benchmarks do not show any performance drop.
> 
> The performance drop increases with higher number of threads. The
> maximum number of threads in each benchmark is the same as number of
> CPUs.
> 
> We have opened a BZ to track the progress:
> https://bugzilla.kernel.org/show_bug.cgi?id=120481
> 
> You can find more details along with graphs and tables there.
> 
> Do you have any hints which commit should we try to reverse?

There were only 66 commits or so, and I think we can rule out the
hotplug changes, which should reduce it even further.

You could see what the parent of this one does:

  2159197d6677 sched/core: Enable increased load resolution on 64-bit kernels

If not that, maybe the parent of:

  c58d25f371f5 sched/fair: Move record_wakee()

After that I suppose you'll have to go bisect.