Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6% will-it-scale.per_process_ops

[Thomas Gleixner]

On Fri, 18 Mar 2016, Huang, Ying wrote:
> Usually we will put most important change we think in the subject of the
> mail, for this email, it is,
> 
> +25.6% will-it-scale.per_process_ops

That is confusing on it's own, because the reader does not know at all whether
this is an improvement or a regression.

So something like this might be useful:

Subject: subsystem: 12digitsha1: 25% performance improvement

or in some other case

Subject: subsystem: 12digitsha1: 25% performance regression

So in the latter case I will look into that mail immediately. The improvement
one can wait until I have cared about urgent stuff.

In the subject line it is pretty much irrelevant which foo-bla-ops test has
produced that result. It really does not matter. If it's a regression, it's
urgent. If it's an improvement it's informal and it can wait to be read.

So in that case it would be:

futex: 65d8fc777f6d: 25% performance improvement

You can grab the subsystem prefix from the commit.

> and, we try to put most important changes at the top of the comparison
> result below.  That is the will-it-scale.xxx below.
> 
> We are thinking about how to improve this.  You input is valuable for
> us.  We are thinking change the "below changes" line to something like
> below.
> 
> FYI, we noticed the +25.6% will-it-scale.per_process_ops improvement on
> ...
> 
> Does this looks better?

A bit, but it still does not tell me much. It's completely non obvious what
'will-it-scale.per_process_ops' means. Let me give you an example how a useful
and easy to understand summary of the change could look like:


 FYI, we noticed 25.6% performance improvement due to commit

   65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()"

 in the will-it-scale.per_process_ops test.

 will-it-scale.per_process_ops tests the futex operations for process shared
 futexes (Or whatever that test really does).

 The commit has no significant impact on any other test in the test suite.

So those few lines tell precisely what this is about. It's something I already
expected, so I really can skip the rest of the mail unless I'm interested in
reproducing the result.

Now lets look at a performance regression.

Subject: futex: 65d8fc777f6d: 25% performance regression

 FYI, we noticed a 25.2% performance regression due to commit

 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()"

 in the will-it-scale.per_process_ops test.

 will-it-scale.per_process_ops tests the futex operations for process shared
 futexes (Or whatever that test really does).

 The commit has no significant impact on any other test in the test suite.

In that case I will certainly be interested how to reproduce that test. So I
need the following information:

Machine description: Intel IvyBridge 2 sockets, 32 cores, 64G RAM
Config file: http://wherever.you.store/your/results/test-nr/config
Test: http://wherever.you.store/your/tests/will-it-scale.per_process_ops.tar.bz2

That tarball should contain:

     README
     test_script.sh
     test_binary

README should tell:

   will-it-scale.per_process_ops

   Short explanation of the test

   Preliminaries: 
   	- perf
	- whatever

So that allows me to reproduce that test more or less with no effort. And
that's the really important part.

You can provide nice charts and full comparison tables for all tests on a web
site for those who are interested in large stats and pretty charts.

Full results: http://wherever.you.store/your/results/test-nr/results

So now lets look at a scenario where that commit results in a performance
improvement on will-it-scale.per_process_ops and a regression on
will-it-scale.per_task.

Subject: futex: 65d8fc777f6d: 25% performance regression

 FYI, we noticed a 25.2% performance regression due to commit

 65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()"

 in the will-it-scale.per_task test.

 will-it-scale.per_tasks tests the futex operations for process private
 futexes (Or whatever that test really does).

 The commit has significant impact on the following tests in the test suite:
 
	will-it-scale.per_process_ops: 25% improvement

The 25% improvement is really not interesting. What's interesting is the
regression. That's what people need to look at. You still can provide the
information about all the tests including the 25% improvement data on 

Full results: http://wherever.you.store/your/results/test-nr/results

Now, if you have commits which have mixed impact on various tests, then it's
important to point out the most relevant issue clearly in the
summary. I.e. you need to filter the tests for the maximum
regression/improvement and use that as the main information. You still can
provide the information about the impact on other tests in a very condensed
form.

 The commit has significant impact on the following tests in the test suite:

     test1: 	0.5% Regression
     test2:	2.0% Improvement
     ....
     testN:	0.2% Regression

That's useful information, but it might be completely irrelevant if the
primary impact is, e.g. a 10% Regression. Then it's nice to know, that it gave
a 2% improvement on test2, but that's about it. The important information is
the 10% regression, which needs to be addressed urgently.

Hope that helps.

Thanks,

	tglx