Re: [PATCH 1/2] perf bench futex: cache align the worer struct

[Davidlohr Bueso]

On Sun, 16 Oct 2016, Sebastian Andrzej Siewior wrote:

> It popped up in perf testing that the worker consumes some amount of
> CPU. It boils down to the increment of `ops` which causes cache line
> bouncing between the individual threads.

Are you referring to this?

      â??         for (i = 0; i < nfutexes; i++, w->ops++) {
      â?? be:   add    $0x1,%ebx
65.87 â??       addq   $0x1,0x18(%r12)

(which is like 65% of 13% on my box with a default futex-hash run).

Even better, could we get rid entirely of the ops increments and just
use a local variable, then update the worker at the end of the function.
The following makes 'perf' pretty much disappear in the profile.

Thanks,
Davidlohr

----8<--------------
From: Davidlohr Bueso <dave@xxxxxxxxxxxx>
Subject: [PATCH] perf/bench-futex: Avoid worker cacheline bouncing

Sebastian noted that overhead for worker thread ops (throughput)
accounting was producing 'perf' to appear in the profiles, consuming
a non-trivial (ie 13%) amount of CPU. This is due to cacheline
bouncing due to the increment of w->ops. We can easily fix this by
just working on a local copy and updating the actual worker once
done running, and ready to show the program summary. There is no
danger of the worker being concurrent, so we can trust that no stale
value is being seen by another thread.

Reported-by: Sebastian Andrzej Siewior <bigeasy@xxxxxxxxxxxxx>
Signed-off-by: Davidlohr Bueso <dbueso@xxxxxxx>
---
tools/perf/bench/futex-hash.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/perf/bench/futex-hash.c b/tools/perf/bench/futex-hash.c
index 8024cd5febd2..0b9583164da0 100644
--- a/tools/perf/bench/futex-hash.c
+++ b/tools/perf/bench/futex-hash.c
@@ -63,8 +63,10 @@ static const char * const bench_futex_hash_usage[] = {
static void *workerfn(void *arg)
{
	int ret;
-	unsigned int i;
	struct worker *w = (struct worker *) arg;
+	unsigned int i;
+	unsigned long ops = w->ops; /* avoid cacheline-bouncing */
+

	pthread_mutex_lock(&thread_lock);
	threads_starting--;
@@ -74,7 +76,7 @@ static void *workerfn(void *arg)
	pthread_mutex_unlock(&thread_lock);

	do {
-		for (i = 0; i < nfutexes; i++, w->ops++) {
+		for (i = 0; i < nfutexes; i++, ops++) {
			/*
			 * We want the futex calls to fail in order to stress
			 * the hashing of uaddr and not measure other steps,
@@ -88,6 +90,8 @@ static void *workerfn(void *arg)
		}
	}  while (!done);

+	w->ops = ops;
+
	return NULL;
}

--
2.6.6