RE: Enquiry on unbalanced memory throughput for dual-Cortex A9 core.

From: Ooi, Tzy Way
Date: Sun Jul 22 2018 - 23:39:35 EST

Hi Russell,

Thanks for the details explanation.

I am running the bw_mem test provided in LM Benchmark to measures the throughput to write data to memory as it seems to be a general benchmark program.

Initially, I tested to run my own's created memory test program and encountered the same unbalanced memory throughput whenever two threads are running on different cores. I tested to run the memory test program on either one core or two cores. The unbalanced memory throughput is seen when running on two cores. Hence, I tried out the bw_mem test and it appears to be alike to my own test case.

Attached is the memory test program memtest.c file and the Linux executable file. The memtest -a1 will forces the two threads running on different cores while memtest -a2 will forces the two threads running on one core.

May I know if is it possible if you could try out my test program on your iMX6D SoC (also dual core Cortex A9) board?

Comparison on the test result one core vs two cores on my board with Linux 4.9 and 4.18-rc2
One core:

========= Multi Thread =========

Thread 3067511920 - data size 1 MB, runs = 1000
Thread 3059123312 - data size 1 MB, runs = 1000
Thread :3059123312: Datarate: 974.887201 MB/s
Thread :3067511920: Datarate: 960.289834 MB/s
Thread :3067511920: Datarate: 1083.249741 MB/s
Thread :3059123312: Datarate: 1055.545769 MB/s
Thread :3067511920: Datarate: 1085.555446 MB/s
Thread :3059123312: Datarate: 1084.503430 MB/s
Thread :3067511920: Datarate: 1063.379303 MB/s
Thread :3059123312: Datarate: 1070.705338 MB/s
Thread :3067511920: Datarate: 1050.933243 MB/s
Thread :3059123312: Datarate: 1050.153330 MB/s
Thread :3067511920: Datarate: 1085.489144 MB/s
Thread :3059123312: Datarate: 1071.774560 MB/s
Thread :3067511920: Datarate: 1084.506795 MB/s
Thread :3059123312: Datarate: 1060.260066 MB/s
Thread :3067511920: Datarate: 1074.058027 MB/s
Thread :3059123312: Datarate: 1069.279388 MB/s
Thread :3067511920: Datarate: 1073.924924 MB/s
Thread :3059123312: Datarate: 1080.818992 MB/s
Thread :3067511920: Datarate: 1081.871683 MB/s
Thread :3067511920: Average Datarate: 1064.325814 MB/s
Thread :3059123312: Datarate: 1097.549768 MB/s
Thread :3059123312: Average Datarate: 1061.547784 MB/s
Finished!

Two Cores:
========= Multi Thread =========

Thread 3067954288 - data size 1 MB, runs = 1000
Thread 3059565680 - data size 1 MB, runs = 1000
Thread :3067954288: Datarate: 741.930805 MB/s
Thread :3059565680: Datarate: 377.979641 MB/s
Thread :3067954288: Datarate: 741.976479 MB/s
Thread :3067954288: Datarate: 740.548015 MB/s
Thread :3059565680: Datarate: 376.706463 MB/s
Thread :3067954288: Datarate: 740.313260 MB/s
Thread :3067954288: Datarate: 740.363440 MB/s
Thread :3059565680: Datarate: 376.129877 MB/s
Thread :3067954288: Datarate: 740.056194 MB/s
Thread :3067954288: Datarate: 740.219191 MB/s
Thread :3059565680: Datarate: 376.114092 MB/s
Thread :3067954288: Datarate: 740.152311 MB/s
Thread :3067954288: Datarate: 724.094688 MB/s
Thread :3059565680: Datarate: 388.118117 MB/s
Thread :3067954288: Datarate: 740.556383 MB/s
Thread :3067954288: Average Datarate: 739.021077 MB/s
Thread :3059565680: Datarate: 1323.735631 MB/s
Thread :3059565680: Datarate: 2072.948256 MB/s
Thread :3059565680: Datarate: 2069.817984 MB/s
Thread :3059565680: Datarate: 2069.295149 MB/s
Thread :3059565680: Datarate: 2040.932474 MB/s
Thread :3059565680: Average Datarate: 1147.177768 MB/s
Finished!

Thank you for your comment and help

Best regards,
Tzy Way

-----Original Message-----
From: Russell King - ARM Linux <linux@xxxxxxxxxxxxxxx>
Sent: Friday, July 20, 2018 6:36 PM
To: Ooi, Tzy Way <tzy.way.ooi@xxxxxxxxx>
Cc: linux-kernel@xxxxxxxxxxxxxxx; See, Chin Liang <chin.liang.see@xxxxxxxxx>; Tan, Ley Foon <ley.foon.tan@xxxxxxxxx>; Nguyen, Dinh <dinh.nguyen@xxxxxxxxx>; Aw, Khai Liang <khai.liang.aw@xxxxxxxxx>
Subject: Re: Enquiry on unbalanced memory throughput for dual-Cortex A9 core.

On Fri, Jul 20, 2018 at 08:49:47AM +0000, Ooi, Tzy Way wrote:
> Hi Russell,
>
> I am trying the memory write operation with the LM benchmark test. I
> tried to execute the memory write operation here
> <http://lmbench.sourceforge.net/cgi-bin/man?section=8&keyword=bw_mem>
> twice to get both Cortex A9 core processor to work on each processes.
> Both processors is going to perform write operation at almost the same
> time to the memory.
>
> As shown in the pictures below, the memory throughput from one of the
> cores is about double the throughput of another core. i.e. 377MB/s VS
> 728MB/s
>
> [cid:image001.png@01D42049.5A7D0070]
>
> I have tested this operation across few dual cores Cortex A9 boards
> and all the board is having the same result. The test is tested on
> kernel version 4.9 and newest Linux kernel version 4.18.0-rc2

Here's how 4.14 behaves on an iMX6D SoC (also dual core Cortex A9):

$ taskset -c 0 ./bw_mem -N 1000 1M fwr & taskset -c 1 ./bw_mem -N 1000 1M fwr [1] 21799
1.00 521.10
1.00 497.27
[1]+ Done taskset -c 0 ./bw_mem -N 1000 1M fwr
$ taskset -c 0 ./bw_mem -N 1000 1M fwr & taskset -c 1 ./bw_mem -N 1000 1M fwr [1] 21803
1.00 520.83
1.00 496.44

which shows some asymmetry but nowhere near yours.

I'm using taskset to force each to be locked to a particular CPU - you'll see why further down. Even without it, I get similar results to those I mention above.

Now, playing around with this, so we can identify which bw_mem output is
which:

$ taskset -c 0 ./bw_mem -N 1000 1M fwr & c1=$(taskset -c 1 ./bw_mem -N 1000 1M fwr 2>&1); echo "c1: $c1"
[1] 21876
1.00 521.92
c1: 1.00 496.69
$ taskset -c 1 ./bw_mem -N 1000 1M fwr & c1=$(taskset -c 0 ./bw_mem -N 1000 1M fwr 2>&1); echo "c0: $c1"
[1] 21881
c0: 1.00 521.83
1.00 496.20

CPU0 is always the slightly faster of the two. If we use /usr/bin/time to time these:

CPU0:
6.10user 0.25system 0:06.56elapsed 96%CPU (0avgtext+0avgdata 1664maxresident)k
0inputs+0outputs (0major+407minor)pagefaults 0swaps

CPU1:
6.36user 0.24system 0:06.77elapsed 97%CPU (0avgtext+0avgdata 1600maxresident)k
0inputs+0outputs (0major+399minor)pagefaults 0swaps

So, CPU1 takes slightly longer in userspace, has less resident pages and less minor faults which is rather odd. Repeatedly running just one instance gives different results each time... disabling virtual address space randomisation solves that:

echo 0 >/proc/sys/kernel/randomize_va_space

which then gives me:

CPU0: 1.00 520.20
6.18user 0.20system 0:06.59elapsed 96%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps
CPU1: 1.00 496.61
6.46user 0.14system 0:06.77elapsed 97%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps

CPU0: 1.00 521.10
6.13user 0.21system 0:06.57elapsed 96%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps
CPU1: 1.00 498.01
6.40user 0.18system 0:06.75elapsed 97%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps

which is rather more stable as far as resource usage goes between the two CPUs, but still an asymmetry in the reported bandwidths and times.
So, this has ruled out differences in VA layout.

Now for the interesting bit... it's important to understand what and how stuff is being measured. Looking at the bw_mem.c and associated source code, it measures the performance against the wall clock, which includes everything that the system is doing on each particular CPU.
So, if a CPU is interrupted by another thread wanting to run, it'll affect the results. Hence, it's best to run on an otherwise quiet system, eg, without an init daemon (eg, booted with init=/bin/sh on the kernel command line - but note there won't be any job control, so ^C won't work!)

However, continuing on...

If I run bw_mem on just one CPU:

CPU1: 1.00 2617.31
5.74user 0.18system 0:06.03elapsed 98%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps

Same number of iterations, same memory size, but notice that it appears to be a lot faster reported by bw_mem, but the time taken is about the same. cpufreq comes to mind, but that's disabled on this system.

So, it brings up a rather obvious question: what exactly is bw_mem measuring, and is it measuring it correctly?

$ /usr/bin/time taskset -c 1 ./bw_mem -P 1 -N 1000 1M fwr
1.00 2601.26
5.80user 0.16system 0:06.06elapsed 98%CPU (0avgtext+0avgdata 1700maxresident)k
0inputs+0outputs (0major+403minor)pagefaults 0swaps
$ /usr/bin/time ./bw_mem -P 2 -N 1000 1M fwr ^CCommand terminated by signal 2 5.54user 0.13system 1:12.20elapsed 7%CPU (0avgtext+0avgdata 1696maxresident)k
0inputs+0outputs (0major+365minor)pagefaults 0swaps

so requesting a parallelism of 2 results in the program never seemingly ending in a reasonable period of time, which suggests a bug somewhere.
Are we sure that bw_mem is actually working as intended?

Maybe if Larry is reading this, he could share some thoughts.

--
RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps up According to speedtest.net: 13Mbps down 490kbps up
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <malloc.h>
#include <pthread.h>
#include <semaphore.h>
#include <string.h>
#include <sys/time.h>
#include <sched.h>
#include <sys/resource.h>
#include <sys/mman.h>

#define MB (1024*1024)
#define QUAD_CORE 0

# define handle_error_en(en, msg) \
do { errno = en; perror(msg); exit(EXIT_FAILURE); } while (0);

unsigned long l_megabytes = 1;
unsigned long l_repeat = 1000;

void SingleFct(void)
{
unsigned long l_size = l_megabytes*MB;
unsigned char * l_memBuffer = memalign(32, l_size);
int i,j;
unsigned long l_time;
double l_datarate;
double t_datarate = 0.0;
struct timeval start, end;

printf("Single Thread - data size %ld MB, runs = %ld\n", l_megabytes, l_repeat);
for(j=0; j<10; j++)
{
gettimeofday(&start, NULL);
for(i=0; i<l_repeat; i++)
{
memset(l_memBuffer, 0x5a, l_size);
}
gettimeofday(&end, NULL);

l_time = (end.tv_sec*1000000 + end.tv_usec) - (start.tv_sec*1000000 + start.tv_usec);
// printf("end time = %u.%06u, start time = %u.%06u\n", end.tv_sec, end.tv_usec, start.tv_sec, start.tv_usec);
// printf("l_time = %u\n", l_time);
l_datarate = (double)l_size*(double)l_repeat/(double)l_time;
printf("Single Thread Datarate: %f MB/s\n", l_datarate);
t_datarate += l_datarate;
}
printf("Single Thread Average Datarate: %f MB/s\n", t_datarate/10.0f);
free(l_memBuffer);
}

void Fct(void)
{
unsigned long l_size = l_megabytes*MB;
unsigned char * l_memBuffer = memalign(32, l_size);
unsigned long l_time;
double l_datarate;
double t_datarate = 0.0;
int i,j;
struct timeval start, end;

printf("Thread %u - data size %ld MB, runs = %ld\n", (unsigned int)pthread_self(), l_megabytes, l_repeat);
for(j=0; j<10; j++)
{
gettimeofday(&start, NULL);
for(i=0; i<l_repeat; i++)
{
memset(l_memBuffer, 0x5a, l_size);
}
gettimeofday(&end, NULL);
l_time = (end.tv_sec*1000000 + end.tv_usec) - (start.tv_sec*1000000 + start.tv_usec);
l_datarate = (double)l_size*(double)l_repeat/(double)l_time;
printf("Thread :%u: Datarate: %lf MB/s\n", (unsigned int)pthread_self(), l_datarate);
t_datarate += l_datarate;

}

printf("Thread :%u: Average Datarate: %lf MB/s\n", (unsigned int)pthread_self(), t_datarate/10.0f);
free(l_memBuffer);

pthread_exit((void *) 0);
}

int main(int argc, char** argv) {

pthread_t t1, t2, t3, t4;

pthread_attr_t t1_attr, t2_attr, t3_attr, t4_attr;
cpu_set_t t1_cpuset, t2_cpuset, t3_cpuset, t4_cpuset;

int rv;
int affinity = 0;
int iarg = 1;

while (iarg < argc) {
if (!strcmp(argv[iarg],"-a1")) {
affinity = 1;
}
else if (!strcmp(argv[iarg],"-a2")) {
affinity = 2;
}
else if (!strcmp(argv[iarg],"-s1")) {
l_megabytes = 1;
}
else if (!strcmp(argv[iarg],"-s2")) {
l_megabytes = 10;
}
else if (!strcmp(argv[iarg],"-s3")) {
l_megabytes = 100;
}
else {
printf("Usage: %s [-a1|-a2 ] [-s1|-s2|-s3]\n",argv[0]);
exit(0);
}
iarg++;
}
printf("========= Single Thread =========\n\n");

SingleFct();

printf("\n\n========= Multi Thread =========\n\n");

rv = pthread_attr_init(&t1_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #1");

rv = pthread_attr_init(&t2_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #2");

if (affinity == 1)
{
CPU_ZERO(&t1_cpuset);
CPU_SET(0,&t1_cpuset);
rv = pthread_attr_setaffinity_np(&t1_attr,sizeof(t1_cpuset),&t1_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #1");

CPU_ZERO(&t2_cpuset);
CPU_SET(1,&t2_cpuset);
rv = pthread_attr_setaffinity_np(&t2_attr,sizeof(t2_cpuset),&t2_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #2");
}
else if (affinity == 2)
{
CPU_ZERO(&t1_cpuset);
CPU_SET(0,&t1_cpuset);
rv = pthread_attr_setaffinity_np(&t1_attr,sizeof(t1_cpuset),&t1_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #1");

CPU_ZERO(&t2_cpuset);
CPU_SET(0,&t2_cpuset);
rv = pthread_attr_setaffinity_np(&t2_attr,sizeof(t2_cpuset),&t2_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #2");
}

rv = pthread_create(&t1, &t1_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #1");

rv = pthread_create(&t2, &t2_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #2");

#if QUAD_CORE
rv = pthread_attr_init(&t3_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #3");

rv = pthread_attr_init(&t4_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #4");

if (affinity == 1)
{
CPU_ZERO(&t3_cpuset);
CPU_SET(2,&t3_cpuset);
rv = pthread_attr_setaffinity_np(&t3_attr,sizeof(t3_cpuset),&t3_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #3");

CPU_ZERO(&t4_cpuset);
CPU_SET(3,&t4_cpuset);
rv = pthread_attr_setaffinity_np(&t4_attr,sizeof(t4_cpuset),&t4_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #4");
}
else if (affinity == 2)
{
CPU_ZERO(&t3_cpuset);
CPU_SET(0,&t3_cpuset);
rv = pthread_attr_setaffinity_np(&t3_attr,sizeof(t3_cpuset),&t3_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #3");

CPU_ZERO(&t4_cpuset);
CPU_SET(0,&t4_cpuset);
rv = pthread_attr_setaffinity_np(&t4_attr,sizeof(t4_cpuset),&t4_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #4");
}

rv = pthread_create(&t3, &t3_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #3");

rv = pthread_create(&t4, &t4_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #4");

#endif

pthread_join(t1, NULL);
pthread_join(t2, NULL);
#if QUAD_CORE
pthread_join(t3, NULL);
pthread_join(t4, NULL);
#endif

printf("Finished!\n");
return 0;
}

Attachment: memtest_2
Description: memtest_2