RE: Enquiry on unbalanced memory throughput for dual-Cortex A9 core.

From: Ooi, Tzy Way
Date: Mon Jul 23 2018 - 22:23:26 EST

> -----Original Message-----
> From: Russell King - ARM Linux <linux@xxxxxxxxxxxxxxx>
> Sent: Friday, July 20, 2018 6:36 PM
> To: Ooi, Tzy Way <tzy.way.ooi@xxxxxxxxx>
> Cc: linux-kernel@xxxxxxxxxxxxxxx; See, Chin Liang
> <chin.liang.see@xxxxxxxxx>; Tan, Ley Foon <ley.foon.tan@xxxxxxxxx>;
> Nguyen, Dinh <dinh.nguyen@xxxxxxxxx>; Aw, Khai Liang
> <khai.liang.aw@xxxxxxxxx>
> Subject: Re: Enquiry on unbalanced memory throughput for dual-Cortex A9
> core.
>
> On Fri, Jul 20, 2018 at 08:49:47AM +0000, Ooi, Tzy Way wrote:
> > Hi Russell,
> >
> > I am trying the memory write operation with the LM benchmark test. I
> > tried to execute the memory write operation here
> > <http://lmbench.sourceforge.net/cgi-
> bin/man?section=8&keyword=bw_mem>
> > twice to get both Cortex A9 core processor to work on each processes.
> > Both processors is going to perform write operation at almost the same
> > time to the memory.
> >
> > As shown in the pictures below, the memory throughput from one of the
> > cores is about double the throughput of another core. i.e. 377MB/s VS
> > 728MB/s
> >
> > [cid:image001.png@01D42049.5A7D0070]
> >
> > I have tested this operation across few dual cores Cortex A9 boards
> > and all the board is having the same result. The test is tested on
> > kernel version 4.9 and newest Linux kernel version 4.18.0-rc2
>
> Here's how 4.14 behaves on an iMX6D SoC (also dual core Cortex A9):
>
> $ taskset -c 0 ./bw_mem -N 1000 1M fwr & taskset -c 1 ./bw_mem -N 1000
> 1M fwr [1] 21799
> 1.00 521.10
> 1.00 497.27
> [1]+ Done taskset -c 0 ./bw_mem -N 1000 1M fwr
> $ taskset -c 0 ./bw_mem -N 1000 1M fwr & taskset -c 1 ./bw_mem -N 1000
> 1M fwr [1] 21803
> 1.00 520.83
> 1.00 496.44
>
> which shows some asymmetry but nowhere near yours.
>
> I'm using taskset to force each to be locked to a particular CPU - you'll see
> why further down. Even without it, I get similar results to those I mention
> above.
>
> Now, playing around with this, so we can identify which bw_mem output is
> which:
>
> $ taskset -c 0 ./bw_mem -N 1000 1M fwr & c1=$(taskset -c 1 ./bw_mem -N
> 1000 1M fwr 2>&1); echo "c1: $c1"
> [1] 21876
> 1.00 521.92
> c1: 1.00 496.69
> $ taskset -c 1 ./bw_mem -N 1000 1M fwr & c1=$(taskset -c 0 ./bw_mem -N
> 1000 1M fwr 2>&1); echo "c0: $c1"
> [1] 21881
> c0: 1.00 521.83
> 1.00 496.20
>
> CPU0 is always the slightly faster of the two. If we use /usr/bin/time to time
> these:
>
> CPU0:
> 6.10user 0.25system 0:06.56elapsed 96%CPU (0avgtext+0avgdata
> 1664maxresident)k
> 0inputs+0outputs (0major+407minor)pagefaults 0swaps
>
> CPU1:
> 6.36user 0.24system 0:06.77elapsed 97%CPU (0avgtext+0avgdata
> 1600maxresident)k
> 0inputs+0outputs (0major+399minor)pagefaults 0swaps
>
> So, CPU1 takes slightly longer in userspace, has less resident pages and less
> minor faults which is rather odd. Repeatedly running just one instance gives
> different results each time... disabling virtual address space randomisation
> solves that:
>
> echo 0 >/proc/sys/kernel/randomize_va_space
>
> which then gives me:
>
> CPU0: 1.00 520.20
> 6.18user 0.20system 0:06.59elapsed 96%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
> CPU1: 1.00 496.61
> 6.46user 0.14system 0:06.77elapsed 97%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
>
> CPU0: 1.00 521.10
> 6.13user 0.21system 0:06.57elapsed 96%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
> CPU1: 1.00 498.01
> 6.40user 0.18system 0:06.75elapsed 97%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
>
> which is rather more stable as far as resource usage goes between the two
> CPUs, but still an asymmetry in the reported bandwidths and times.
> So, this has ruled out differences in VA layout.
>
> Now for the interesting bit... it's important to understand what and how stuff
> is being measured. Looking at the bw_mem.c and associated source code, it
> measures the performance against the wall clock, which includes everything
> that the system is doing on each particular CPU.
> So, if a CPU is interrupted by another thread wanting to run, it'll affect the
> results. Hence, it's best to run on an otherwise quiet system, eg, without an
> init daemon (eg, booted with init=/bin/sh on the kernel command line - but
> note there won't be any job control, so ^C won't work!)
>
> However, continuing on...
>
> If I run bw_mem on just one CPU:
>
> CPU1: 1.00 2617.31
> 5.74user 0.18system 0:06.03elapsed 98%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
>
> Same number of iterations, same memory size, but notice that it appears to
> be a lot faster reported by bw_mem, but the time taken is about the same.
> cpufreq comes to mind, but that's disabled on this system.
>
> So, it brings up a rather obvious question: what exactly is bw_mem
> measuring, and is it measuring it correctly?
>
> $ /usr/bin/time taskset -c 1 ./bw_mem -P 1 -N 1000 1M fwr
> 1.00 2601.26
> 5.80user 0.16system 0:06.06elapsed 98%CPU (0avgtext+0avgdata
> 1700maxresident)k
> 0inputs+0outputs (0major+403minor)pagefaults 0swaps
> $ /usr/bin/time ./bw_mem -P 2 -N 1000 1M fwr ^CCommand terminated by
> signal 2 5.54user 0.13system 1:12.20elapsed 7%CPU (0avgtext+0avgdata
> 1696maxresident)k
> 0inputs+0outputs (0major+365minor)pagefaults 0swaps
>
> so requesting a parallelism of 2 results in the program never seemingly
> ending in a reasonable period of time, which suggests a bug somewhere.
> Are we sure that bw_mem is actually working as intended?
>
> Maybe if Larry is reading this, he could share some thoughts.
>
> --
> RMK's Patch system: http://www.armlinux.org.uk/developer/patches/
> FTTC broadband for 0.8mile line in suburbia: sync at 13.8Mbps down 630kbps
> up According to speedtest.net: 13Mbps down 490kbps up

Thanks for the detail explanation on LM bench test.

Initially, I tested to run my own's created memory test program and encountered the same unbalanced memory throughput whenever two threads are running on different cores. I tested to run the memory test program on either one core or two cores. The unbalanced memory throughput is seen when running on two cores. Hence, I tried out the bw_mem test as it is a general benchmark program and it appears to be alike to my own test case.

Attached is the memory test program memtest.c file and the Linux executable file. The memtest -a1 will forces the two threads running on different cores while memtest -a2 will forces the two threads running on one core.
May I know if is it possible if you could try out my test program on your iMX6D SoC (also dual core Cortex A9) board?

Below show the comparison between two threads running on one core vs two cores

One core:
========= Multi Thread =========

Thread 3067511920 - data size 1 MB, runs = 1000
Thread 3059123312 - data size 1 MB, runs = 1000
Thread :3059123312: Datarate: 974.887201 MB/s
Thread :3067511920: Datarate: 960.289834 MB/s
Thread :3067511920: Datarate: 1083.249741 MB/s
Thread :3059123312: Datarate: 1055.545769 MB/s
Thread :3067511920: Datarate: 1085.555446 MB/s
Thread :3059123312: Datarate: 1084.503430 MB/s
Thread :3067511920: Datarate: 1063.379303 MB/s
Thread :3059123312: Datarate: 1070.705338 MB/s
Thread :3067511920: Datarate: 1050.933243 MB/s
Thread :3059123312: Datarate: 1050.153330 MB/s
Thread :3067511920: Datarate: 1085.489144 MB/s
Thread :3059123312: Datarate: 1071.774560 MB/s
Thread :3067511920: Datarate: 1084.506795 MB/s
Thread :3059123312: Datarate: 1060.260066 MB/s
Thread :3067511920: Datarate: 1074.058027 MB/s
Thread :3059123312: Datarate: 1069.279388 MB/s
Thread :3067511920: Datarate: 1073.924924 MB/s
Thread :3059123312: Datarate: 1080.818992 MB/s
Thread :3067511920: Datarate: 1081.871683 MB/s
Thread :3067511920: Average Datarate: 1064.325814 MB/s
Thread :3059123312: Datarate: 1097.549768 MB/s
Thread :3059123312: Average Datarate: 1061.547784 MB/s
Finished!

Two cores:
========= Multi Thread =========

Thread 3067954288 - data size 1 MB, runs = 1000
Thread 3059565680 - data size 1 MB, runs = 1000
Thread :3067954288: Datarate: 741.930805 MB/s
Thread :3059565680: Datarate: 377.979641 MB/s
Thread :3067954288: Datarate: 741.976479 MB/s
Thread :3067954288: Datarate: 740.548015 MB/s
Thread :3059565680: Datarate: 376.706463 MB/s
Thread :3067954288: Datarate: 740.313260 MB/s
Thread :3067954288: Datarate: 740.363440 MB/s
Thread :3059565680: Datarate: 376.129877 MB/s
Thread :3067954288: Datarate: 740.056194 MB/s
Thread :3067954288: Datarate: 740.219191 MB/s
Thread :3059565680: Datarate: 376.114092 MB/s
Thread :3067954288: Datarate: 740.152311 MB/s
Thread :3067954288: Datarate: 724.094688 MB/s
Thread :3059565680: Datarate: 388.118117 MB/s
Thread :3067954288: Datarate: 740.556383 MB/s
Thread :3067954288: Average Datarate: 739.021077 MB/s
Thread :3059565680: Datarate: 1323.735631 MB/s
Thread :3059565680: Datarate: 2072.948256 MB/s
Thread :3059565680: Datarate: 2069.817984 MB/s
Thread :3059565680: Datarate: 2069.295149 MB/s
Thread :3059565680: Datarate: 2040.932474 MB/s
Thread :3059565680: Average Datarate: 1147.177768 MB/s
Finished!

Thanks
Tzy Way
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <malloc.h>
#include <pthread.h>
#include <semaphore.h>
#include <string.h>
#include <sys/time.h>
#include <sched.h>
#include <sys/resource.h>
#include <sys/mman.h>

#define MB (1024*1024)
#define QUAD_CORE 0

# define handle_error_en(en, msg) \
do { errno = en; perror(msg); exit(EXIT_FAILURE); } while (0);

unsigned long l_megabytes = 1;
unsigned long l_repeat = 1000;

void SingleFct(void)
{
unsigned long l_size = l_megabytes*MB;
unsigned char * l_memBuffer = memalign(32, l_size);
int i,j;
unsigned long l_time;
double l_datarate;
double t_datarate = 0.0;
struct timeval start, end;

printf("Single Thread - data size %ld MB, runs = %ld\n", l_megabytes, l_repeat);
for(j=0; j<10; j++)
{
gettimeofday(&start, NULL);
for(i=0; i<l_repeat; i++)
{
memset(l_memBuffer, 0x5a, l_size);
}
gettimeofday(&end, NULL);

l_time = (end.tv_sec*1000000 + end.tv_usec) - (start.tv_sec*1000000 + start.tv_usec);
// printf("end time = %u.%06u, start time = %u.%06u\n", end.tv_sec, end.tv_usec, start.tv_sec, start.tv_usec);
// printf("l_time = %u\n", l_time);
l_datarate = (double)l_size*(double)l_repeat/(double)l_time;
printf("Single Thread Datarate: %f MB/s\n", l_datarate);
t_datarate += l_datarate;
}
printf("Single Thread Average Datarate: %f MB/s\n", t_datarate/10.0f);
free(l_memBuffer);
}

void Fct(void)
{
unsigned long l_size = l_megabytes*MB;
unsigned char * l_memBuffer = memalign(32, l_size);
unsigned long l_time;
double l_datarate;
double t_datarate = 0.0;
int i,j;
struct timeval start, end;

printf("Thread %u - data size %ld MB, runs = %ld\n", (unsigned int)pthread_self(), l_megabytes, l_repeat);
for(j=0; j<10; j++)
{
gettimeofday(&start, NULL);
for(i=0; i<l_repeat; i++)
{
memset(l_memBuffer, 0x5a, l_size);
}
gettimeofday(&end, NULL);
l_time = (end.tv_sec*1000000 + end.tv_usec) - (start.tv_sec*1000000 + start.tv_usec);
l_datarate = (double)l_size*(double)l_repeat/(double)l_time;
printf("Thread :%u: Datarate: %lf MB/s\n", (unsigned int)pthread_self(), l_datarate);
t_datarate += l_datarate;

}

printf("Thread :%u: Average Datarate: %lf MB/s\n", (unsigned int)pthread_self(), t_datarate/10.0f);
free(l_memBuffer);

pthread_exit((void *) 0);
}

int main(int argc, char** argv) {

pthread_t t1, t2, t3, t4;

pthread_attr_t t1_attr, t2_attr, t3_attr, t4_attr;
cpu_set_t t1_cpuset, t2_cpuset, t3_cpuset, t4_cpuset;

int rv;
int affinity = 0;
int iarg = 1;

while (iarg < argc) {
if (!strcmp(argv[iarg],"-a1")) {
affinity = 1;
}
else if (!strcmp(argv[iarg],"-a2")) {
affinity = 2;
}
else if (!strcmp(argv[iarg],"-s1")) {
l_megabytes = 1;
}
else if (!strcmp(argv[iarg],"-s2")) {
l_megabytes = 10;
}
else if (!strcmp(argv[iarg],"-s3")) {
l_megabytes = 100;
}
else {
printf("Usage: %s [-a1|-a2 ] [-s1|-s2|-s3]\n",argv[0]);
exit(0);
}
iarg++;
}
printf("========= Single Thread =========\n\n");

SingleFct();

printf("\n\n========= Multi Thread =========\n\n");

rv = pthread_attr_init(&t1_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #1");

rv = pthread_attr_init(&t2_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #2");

if (affinity == 1)
{
CPU_ZERO(&t1_cpuset);
CPU_SET(0,&t1_cpuset);
rv = pthread_attr_setaffinity_np(&t1_attr,sizeof(t1_cpuset),&t1_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #1");

CPU_ZERO(&t2_cpuset);
CPU_SET(1,&t2_cpuset);
rv = pthread_attr_setaffinity_np(&t2_attr,sizeof(t2_cpuset),&t2_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #2");
}
else if (affinity == 2)
{
CPU_ZERO(&t1_cpuset);
CPU_SET(0,&t1_cpuset);
rv = pthread_attr_setaffinity_np(&t1_attr,sizeof(t1_cpuset),&t1_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #1");

CPU_ZERO(&t2_cpuset);
CPU_SET(0,&t2_cpuset);
rv = pthread_attr_setaffinity_np(&t2_attr,sizeof(t2_cpuset),&t2_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #2");
}

rv = pthread_create(&t1, &t1_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #1");

rv = pthread_create(&t2, &t2_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #2");

#if QUAD_CORE
rv = pthread_attr_init(&t3_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #3");

rv = pthread_attr_init(&t4_attr);
if (rv != 0) handle_error_en(rv,"pthread_attr_init thread #4");

if (affinity == 1)
{
CPU_ZERO(&t3_cpuset);
CPU_SET(2,&t3_cpuset);
rv = pthread_attr_setaffinity_np(&t3_attr,sizeof(t3_cpuset),&t3_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #3");

CPU_ZERO(&t4_cpuset);
CPU_SET(3,&t4_cpuset);
rv = pthread_attr_setaffinity_np(&t4_attr,sizeof(t4_cpuset),&t4_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #4");
}
else if (affinity == 2)
{
CPU_ZERO(&t3_cpuset);
CPU_SET(0,&t3_cpuset);
rv = pthread_attr_setaffinity_np(&t3_attr,sizeof(t3_cpuset),&t3_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #3");

CPU_ZERO(&t4_cpuset);
CPU_SET(0,&t4_cpuset);
rv = pthread_attr_setaffinity_np(&t4_attr,sizeof(t4_cpuset),&t4_cpuset);
if (rv != 0) handle_error_en(rv,"pthread_attr_setaffinity_np thread #4");
}

rv = pthread_create(&t3, &t3_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #3");

rv = pthread_create(&t4, &t4_attr, (void *)&Fct, (void *)NULL);
if (rv != 0) handle_error_en(rv,"pthread_create thread #4");

#endif

pthread_join(t1, NULL);
pthread_join(t2, NULL);
#if QUAD_CORE
pthread_join(t3, NULL);
pthread_join(t4, NULL);
#endif

printf("Finished!\n");
return 0;
}

Attachment: memtest_2
Description: memtest_2