Re: [LKP] Re: [sched] bdfcae1140: will-it-scale.per_thread_ops -37.0% regression

[Xing Zhengjun]

On 10/7/2020 10:50 PM, Mathieu Desnoyers wrote:
> ----- On Oct 2, 2020, at 4:33 AM, Rong Chen rong.a.chen@xxxxxxxxx wrote:
>
> > Greeting,
> >
> > FYI, we noticed a -37.0% regression of will-it-scale.per_thread_ops due to
> > commit:
> >
> >
> > commit: bdfcae11403e5099769a7c8dc3262e3c4193edef ("[RFC PATCH 2/3] sched:
> > membarrier: cover kthread_use_mm (v3)")
> > url:
> > https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/Membarrier-updates/20200925-012549
> > base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git
> > 848785df48835eefebe0c4eb5da7690690b0a8b7
> >
> > in testcase: will-it-scale
> > on test machine: 104 threads Skylake with 192G memory
> > with following parameters:
> >
> > 	nr_task: 50%
> > 	mode: thread
> > 	test: context_switch1
> > 	cpufreq_governor: performance
> > 	ucode: 0x2006906
> >
> > test-description: Will It Scale takes a testcase and runs it from 1 through to n
> > parallel copies to see if the testcase will scale. It builds both a process and
> > threads based test in order to see any differences between the two.
> > test-url: https://github.com/antonblanchard/will-it-scale
>
> Hi,
>
> I would like to report what I suspect is a random thread placement issue in the
> context_switch1 test used by the 0day bot when running on a machine with hyperthread
> enabled.
>
> AFAIU the test code uses hwloc for thread placement which should theoretically ensure
> that each thread is placed on same processing unit, core and numa node between runs.
>
> We can find the test code here:
>
> https://github.com/antonblanchard/will-it-scale/blob/master/tests/context_switch1.c
>
> And the main file containing thread setup is here:
>
> https://github.com/antonblanchard/will-it-scale/blob/master/main.c
>
> AFAIU, the test is started without the "-m" switch, which therefore affinitizes
> tasks on cores rather than on processing units (SMT threads).
>
> When testcase() creates the child thread with new_task(), it basically issues:
>
>    pthread_create(&threads[nr_threads++], NULL, func, arg);
>
> passing a NULL pthread_attr_t, and not executing any pre_trampoline on the child.
> The pre_trampoline would have issued hwloc_set_thread_cpubind if it were executed on
> the child, but it's not. Therefore, we expect the cpu affinity mask of the parent to
> be copied on clone and used by the child.
>
> A quick test on a machine with hyperthreading enabled shows that the cpu affinity mask
> for the parent and child has two bits set:
>
> taskset -p 1868607
> pid 1868607's current affinity mask: 10001
> taskset -p 1868606
> pid 1868606's current affinity mask: 10001
>
> So AFAIU the placement of the parent and child will be random on either the same
> processing unit, or on separate processing units within the same core.
>
> I suspect this randomness can significantly affect the performance number between
> runs, and trigger unwarranted performance regression warnings.
>
> Thanks,
>
> Mathieu
Yes, the randomness may happen in some special cases.  But in 0-day, we 
test multi times (>=3), the report is the average number.
For this case, we test 4 times, it is stable, the wave is ±  2%.
So I don't think the -37.0% regression is caused by the randomness.

0/stats.json:  "will-it-scale.per_thread_ops": 105228,
1/stats.json:  "will-it-scale.per_thread_ops": 100443,
2/stats.json:  "will-it-scale.per_thread_ops": 98786,
3/stats.json:  "will-it-scale.per_thread_ops": 102821,

c2daff748f0ea954 bdfcae11403e5099769a7c8dc32
---------------- ---------------------------
         %stddev     %change         %stddev
             \          |                \
    161714 ±  2%     -37.0%     101819 ±  2%  will-it-scale.per_thread_ops


--
Zhengjun Xing