RE: [PATCH v7 00/13] selftests/resctrl: Add resctrl selftest

[Yu, Fenghua]

>On Friday, May 10, 2019 10:36 AM
>Andre Przywara [mailto:andre.przywara@xxxxxxx] wrote:
> On Sat,  9 Feb 2019 18:50:29 -0800
> Fenghua Yu <fenghua.yu@xxxxxxxxx> wrote:
> 
> Hi Fenghua, Babu,
> 
> > With more and more resctrl features are being added by Intel, AMD and
> > ARM, a test tool is becoming more and more useful to validate that
> > both hardware and software functionalities work as expected.
> 
> That's very much appreciated! We decided to use that tool here to detect
> regressions in James' upcoming resctrl rework series. While doing so we
> spotted some shortcomings:

This is great!

> - There is some unconditional x86 inline assembly which obviously breaks
> the build on ARM.

Will fix this as much as possible.

BTW, does ARM support perf imc_count events which are used in CAT tests?

> - The result output is somewhat confusing, I find it hard to see whether a
> test succeeded or not, also it's not easily parseable by scripts.
> - There are some basic tests missing (does the filesystem exist? Is the
> mount point visible?)

Is it OK to fold the basic tests into bigger tests? E.g. the filesystem and mount point
tests are actually part of CAT test and we just dump the sub-test results during CAT test.
Seems your test results show the sub-test results during CAT test.

> - Some tests create files in the current directory. This is somewhat confusing
> if you happen to be in some sysfs directory, for instance. I don't think we
> really need temporary files (pipes should cover most of the cases), but in
> case we do, we should create them in some /tmp directory, probably by
> using a glibc function for this.

You are right. Will generate the temp file in /tmp.

> - There were some problems that newer GCC and clang complained about.

Which GCC do you use to see the issues? We use 8.2.1 and no compilation issue is reported.

> 
> For now I didn't look too closely at the actual test algorithms, but decided
> to fix those things first. I replied to some of the smaller issues I spotted on
> the way in the individual patch mails.
> 
> For the bigger things (new tests, result formatting) I just went ahead and
> fixed them in my own branch [1] (work in progress!). I went with TAP
> formatting, because this is both readable by humans and can be processed
> by the "prove" tool (or any other simple UNIX tool, for that matter).
> 
> I used this amended version to do some regression testing on James'
> resctrl rework series, which involved boot testing more than 100 patches, so
> automation is key here. The result looks like this:
> ===================
> TAP version 13
> ok kernel supports resctrl filesystem
> ok resctrl mountpoint "/sys/fs/resctrl" exists # resctrl filesystem not
> mounted # Starting MBM BW change ...
> ok MBM: bw change # SKIP memory bandwidth not supported # Starting
> MBA Schemata Change ...
> ok MBA: change-schemata # SKIP memory bandwidth not supported #
> Starting CQM test ...
> ok mounting resctrl to "/sys/fs/resctrl"
> ok CQM: test # SKIP L3 not supported
> # Starting CAT test ...
> ok mounting resctrl to "/sys/fs/resctrl"
> cache size :31457280
> ok writing benchmark parameters to resctrl FS ok Write schema "L3:0=7fff"
> to resctrl FS ok cache difference more than 4 % # Percent diff=16 # Number
> of bits: 15 # Avg_llc_perf_miss: 306994 # Allocated cache lines: 368640 ok
> CAT: test
> 1..11
> =================
> 
> Appreciate any feedback on this!
> 
> Cheers,
> Andre.
> 
> [1] http://linux-arm.org/git?p=linux-ap.git;a=shortlog;h=refs/heads/resctrl-
> selftests-wip
> git://linux-arm.org/linux-ap.git    branch: resctrl-selftests-wip