Re: [PATCH v2 5/9] test_sysctl: add generic script to expand on tests

[Kees Cook]

On Fri, Feb 10, 2017 at 4:36 PM, Luis R. Rodriguez <mcgrof@xxxxxxxxxx> wrote:
> This adds a generic script to let us more easily add more tests
> cases. Since we really have only two types of tests cases just
> fold them into the one file. Each test unit is now identified
> into its separate function:
>
>   # ./sysctl.sh -l
> Test ID list:
>
> TEST_ID x NUM_TEST
> TEST_ID:   Test ID
> NUM_TESTS: Number of recommended times to run the test
>
> 0001 x 1 - tests proc_dointvec_minmax()
> 0002 x 1 - tests proc_dostring()
>
> For now we start off with what we had before, and run only each test once.
> We can now watch a test case until it fails:
>
> ./sysctl.sh -w 0002
>
> We can also run a test case x number of times, say we want to run
> a test case 100 times:
>
> ./sysctl.sh -c 0001 100
>
> To run a test case only once, for example:
>
> ./sysctl.sh -s 0002
>
> The default settings are specified at the top of sysctl.sh.
>
> Signed-off-by: Luis R. Rodriguez <mcgrof@xxxxxxxxxx>

I'm not a fan of this: it consolidates tests when it's not needed and
creates a test running infrastructure at the wrong level of
abstraction. I'd like to see individual tests that are one-off
runnable. Whatever consumes the tools/testing/selftests/ tree is what
should be doing the -w, -c, etc style options.

-Kees

-- 
Kees Cook
Pixel Security