Re: [PATCH] kunit: tool: improve compatibility of kunit_parser with KTAP specification

[Brendan Higgins]

On Fri, Aug 20, 2021 at 1:01 PM Rae Moar <rmoar@xxxxxxxxxx> wrote:
>
> Update to kunit_parser to improve compatibility with KTAP
> specification including arbitrarily nested tests. Patch accomplishes
> three major changes:
>
> - Use a general Test object to represent all tests rather than TestCase
> and TestSuite objects. This allows for easier implementation of arbitrary
> levels of nested test and promotes the idea that both test suites and test
> cases are tests.
>
> - Print errors incrementally rather than all at once after the
> parsing finishes to maximize information given to the user in the
> case of the parser given invalid input and to increase the helpfulness
> of the timestamps given during printing.
>
> - Increase compatibility for different formats of input. Arbitrary levels
> of nested tests supported. Also, test cases and test suites are now
> supported to be present on the same level of testing.
>
> This patch now implements the KTAP specification as described here:
> https://lore.kernel.org/linux-kselftest/CA+GJov6tdjvY9x12JsJT14qn6c7NViJxqaJk+r-K1YJzPggFDQ@xxxxxxxxxxxxxx/.
>
> This patch adjusts the kunit_tool_test.py file to check for
> the correct outputs from the new parser and adds a new test to check
> the parsing for a KTAP result log with correct format for multiple nested
> subtests (test_is_test_passed-all_passed_nested.log).
>
> This patch also alters the kunit_json.py file to allow for arbitrarily
> nested tests.
>
> Signed-off-by: Rae Moar <rmoar@xxxxxxxxxx>

One minor question/potential issue below, otherwise:

Reviewed-by: Brendan Higgins <brendanhiggins@xxxxxxxxxx>

[...]
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index 75045aa0f8a1..ca760ee32096 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -106,10 +106,10 @@ class KUnitParserTest(unittest.TestCase):
>                 with open(log_path) as file:
>                         result = kunit_parser.extract_tap_lines(file.readlines())
>                 self.assertContains('TAP version 14', result)
> -               self.assertContains('   # Subtest: example', result)
> -               self.assertContains('   1..2', result)
> -               self.assertContains('   ok 1 - example_simple_test', result)
> -               self.assertContains('   ok 2 - example_mock_test', result)
> +               self.assertContains('# Subtest: example', result)
> +               self.assertContains('1..2', result)
> +               self.assertContains('ok 1 - example_simple_test', result)
> +               self.assertContains('ok 2 - example_mock_test', result)
>                 self.assertContains('ok 1 - example', result)

Do you have any test cases which accept "ok/not ok" lines without a
"-"? (Or tests for any other KTAP compliant Kselftest output?)

>         def test_output_with_prefix_isolated_correctly(self):

Cheers!