Re: [KTAP V2 PATCH v2] ktap_v2: add test metadata

[Rae Moar]

On Sun, Feb 4, 2024 at 8:03 AM Kees Cook <kees@xxxxxxxxxx> wrote:
>
>
>
> On January 26, 2024 11:14:26 PM GMT+01:00, Rae Moar <rmoar@xxxxxxxxxx> wrote:
> > KTAP version 2
> > # ktap_test: main
> > # ktap_arch: uml
> > 1..1
> >     KTAP version 2
> >     # ktap_test: suite_1
> >     # ktap_subsystem: example
> >     # ktap_test_file: lib/test.c
>
> I think it's a mistake to mix "diagnostics" lines with semantic lines. Since the diagnostic prefix is [# ] (hash space) how about make the test metadata lines be [#:] (hash colon). For example:
>
>
>      1..2
>      ok 1 test_1
>      #:ktap_test: test_2
>      #:ktap_speed: very_slow
>      #:custom_is_flaky: true
>      # format-free stuff goes here
>      ok 2 test_2
> ...

Hello!

I really like this idea. The reason I chose the diagnostic line format
was to make it easier for existing parsers to parse the KTAP Metadata
lines. However, if it won't be too much of an issue for current
parsers, I think this idea would be better. So I am happy to change
this in the next version if there are no complaints.

>
> > ok 1 test_suite
> >
> >The changes to the KTAP specification outline the format, location, and
> >different types of metadata.
> >
> >Here is a link to a version of the KUnit parser that is able to parse test
> >metadata lines for KTAP version 2. Note this includes test metadata
> >lines for the main level of KTAP.
> >
> >Link: https://kunit-review.googlesource.com/c/linux/+/5889
> >
> >Signed-off-by: Rae Moar <rmoar@xxxxxxxxxx>
> >---
> > Documentation/dev-tools/ktap.rst | 163 ++++++++++++++++++++++++++++++-
> > 1 file changed, 159 insertions(+), 4 deletions(-)
> >
> >diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/ktap.rst
> >index ff77f4aaa6ef..4480eaf5bbc3 100644
> >--- a/Documentation/dev-tools/ktap.rst
> >+++ b/Documentation/dev-tools/ktap.rst
> >@@ -17,19 +17,20 @@ KTAP test results describe a series of tests (which may be nested: i.e., test
> > can have subtests), each of which can contain both diagnostic data -- eg., log
> > lines -- and a final result. The test structure and results are
> > machine-readable, whereas the diagnostic data is unstructured and is there to
>
> We even say it's unstructured... :)
>
>
> >+prefix must not include spaces or the characters ":" or "_".
>
> Why not _?

My thought here was that the first "_" character in the KTAP Metadata
line would indicate the separation between the prefix and metadata
type. So the prefix could not contain "_". This makes it easier to
parse. I'm inclined to keep this but also willing to change it if
there is a proposed prefix that contains "_".

Thanks!
-Rae

>
> --
> Kees Cook