Re: RFC: KTAP documentation - expected messages

[David Gow]

On Mon, Jun 22, 2020 at 6:45 AM Frank Rowand <frowand.list@xxxxxxxxx> wrote:
>
> Tim Bird started a thread [1] proposing that he document the selftest result
> format used by Linux kernel tests.
>
> [1] https://lore.kernel.org/r/CY4PR13MB1175B804E31E502221BC8163FD830@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> The issue of messages generated by the kernel being tested (that are not
> messages directly created by the tests, but are instead triggered as a
> side effect of the test) came up.  In this thread, I will call these
> messages "expected messages".  Instead of sidetracking that thread with
> a proposal to handle expected messages, I am starting this new thread.

Thanks for doing this: I think there are quite a few tests which could
benefit from something like this.

I think there were actually two separate questions: what do we do with
unexpected messages (most of which I expect are useless, but some of
which may end up being related to an unexpected test failure), and how
to have tests "expect" a particular message to appear. I'll stick to
talking about the latter for this thread, but even there there's two
possible interpretations of "expected messages" we probably want to
explicitly distinguish between: a message which must be present for
the test to pass (which I think best fits the "expected message"
name), and a message which the test is likely to produce, but which
shouldn't alter the result (an "ignored message"). I don't see much
use for the latter at present, but if we wanted to do more things with
messages and had some otherwise very verbose tests, it could
potentially be useful.

The other thing I'd note here is that this proposal seems to be doing
all of the actual message filtering in userspace, which makes a lot of
sense for kselftest tests, but does mean that the kernel can't know if
the test has passed or failed. There's definitely a tradeoff between
trying to put too much needless string parsing in the kernel and
having to have a userland tool determine the test results. The
proposed KCSAN test suite[1] is using tracepoints to do this in the
kernel. It's not the cleanest thing, but there's no reason KUnit or
similar couldn't implement a nicer API around it.

[1]: https://lkml.org/lkml/2020/6/22/1506

> I implemented an API for expected messages that are triggered by tests
> in the Devicetree unittest code, with the expectation that the specific
> details may change when the Devicetree unittest code adapts the KUnit
> API.  It seems appropriate to incorporate the concept of expected
> messages in Tim's documentation instead of waiting to address the
> subject when the Devicetree unittest code adapts the KUnit API, since
> Tim's document may become the kernel selftest standard.

Is having a nice way to handle expected messages the only thing
holding up porting this to KUnit?

> Instead of creating a very long email containing multiple objects,
> I will reply to this email with a separate reply for each of:
>
>   The "expected messages" API implemention and use can be from
>   drivers/of/unittest.c in the mainline kernel.
>
>   of_unittest_expect - A proof of concept perl program to filter console
>                        output containing expected messages output
>
>                        of_unittest_expect is also available by cloning
>                        https://github.com/frowand/dt_tools.git
>
>   An example raw console output with timestamps and expect messages.
>
>   An example of console output processed by filter program
>   of_unittest_expect to be more human readable.  The expected
>   messages are not removed, but are flagged.
>
>   An example of console output processed by filter program
>   of_unittest_expect to be more human readable.  The expected
>   messages are removed instead of being flagged.

Cheers,
-- David