Re: [PATCH v3 1/3] kunit: Report the count of test suites in a module

[Janusz Krzysztofik]

Hi Rae,

On Thursday, 3 August 2023 22:57:43 CEST Rae Moar wrote:
> On Mon, Jul 31, 2023 at 10:12 AM Janusz Krzysztofik
> <janusz.krzysztofik@xxxxxxxxxxxxxxx> wrote:
> >
> > According to KTAP specification[1], results should always start from a
> > header that provides a TAP protocol version, followed by a test plan with
> > a count of items to be executed.  That pattern should be followed at each
> > nesting level.  In the current implementation of the top-most, i.e., test
> > suite level, those rules apply only for test suites built into the kernel,
> > executed and reported on boot.  Results submitted to dmesg from kunit test
> > modules loaded later are missing those top-level headers.
> >
> > As a consequence, if a kunit test module provides more than one test suite
> > then, without the top level test plan, external tools that are parsing
> > dmesg for kunit test output are not able to tell how many test suites
> > should be expected and whether to continue parsing after complete output
> > from the first test suite is collected.
> >
> > Submit the top-level headers also from the kunit test module notifier
> > initialization callback.
> >
> > [1] https://docs.kernel.org/dev-tools/ktap.html#
> >
> > Signed-off-by: Janusz Krzysztofik <janusz.krzysztofik@xxxxxxxxxxxxxxx>
> > ---
> 
> Hi!
> 
> I think this is a really great idea to improve the KTAP compatibility
> for module output. I do agree with Mauro and I wonder if this could be
> replaced with using kunit_exec_run_tests. However, if the output of
> 1..0 for a module with no KUnit tests run is not wanted, 

I do believe we really don't want that.  As soon as kunit framework registers 
its notifier callbacks, those callbacks are executed by generic module 
handling code on load / unload of every module, not only those providing kunit 
tests.  If our module initialization callback called unmodified 
kunit_exec_run_tests() that deliberately prints these two lines 
unconditionally:

	KTAP version 1
	1..n

then there would be a lot of unnecessary noise.

To avoid that noise, I decided to teach the callback itself to display the 
header with the number of test suits provided by the module before processing 
them if there is at least one, and be silent otherwise.  But since both you 
and Mauro think that kunit_exec_run_tests() should be reused, I can do that by 
moving that logic to kunit_exec_run_tests() and passing an additional flag 
that controls that logic from kunit_module_init() to kunit_exec_run_tests().  
Would that approach be more acceptable?

> I am ok with
> this change as is.
> 
> LGTM.
> 
> Tested-by: Rae Moar <rmoar@xxxxxxxxxx>

Thank you for testing.

Janusz

> 
> -Rae
> 
> >  lib/kunit/test.c | 5 +++++
> >  1 file changed, 5 insertions(+)
> >
> > diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> > index 84e4666555c94..a29ca1acc4d81 100644
> > --- a/lib/kunit/test.c
> > +++ b/lib/kunit/test.c
> > @@ -729,6 +729,11 @@ EXPORT_SYMBOL_GPL(__kunit_test_suites_exit);
> >  #ifdef CONFIG_MODULES
> >  static void kunit_module_init(struct module *mod)
> >  {
> > +       if (mod->num_kunit_suites > 0) {
> > +               pr_info("KTAP version 1\n");
> > +               pr_info("1..%d\n", mod->num_kunit_suites);
> > +       }
> > +
> >         __kunit_test_suites_init(mod->kunit_suites, mod-
>num_kunit_suites);
> >  }
> >
> > --
> > 2.41.0
> >
> > --
> > You received this message because you are subscribed to the Google Groups 
"KUnit Development" group.
> > To unsubscribe from this group and stop receiving emails from it, send an 
email to kunit-dev+unsubscribe@xxxxxxxxxxxxxxxx.
> > To view this discussion on the web visit https://groups.google.com/d/
msgid/kunit-dev/20230731141021.2854827-6-janusz.krzysztofik%40linux.intel.com.
>