Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment

[Anders Roxell]

On Sat, 2 May 2020 at 04:11, David Gow <davidgow@xxxxxxxxxx> wrote:
>
> On Sat, May 2, 2020 at 4:31 AM Brendan Higgins
> <brendanhiggins@xxxxxxxxxx> wrote:
> >
> > On Fri, May 1, 2020 at 1:35 AM Anders Roxell <anders.roxell@xxxxxxxxxx> wrote:
> > >
> > > Make it easier to enable all KUnit fragments.  This is needed for kernel
> > > test-systems, so its easy to get all KUnit tests enabled and if new gets
> > > added they will be enabled as well.  Fragments that has to be builtin
> > > will be missed if CONFIG_KUNIT_RUN_ALL is set as a module.
> > >
> > > Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if
> > > someone wants that even though KUNIT_RUN_ALL is enabled.
> >
> > I would LOVE IT, if you could make this work! I have been trying to
> > figure out the best way to run all KUnit tests for a long time now.
> >
> > That being said, I am a bit skeptical that this approach will be much
> > more successful than just using allyesconfig. Either way, there are
> > tests coming down the pipeline that are incompatible with each other
> > (the KASAN test and the KCSAN test will be incompatible). Even so,
> > tests like the apparmor test require a lot of non-default
> > configuration to compile. In the end, I am not sure how many tests we
> > will really be able to turn on this way.
> >
> > Thoughts?
>
> I think there's still some value in this which the allyesconfig option
> doesn't provide. As you point out, it's not possible to have a generic
> "run all tests" option due to potential conflicting dependencies, but
> this does provide a way to run all tests for things enabled in the
> current config. This could be really useful for downstream developers
> who want a way of running all tests relevant to their config without
> the overhead of running irrelevant tests (e.g., for drivers they don't
> build).

It will solve that as well as for a tester doesn't have to go through all KUnit
tests fragments to turn them on.

> Using allyesconfig doesn't make that distinction.

We could also create a config fragment file in kernel/configs/kunit.config
where we set
------start
CONFIG_KUNIT=y
CONFIG_KUNIT_RUN_ALL=y
CONFIG_SECURITY_APPARMOR=y
------end


So, these two can only be enabled if KUNIT=y
CONFIG_KUNIT_DRIVER_PE_TEST=y
CONFIG_PM_QOS_KUNIT_TEST=y

and for this one we have a pre-request of SECURITY_APPARMOR=y
CONFIG_SECURITY_APPARMOR_KUNIT_TEST=y

Other tests solves the dependencies with 'select' like
CONFIG_EXT4_KUNIT_TESTS, that adds this row in
fs/ext4/Kconfig, 'select EXT4_FS'

But I think we should try to minimize the number of 'select' statements,
in order to avoid circular dependencies and unexpected behaviours.
Maybe we should add the CONFIG_EXT4_FS=y into the kunit.config
file instead ?


>
> Ultimately, we'll probably still want something which enables a
> broader set of tests for upstream development: whether that's based on
> this, allyesconfig, or something else entirely remains to be seen, I
> think. I suspect we're going to end up with something
> subsystem-specific (having a kunitconfig per subsystem, or a testing
> line in MAINTAINERS or similar are ideas which have been brought up in
> the past).
>
> This is a great looking tool to have in the toolbox, though.

I agree!

I'll prepare a patchset with individual patches as was suggested by Marco
shortly.

Cheers,
Anders