[PATCH v2] Documentation: kunit: Update kunit_tool page

From: David Gow
Date: Fri Apr 16 2021 - 23:46:00 EST

The kunit_tool documentation page was pretty minimal, and a bit
outdated. Update it and flesh it out a bit.

In particular,
- Mention that .kunitconfig is now in the build directory
- Describe the use of --kunitconfig to specify a different config
- Mention the split functionality (i.e., commands other than 'run')
- Describe --raw_output and kunit.py parse
- Mention the globbing support
- Provide a quick overview of other options, including --build_dir and

Note that this does overlap a little with the new running_tips page. I
don't think it's a problem having both: this page is supposed to be a
bit more of a reference, rather than a list of useful tips, so the fact
that they both describe the same features isn't a problem.

Signed-off-by: David Gow <davidgow@xxxxxxxxxx>
Reviewed-by: Daniel Latypov <dlatypov@xxxxxxxxxx>

Adopted the changes from Daniel.

Changes since v1:
- Mention that the default build directory is '.kunit' when discussing
'.kunitconfig' files.
- Reword the discussion of 'CONFIG_KUNIT_ALL_TESTS' under '--alltests'

Documentation/dev-tools/kunit/kunit-tool.rst | 140 +++++++++++++++++--
1 file changed, 132 insertions(+), 8 deletions(-)

diff --git a/Documentation/dev-tools/kunit/kunit-tool.rst b/Documentation/dev-tools/kunit/kunit-tool.rst
index 29ae2fee8123..4247b7420e3b 100644
--- a/Documentation/dev-tools/kunit/kunit-tool.rst
+++ b/Documentation/dev-tools/kunit/kunit-tool.rst
@@ -22,14 +22,19 @@ not require any virtualization support: it is just a regular program.
What is a .kunitconfig?

-It's just a defconfig that kunit_tool looks for in the base directory.
-kunit_tool uses it to generate a .config as you might expect. In addition, it
-verifies that the generated .config contains the CONFIG options in the
-.kunitconfig; the reason it does this is so that it is easy to be sure that a
-CONFIG that enables a test actually ends up in the .config.
+It's just a defconfig that kunit_tool looks for in the build directory
+(``.kunit`` by default). kunit_tool uses it to generate a .config as you might
+expect. In addition, it verifies that the generated .config contains the CONFIG
+options in the .kunitconfig; the reason it does this is so that it is easy to
+be sure that a CONFIG that enables a test actually ends up in the .config.

-How do I use kunit_tool?
+It's also possible to pass a separate .kunitconfig fragment to kunit_tool,
+which is useful if you have several different groups of tests you wish
+to run independently, or if you want to use pre-defined test configs for
+certain subsystems.
+Getting Started with kunit_tool

If a kunitconfig is present at the root directory, all you have to do is:

@@ -48,10 +53,129 @@ However, you most likely want to use it with the following options:

.. note::
This command will work even without a .kunitconfig file: if no
- .kunitconfig is present, a default one will be used instead.
+ .kunitconfig is present, a default one will be used instead.
+If you wish to use a different .kunitconfig file (such as one provided for
+testing a particular subsystem), you can pass it as an option.
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig

For a list of all the flags supported by kunit_tool, you can run:

.. code-block:: bash

./tools/testing/kunit/kunit.py run --help
+Configuring, Building, and Running Tests
+It's also possible to run just parts of the KUnit build process independently,
+which is useful if you want to make manual changes to part of the process.
+A .config can be generated from a .kunitconfig by using the ``config`` argument
+when running kunit_tool:
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py config
+Similarly, if you just want to build a KUnit kernel from the current .config,
+you can use the ``build`` argument:
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py build
+And, if you already have a built UML kernel with built-in KUnit tests, you can
+run the kernel and display the test results with the ``exec`` argument:
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py exec
+The ``run`` command which is discussed above is equivalent to running all three
+of these in sequence.
+All of these commands accept a number of optional command-line arguments. The
+``--help`` flag will give a complete list of these, or keep reading this page
+for a guide to some of the more useful ones.
+Parsing Test Results
+KUnit tests output their results in TAP (Test Anything Protocol) format.
+kunit_tool will, when running tests, parse this output and print a summary
+which is much more pleasant to read. If you wish to look at the raw test
+results in TAP format, you can pass the ``--raw_output`` argument.
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py run --raw_output
+.. note::
+ The raw output from test runs may contain other, non-KUnit kernel log
+ lines.
+If you have KUnit results in their raw TAP format, you can parse them and print
+the human-readable summary with the ``parse`` command for kunit_tool. This
+accepts a filename for an argument, or will read from standard input.
+.. code-block:: bash
+ # Reading from a file
+ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
+ # Reading from stdin
+ dmesg | ./tools/testing/kunit/kunit.py parse
+This is very useful if you wish to run tests in a configuration not supported
+by kunit_tool (such as on real hardware, or an unsupported architecture).
+Filtering Tests
+It's possible to run only a subset of the tests built into a kernel by passing
+a filter to the ``exec`` or ``run`` commands. For example, if you only wanted
+to run KUnit resource tests, you could use:
+.. code-block:: bash
+ ./tools/testing/kunit/kunit.py run 'kunit-resource*'
+This uses the standard glob format for wildcards.
+Other Useful Options
+kunit_tool has a number of other command-line arguments which can be useful
+when adapting it to fit your environment or needs.
+Some of the more useful ones are:
+ Lists all of the available options. Note that different commands
+ (``config``, ``build``, ``run``, etc) will have different supported
+ options. Place ``--help`` before the command to list common options,
+ and after the command for options specific to that command.
+ Specifies the build directory that kunit_tool will use. This is where
+ the .kunitconfig file is located, as well as where the .config and
+ compiled kernel will be placed. Defaults to ``.kunit``.
+ Specifies additional options to pass to ``make`` when compiling a
+ kernel (with the ``build`` or ``run`` commands). For example, to enable
+ compiler warnings, you can pass ``--make_options W=1``.
+ Builds a UML kernel with all config options enabled using ``make
+ allyesconfig``. This allows you to run as many tests as is possible,
+ but is very slow and prone to breakage as new options are added or
+ modified. In most cases, enabling all tests which have satisfied
+ dependencies by adding ``CONFIG_KUNIT_ALL_TESTS=1`` to your
+ .kunitconfig is preferable.
+There are several other options (and new ones are often added), so do check
+``--help`` if you're looking for something not mentioned here.