[PATCH 0/2] RFC: A common test container

From: steffen.schwigon
Date: Tue Oct 23 2012 - 04:03:33 EST

From: Steffen Schwigon <steffen.schwigon@xxxxxxx>


ABSTRACT: This patch set introduces a mature test library and t/ as a
subdirectory for placing tests that can be run via the existing Linux
tool 'prove' (available in most major Linux distributions). I already
presented and discussed the approach with several people at LinuxCon
Europe 2011 as part of my talk "Linux and virtualization testing".

Tests in t/ are assumed to produce TAP, the "Test Anything Protocol"
(see http://testanything.org) which is well established in many test
communities. The advantage for Linux is that it is easy to generate
which makes it suitable for restricted environments like OS testing.

To write such tests for Linux a library with special utilities is
provided: 'bash-test-utils'.

It provides utility functions to write tests in baremetal and Xen/KVM
contexts (host and guest support) and was polished to work with i686,
x86_64, and ARM under lots of major Linux distributions (Debian,
Ubuntu, RHEL, SLES, Slackware, Fedora, openSUSE). We actively use this
library in all the above contexts for about 2 years.

Other TAP libraries (without special Linux support) are available for
many programming languages.

The autotest project already provides TAP support.


Provide a container subdir where to place functional tests that are
*in sync* with Linux' git history.

Such tests are usually "unit tests" for features and can also be used
for bisecting regressions of those feature, e.g., by executing 'prove'
with 'ktest' (already part of Linux).

It is possible to organize such tests in a subdir hierarchy (though
not contained with this patch) where separate functional areas as well
as Linux contributors and vendors can be organized without conflicts
and executed separately, for example:


Separate testprojects like autotest can be kept in sync with the Linux
kernel via git submodules under t/ and called via frontend wrapper
scripts to trigger TAP output.


Testscripts are by convention named "*.t" to be chosen automatically
by 'prove' if executing directories and to differentiate between tests
and other utilities. The subdirectory t/ is also a default convention
used by 'prove'.

Tests can be executed like this:

$ prove -v t/example/basic.t
$ prove -v t/example/
$ prove -r t/


Q: Why not submit it to one of the Linux test projects, like LTP or
A: External test projects are not in sync with the Linux kernel. They
regularly do not fit older or newer kernels.

Having a test container (subdir t/ with the TAP convention) in
Linux itself allows

- contributors, vendors, and companies to place a functional test
nearby the functionality it belongs to.
- integrate and sync with existing test projects, e.g., via git
submodules or wrapper scripts.

Q: Why TAP? Why not XUNIT or other test protocols?
A: There are not that many test protocols. TAP is well established,
easy, language and platform independent, and already supported in
every Linux system out-of-the-box. It looks like this:

ok - feature a
not ok - feature b
# diagnostic lines with details
ok - feature c # SKIP not done for this architecture
not ok - feature d # TODO yet to be implemented

This can be produced even with the most minimal tools like printf
from a test inside a strange subsystem, though for practically all
major programming languages libraries exist. And for evaluating
tests the tool 'prove' already exists in your Linux system.

The syntax is easy, yet allows expressing complex test results if
needed. Besides the #TODO/#SKIP semantics it also allows to embed
structured data, eg. for benchmark results.

Q: Do we need more framework overhead to execute the tests?
A: Not in the Linux kernel. There are enough tools to execute them:

- prove ... to run functional tests
- ktest ... to compile a kernel and boot a machine

Everything is in place, we just need the common container, per

Steffen Schwigon (2):
bash-test-utils: support writing functional tests
A common test subdirectory

t/examples/basic.t | 49 ++
tools/testing/tap/bash-test-utils | 1508 +++++++++++++++++++++++++++++++++
tools/testing/tap/bash-test-utils.txt | 320 +++++++
3 files changed, 1877 insertions(+)
create mode 100755 t/examples/basic.t
create mode 100755 tools/testing/tap/bash-test-utils
create mode 100644 tools/testing/tap/bash-test-utils.txt


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/