Re: [PATCH 00/17] nolibc: add preliminary self tests

[Paul E. McKenney]

On Tue, Jul 19, 2022 at 11:44:31PM +0200, Willy Tarreau wrote:
> Hi Paul,
> 
> as previously promised, here comes the nolibc update which introduces the
> minimal self-test infrastructure that aims at being reasonably easy to
> expand further.
> 
> It's based on your branch "dev.2022.06.30b" that contains the previous
> minor fixes that aimed at addressing Linus' concerns about the build
> process inconsistencies.
> 
> The way it works tries to mimmick as much as possible the regular build
> process, so that it reuses the same ARCH, CC, CROSS_COMPILE to build the
> test program, that will be embedded into an initramfs and the kernel is
> (re)built with that initramfs. Then you can decide to run that kernel
> under QEMU for the supported archs, and the output of the tests appears
> in an output text file in a format that's easily greppable and diffable.
> A single target "run" does everything.
> 
> By default it will reuse your existing .config (so that developers
> continue to use their regular config handling), though it can also
> create a known-to-work defconfig for each arch. The reason behind this
> is that it took me a moment to figure certain defconfig + machine name
> combinations and I found it better to put them there once for all.
> 
> I've successfully tested it on arm, arm64, i386, x86_64. riscv64 works
> except two syscalls which return unexpected errors, and mips segfaults
> in sbrk(). I don't know why yet, but this proves that it's worth having
> such a test.

Excellent, thank you!!!

As we often said during my misspent youth, "If it ain't tested, it
don't work."

But I do get "71 test(s) passed." when running on x86.  I will let you
decide whether that constitutes all being well or indicates a bug in
the tests.   ;-)

> There are not that many tests yet (71), those that have to run can be
> filtered either from the program's command line or from a NOLIBC_TEST
> environment variable so that it's possible to skip broken ones or to
> focus on a few ranges only.
> 
> Tests are numerically numbered, and are conveniently handled in a
> switch/case statement so that a relative line number assigns the number
> to the test. That's convenient because the vast majority of syscall tests
> are one-liners. This sometimes slightly upsets check-patch when lines get
> moderately long but that significantly improves legibility.
> 
> There are expectation for both successes and failures (e.g. -1 ENOTDIR).
> I'm sure this can be improved later (and that's the goal). Right now it
> covers two test families:
>   - syscalls
>   - stdlib    (str* functions mostly)
> 
> I suspect that over time we might want to split syscalls into different
> parts (e.g. core, fs, etc maybe) but I could be wrong.

This is a good start, and we can let experience drive any additional
changes that might be required.

> The program can automatically modulate QEMU's return value on x86 when
> QEMU is run with the appropriate options, but for now I'm not using it
> as I felt like it didn't bring much value, and the output is more useful.
> That's debatable, and maybe some might want to use it in bisect scripts
> for example. It's too early to say IMHO.

For the moment, grepping the output works.  And perhaps indefinitely.

> Oh, I also arranged the code so that the test also builds with glibc. I
> noticed that when adding a new test that fails, sometimes it's convenient
> to see if it's the nolibc part that's broken or the test. I don't find
> this critical but the required includes and ifdefs are there so that it
> should be easy to maintain over time as well.

If nothing else, the ability to run against glibc is a good way to test
the test.

> I'm obviously interested in comments, but really, I don't want to
> overdesign something for a first step, it remains a very modest test
> program and I'd like that it remains easy to hack on it and to contribute
> new tests that are deemed useful.

I am good with a simple starting point.

> I'm CCing the few who already contributed some patches and/or expressed
> interest, as well as Linus who had a first bad experience when trying to
> test it, hoping this one will be better. I'm pasting below [1] a copy of
> a test on x86_64 below, that's summed up as "71 test(s) passed" at the
> end of the "run" target.
> 
> If there's no objection, it would be nice to have this with your current
> series, as it definitely helps spot and fix the bugs. In parallel I'll see
> if I can figure the problems with the two tests that fail each on a
> specific arch and I might possibly have a few extra fixes for the current
> nolibc.

This series is now on the -rcu tree's "dev" branch.  I got two almost
identical copies of patch 7, so I took the later of the two.  Please let
me know if I guessed wrong.

							Thanx, Paul

> Thank you!
> Willy
> 
> [1] example output
> ----8<----
> Running test 'syscall'
> 0 getpid = 1                             [OK]
> 1 getppid = 0                            [OK]
> 5 getpgid_self = 0                       [OK]
> 6 getpgid_bad = -1 ESRCH                 [OK]
> 7 kill_0 = 0                             [OK]
> 8 kill_CONT = 0                          [OK]
> 9 kill_BADPID = -1 ESRCH                 [OK]
> 10 sbrk = 0                              [OK]
> 11 brk = 0                               [OK]
> 12 chdir_root = 0                        [OK]
> 13 chdir_dot = 0                         [OK]
> 14 chdir_blah = -1 ENOENT                [OK]
> 15 chmod_net = 0                         [OK]
> 16 chmod_self = -1 EPERM                 [OK]
> 17 chown_self = -1 EPERM                 [OK]
> 18 chroot_root = 0                       [OK]
> 19 chroot_blah = -1 ENOENT               [OK]
> 20 chroot_exe = -1 ENOTDIR               [OK]
> 21 close_m1 = -1 EBADF                   [OK]
> 22 close_dup = 0                         [OK]
> 23 dup_0 = 3                             [OK]
> 24 dup_m1 = -1 EBADF                     [OK]
> 25 dup2_0 = 100                          [OK]
> 26 dup2_m1 = -1 EBADF                    [OK]
> 27 dup3_0 = 100                          [OK]
> 28 dup3_m1 = -1 EBADF                    [OK]
> 29 execve_root = -1 EACCES               [OK]
> 30 getdents64_root = 120                 [OK]
> 31 getdents64_null = -1 ENOTDIR          [OK]
> 32 gettimeofday_null = 0                 [OK]
> 38 ioctl_tiocinq = 0                     [OK]
> 39 ioctl_tiocinq = 0                     [OK]
> 40 link_root1 = -1 EEXIST                [OK]
> 41 link_blah = -1 ENOENT                 [OK]
> 42 link_dir = -1 EPERM                   [OK]
> 43 link_cross = -1 EXDEV                 [OK]
> 44 lseek_m1 = -1 EBADF                   [OK]
> 45 lseek_0 = -1 ESPIPE                   [OK]
> 46 mkdir_root = -1 EEXIST                [OK]
> 47 open_tty = 3                          [OK]
> 48 open_blah = -1 ENOENT                 [OK]
> 49 poll_null = 0                         [OK]
> 50 poll_stdout = 1                       [OK]
> 51 poll_fault = -1 EFAULT                [OK]
> 52 read_badf = -1 EBADF                  [OK]
> 53 sched_yield = 0                       [OK]
> 54 select_null = 0                       [OK]
> 55 select_stdout = 1                     [OK]
> 56 select_fault = -1 EFAULT              [OK]
> 57 stat_blah = -1 ENOENT                 [OK]
> 58 stat_fault = -1 EFAULT                [OK]
> 59 symlink_root = -1 EEXIST              [OK]
> 60 unlink_root = -1 EISDIR               [OK]
> 61 unlink_blah = -1 ENOENT               [OK]
> 62 wait_child = -1 ECHILD                [OK]
> 63 waitpid_min = -1 ESRCH                [OK]
> 64 waitpid_child = -1 ECHILD             [OK]
> 65 write_badf = -1 EBADF                 [OK]
> 66 write_zero = 0                        [OK]
> Errors during this test: 0
> 
> Running test 'stdlib'
> 0 getenv_TERM = <linux>                  [OK]
> 1 getenv_blah = <(null)>                 [OK]
> 2 setcmp_blah_blah = 0                   [OK]
> 3 setcmp_blah_blah2 = -50                [OK]
> 4 setncmp_blah_blah = 0                  [OK]
> 5 setncmp_blah_blah4 = 0                 [OK]
> 6 setncmp_blah_blah5 = -53               [OK]
> 7 setncmp_blah_blah6 = -54               [OK]
> 8 strchr_foobar_o = <oobar>              [OK]
> 9 strchr_foobar_z = <(null)>             [OK]
> 10 strrchr_foobar_o = <obar>             [OK]
> 11 strrchr_foobar_z = <(null)>           [OK]
> Errors during this test: 0
> 
> Total number of errors: 0
> ---->8----
> 
> 
> --
> 
> Willy Tarreau (17):
>   tools/nolibc: make argc 32-bit in riscv startup code
>   tools/nolibc: fix build warning in sys_mmap() when my_syscall6 is not
>     defined
>   tools/nolibc: make sys_mmap() automatically use the right __NR_mmap
>     definition
>   selftests/nolibc: add basic infrastructure to ease creation of nolibc
>     tests
>   selftests/nolibc: support a test definition format
>   selftests/nolibc: implement a few tests for various syscalls
>   selftests/nolibc: add a few tests for some stdlib functions
>   selftests/nolibc: exit with poweroff on success when getpid() == 1
>   selftests/nolibc: on x86, support exiting with isa-debug-exit
>   selftests/nolibc: recreate and populate /dev and /proc if missing
>   selftests/nolibc: condition some tests on /proc existence
>   selftests/nolibc: support glibc as well
>   selftests/nolibc: add a "kernel" target to build the kernel with the
>     initramfs
>   selftests/nolibc: add a "defconfig" target
>   selftests/nolibc: add a "run" target to start the kernel in QEMU
>   selftests/nolibc: "sysroot" target installs a local copy of the
>     sysroot
>   selftests/nolibc: add a "help" target
> 
>  MAINTAINERS                                  |   1 +
>  tools/include/nolibc/arch-riscv.h            |   2 +-
>  tools/include/nolibc/sys.h                   |   4 +-
>  tools/testing/selftests/nolibc/Makefile      | 135 ++++
>  tools/testing/selftests/nolibc/nolibc-test.c | 757 +++++++++++++++++++
>  5 files changed, 896 insertions(+), 3 deletions(-)
>  create mode 100644 tools/testing/selftests/nolibc/Makefile
>  create mode 100644 tools/testing/selftests/nolibc/nolibc-test.c
> 
> -- 
> 2.17.5
>