Re: [PATCH v3] selftests/vm: enable running select groups of tests

[Joel Savitz]

Hello,

Sorry for the late reply.

Setting TEST_ITEMS="xx yy zz" and invoking the script has the same
effect as passing -t "xx yy zz" to the script. VM_TEST_ITEMS may be a
better env variable name however.

Best,
Joel Savitz

On Fri, Jul 15, 2022 at 3:55 PM Nico Pache <npache@xxxxxxxxxx> wrote:
>
> Hi Andrew and Joel,
>
> I noticed a flaw in switching to the cmdline argument over the
> environment variable. If you are utilizing the exerciser script (in
> tools/testing/selftests/run_kselftest.sh), this is not designed to
> pass cmdline variables to the individual TEST_PROGS. On the other hand
> if we utilize Env. variables we can still specify which tests we'd
> like to run. Our infrastructure is designed to utilize this wrapper,
> so we inevitably will not be able to use this the way we thought.
>
> Would you be ok with switching back to the initial approach? If so I
> believe making the name more specific would be ideal, ie)
> VM_TEST_ITEMS.
>
> Cheers,
> -- Nico
>
> On Wed, Jul 6, 2022 at 4:55 PM Joel Savitz <jsavitz@xxxxxxxxxx> wrote:
> >
> > Our memory management kernel CI testing at Red Hat uses the VM
> > selftests and we have run into two problems:
> >
> > First, our LTP tests overlap with the VM selftests.
> >
> > We want to avoid unhelpful redundancy in our testing practices.
> >
> > Second, we have observed the current run_vmtests.sh to report overall
> > failure/ambiguous results in the case that a machine lacks the necessary
> > hardware to perform one or more of the tests. E.g. ksm tests that
> > require more than one numa node.
> >
> > We want to be able to run the vm selftests suitable to particular hardware.
> >
> > Add the ability to run one or more groups of vm tests via run_vmtests.sh
> > instead of simply all-or-none in order to solve these problems.
> >
> > Preserve existing default behavior of running all tests when the script
> > is invoked with no arguments.
> >
> > Documentation of test groups is included in the patch as follows:
> >
> >     # ./run_vmtests.sh [ -h || --help ]
> >
> >     usage: ./tools/testing/selftests/vm/run_vmtests.sh [ -h | -t "<categories>"]
> >       -t: specify specific categories to tests to run
> >       -h: display this message
> >
> >     The default behavior is to run all tests.
> >
> >     Alternatively, specific groups tests can be run by passing a string
> >     to the -t argument containing one or more of the following categories
> >     separated by spaces:
> >     - mmap
> >             tests for mmap(2)
> >     - gup_test
> >             tests for gup using gup_test interface
> >     - userfaultfd
> >             tests for  userfaultfd(2)
> >     - compaction
> >             a test for the patch "Allow compaction of unevictable pages"
> >     - mlock
> >             tests for mlock(2)
> >     - mremap
> >             tests for mremap(2)
> >     - hugevm
> >             tests for very large virtual address space
> >     - vmalloc
> >             vmalloc smoke tests
> >     - hmm
> >             hmm smoke tests
> >     - madv_populate
> >             test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> >     - memfd_secret
> >             test memfd_secret(2)
> >     - process_mrelease
> >             test process_mrelease(2)
> >     - ksm
> >             ksm tests that do not require >=2 NUMA nodes
> >     - ksm_numa
> >             ksm tests that require >=2 NUMA nodes
> >     - pkey
> >             memory protection key tests
> >     example: ./run_vmtests.sh -t "hmm mmap ksm"
> >
> > Changes from v2:
> >         - rebase onto the mm-everyting branch in
> >         https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git
> >         - integrate this functionality with new the tests
> >
> > Changes from v1:
> >         - use a command line argument to pass the test categories to the
> >           script instead of an environmet variable
> >         - remove novel prints to avoid messing with extant parsers of this
> >           script
> >         - update the usage text
> >
> > Signed-off-by: Joel Savitz <jsavitz@xxxxxxxxxx>
> > ---
> >  tools/testing/selftests/vm/run_vmtests.sh | 241 +++++++++++++++-------
> >  1 file changed, 161 insertions(+), 80 deletions(-)
> >
> > diff --git a/tools/testing/selftests/vm/run_vmtests.sh b/tools/testing/selftests/vm/run_vmtests.sh
> > index d84fe0fa15e1..fb72a1338d07 100755
> > --- a/tools/testing/selftests/vm/run_vmtests.sh
> > +++ b/tools/testing/selftests/vm/run_vmtests.sh
> > @@ -1,6 +1,6 @@
> >  #!/bin/bash
> >  # SPDX-License-Identifier: GPL-2.0
> > -#please run as root
> > +# Please run as root
> >
> >  # Kselftest framework requirement - SKIP code is 4.
> >  ksft_skip=4
> > @@ -8,15 +8,75 @@ ksft_skip=4
> >  mnt=./huge
> >  exitcode=0
> >
> > -#get huge pagesize and freepages from /proc/meminfo
> > -while read -r name size unit; do
> > -       if [ "$name" = "HugePages_Free:" ]; then
> > -               freepgs="$size"
> > -       fi
> > -       if [ "$name" = "Hugepagesize:" ]; then
> > -               hpgsize_KB="$size"
> > +usage() {
> > +       cat <<EOF
> > +usage: ${BASH_SOURCE[0]:-$0} [ -h | -t "<categories>"]
> > +  -t: specify specific categories to tests to run
> > +  -h: display this message
> > +
> > +The default behavior is to run all tests.
> > +
> > +Alternatively, specific groups tests can be run by passing a string
> > +to the -t argument containing one or more of the following categories
> > +separated by spaces:
> > +- mmap
> > +       tests for mmap(2)
> > +- gup_test
> > +       tests for gup using gup_test interface
> > +- userfaultfd
> > +       tests for  userfaultfd(2)
> > +- compaction
> > +       a test for the patch "Allow compaction of unevictable pages"
> > +- mlock
> > +       tests for mlock(2)
> > +- mremap
> > +       tests for mremap(2)
> > +- hugevm
> > +       tests for very large virtual address space
> > +- vmalloc
> > +       vmalloc smoke tests
> > +- hmm
> > +       hmm smoke tests
> > +- madv_populate
> > +       test memadvise(2) MADV_POPULATE_{READ,WRITE} options
> > +- memfd_secret
> > +       test memfd_secret(2)
> > +- process_mrelease
> > +       test process_mrelease(2)
> > +- ksm
> > +       ksm tests that do not require >=2 NUMA nodes
> > +- ksm_numa
> > +       ksm tests that require >=2 NUMA nodes
> > +- pkey
> > +       memory protection key tests
> > +example: ./run_vmtests.sh -t "hmm mmap ksm"
> > +EOF
> > +       exit 0
> > +}
> > +
> > +
> > +while getopts "ht:" OPT; do
> > +       case ${OPT} in
> > +               "h") usage ;;
> > +               "t") TEST_ITEMS=${OPTARG} ;;
> > +       esac
> > +done
> > +shift $((OPTIND -1))
> > +
> > +# default behavior: run all tests
> > +TEST_ITEMS=${TEST_ITEMS:-default}
> > +
> > +test_selected() {
> > +       if [ "$TEST_ITEMS" == "default" ]; then
> > +               # If no TEST_ITEMS are specified, run all tests
> > +               return 0
> >         fi
> > -done < /proc/meminfo
> > +       echo ${TEST_ITEMS} | grep ${1} 2>&1 >/dev/null
> > +       return ${?}
> > +}
> > +
> > +# Hugepage setup only needed for hugetlb tests
> > +if test_selected "hugetlb"; then
> >
> >  # Simple hugetlbfs tests have a hardcoded minimum requirement of
> >  # huge pages totaling 256MB (262144KB) in size.  The userfaultfd
> > @@ -28,7 +88,17 @@ hpgsize_MB=$((hpgsize_KB / 1024))
> >  half_ufd_size_MB=$((((nr_cpus * hpgsize_MB + 127) / 128) * 128))
> >  needmem_KB=$((half_ufd_size_MB * 2 * 1024))
> >
> > -#set proper nr_hugepages
> > +# get huge pagesize and freepages from /proc/meminfo
> > +while read -r name size unit; do
> > +       if [ "$name" = "HugePages_Free:" ]; then
> > +               freepgs="$size"
> > +       fi
> > +       if [ "$name" = "Hugepagesize:" ]; then
> > +               hpgsize_KB="$size"
> > +       fi
> > +done < /proc/meminfo
> > +
> > +# set proper nr_hugepages
> >  if [ -n "$freepgs" ] && [ -n "$hpgsize_KB" ]; then
> >         nr_hugepgs=$(cat /proc/sys/vm/nr_hugepages)
> >         needpgs=$((needmem_KB / hpgsize_KB))
> > @@ -57,144 +127,155 @@ else
> >         exit 1
> >  fi
> >
> > -#filter 64bit architectures
> > +fi # test_selected "hugetlb"
> > +
> > +# filter 64bit architectures
> >  ARCH64STR="arm64 ia64 mips64 parisc64 ppc64 ppc64le riscv64 s390x sh64 sparc64 x86_64"
> >  if [ -z "$ARCH" ]; then
> >         ARCH=$(uname -m 2>/dev/null | sed -e 's/aarch64.*/arm64/')
> >  fi
> >  VADDR64=0
> > -echo "$ARCH64STR" | grep "$ARCH" && VADDR64=1
> > +echo "$ARCH64STR" | grep "$ARCH" &>/dev/null && VADDR64=1
> >
> >  # Usage: run_test [test binary] [arbitrary test arguments...]
> >  run_test() {
> > -       local title="running $*"
> > -       local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > -       printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > -
> > -       "$@"
> > -       local ret=$?
> > -       if [ $ret -eq 0 ]; then
> > -               echo "[PASS]"
> > -       elif [ $ret -eq $ksft_skip ]; then
> > -               echo "[SKIP]"
> > -               exitcode=$ksft_skip
> > -       else
> > -               echo "[FAIL]"
> > -               exitcode=1
> > -       fi
> > +       if test_selected ${CATEGORY}; then
> > +               local title="running $*"
> > +               local sep=$(echo -n "$title" | tr "[:graph:][:space:]" -)
> > +               printf "%s\n%s\n%s\n" "$sep" "$title" "$sep"
> > +
> > +               "$@"
> > +               local ret=$?
> > +               if [ $ret -eq 0 ]; then
> > +                       echo "[PASS]"
> > +               elif [ $ret -eq $ksft_skip ]; then
> > +                       echo "[SKIP]"
> > +                       exitcode=$ksft_skip
> > +               else
> > +                       echo "[FAIL]"
> > +                       exitcode=1
> > +               fi
> > +       fi # test_selected
> >  }
> >
> > -mkdir "$mnt"
> > -mount -t hugetlbfs none "$mnt"
> > +# setup only needed for hugetlb tests
> > +if test_selected "hugetlb"; then
> > +       mkdir "$mnt"
> > +       mount -t hugetlbfs none "$mnt"
> > +fi
> >
> > -run_test ./hugepage-mmap
> > +CATEGORY="hugetlb" run_test ./hugepage-mmap
> >
> >  shmmax=$(cat /proc/sys/kernel/shmmax)
> >  shmall=$(cat /proc/sys/kernel/shmall)
> >  echo 268435456 > /proc/sys/kernel/shmmax
> >  echo 4194304 > /proc/sys/kernel/shmall
> > -run_test ./hugepage-shm
> > +CATEGORY="hugetlb" run_test ./hugepage-shm
> >  echo "$shmmax" > /proc/sys/kernel/shmmax
> >  echo "$shmall" > /proc/sys/kernel/shmall
> >
> > -run_test ./map_hugetlb
> > +CATEGORY="hugetlb" run_test ./map_hugetlb
> >
> > -run_test ./hugepage-mremap "$mnt"/huge_mremap
> > -rm -f "$mnt"/huge_mremap
> > +CATEGORY="hugetlb" run_test ./hugepage-mremap "$mnt"/huge_mremap
> > +test_selected "hugetlb" && rm -f "$mnt"/huge_mremap
> >
> > -run_test ./hugepage-vmemmap
> > +CATEGORY="hugetlb" run_test ./hugepage-vmemmap
> >
> > -run_test ./hugetlb-madvise "$mnt"/madvise-test
> > -rm -f "$mnt"/madvise-test
> > +CATEGORY="hugetlb" run_test ./hugetlb-madvise "$mnt"/madvise-test
> > +test_selected "hugetlb" && rm -f "$mnt"/madvise-test
> >
> > -echo "NOTE: The above hugetlb tests provide minimal coverage.  Use"
> > -echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > -echo "      hugetlb regression testing."
> > +if test_selected "hugetlb"; then
> > +       echo "NOTE: These hugetlb tests provide minimal coverage.  Use"
> > +       echo "      https://github.com/libhugetlbfs/libhugetlbfs.git for"
> > +       echo "      hugetlb regression testing."
> > +fi
> >
> > -run_test ./map_fixed_noreplace
> > +CATEGORY="mmap" run_test ./map_fixed_noreplace
> >
> >  # get_user_pages_fast() benchmark
> > -run_test ./gup_test -u
> > +CATEGORY="gup_test" run_test ./gup_test -u
> >  # pin_user_pages_fast() benchmark
> > -run_test ./gup_test -a
> > +CATEGORY="gup_test" run_test ./gup_test -a
> >  # Dump pages 0, 19, and 4096, using pin_user_pages:
> > -run_test ./gup_test -ct -F 0x1 0 19 0x1000
> > +CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000
> >
> > -run_test ./userfaultfd anon 20 16
> > -run_test ./userfaultfd anon:dev 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd anon 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd anon:dev 20 16
> >  # Hugetlb tests require source and destination huge pages. Pass in half the
> >  # size ($half_ufd_size_MB), which is used for *each*.
> > -run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > -run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > -run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb "$half_ufd_size_MB" 32
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb:dev "$half_ufd_size_MB" 32
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> >  rm -f "$mnt"/uffd-test
> > -run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> > +CATEGORY="userfaultfd" run_test ./userfaultfd hugetlb_shared:dev "$half_ufd_size_MB" 32 "$mnt"/uffd-test
> >  rm -f "$mnt"/uffd-test
> > -run_test ./userfaultfd shmem 20 16
> > -run_test ./userfaultfd shmem:dev 20 16
> > -
> > -#cleanup
> > -umount "$mnt"
> > -rm -rf "$mnt"
> > -echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem 20 16
> > +CATEGORY="userfaultfd" run_test ./userfaultfd shmem:dev 20 16
> > +
> > +# cleanup (only needed when running hugetlb tests)
> > +if test_selected "hugetlb"; then
> > +       umount "$mnt"
> > +       rm -rf "$mnt"
> > +       echo "$nr_hugepgs" > /proc/sys/vm/nr_hugepages
> > +fi
> >
> > -run_test ./compaction_test
> > +CATEGORY="compaction" run_test ./compaction_test
> >
> > -run_test sudo -u nobody ./on-fault-limit
> > +CATEGORY="mlock" run_test sudo -u nobody ./on-fault-limit
> >
> > -run_test ./map_populate
> > +CATEGORY="mmap" run_test ./map_populate
> >
> > -run_test ./mlock-random-test
> > +CATEGORY="mlock" run_test ./mlock-random-test
> >
> > -run_test ./mlock2-tests
> > +CATEGORY="mlock" run_test ./mlock2-tests
> >
> > -run_test ./mrelease_test
> > +CATEGORY="process_mrelease" run_test ./mrelease_test
> >
> > -run_test ./mremap_test
> > +CATEGORY="mremap" run_test ./mremap_test
> >
> > -run_test ./thuge-gen
> > +CATEGORY="hugetlb" run_test ./thuge-gen
> >
> >  if [ $VADDR64 -ne 0 ]; then
> > -       run_test ./virtual_address_range
> > +       CATEGORY="hugevm" run_test ./virtual_address_range
> >
> >         # virtual address 128TB switch test
> > -       run_test ./va_128TBswitch.sh
> > +       CATEGORY="hugevm" run_test ./va_128TBswitch.sh
> >  fi # VADDR64
> >
> >  # vmalloc stability smoke test
> > -run_test ./test_vmalloc.sh smoke
> > +CATEGORY="vmalloc" run_test ./test_vmalloc.sh smoke
> >
> > -run_test ./mremap_dontunmap
> > +CATEGORY="mremap" run_test ./mremap_dontunmap
> >
> > -run_test ./test_hmm.sh smoke
> > +CATEGORY="hmm" run_test ./test_hmm.sh smoke
> >
> >  # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests
> > -run_test ./madv_populate
> > +CATEGORY="madv_populate" run_test ./madv_populate
> >
> > -run_test ./memfd_secret
> > +CATEGORY="memfd_secret" run_test ./memfd_secret
> >
> >  # KSM MADV_MERGEABLE test with 10 identical pages
> > -run_test ./ksm_tests -M -p 10
> > +CATEGORY="ksm" run_test ./ksm_tests -M -p 10
> >  # KSM unmerge test
> > -run_test ./ksm_tests -U
> > +CATEGORY="ksm" run_test ./ksm_tests -U
> >  # KSM test with 10 zero pages and use_zero_pages = 0
> > -run_test ./ksm_tests -Z -p 10 -z 0
> > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 0
> >  # KSM test with 10 zero pages and use_zero_pages = 1
> > -run_test ./ksm_tests -Z -p 10 -z 1
> > +CATEGORY="ksm" run_test ./ksm_tests -Z -p 10 -z 1
> >  # KSM test with 2 NUMA nodes and merge_across_nodes = 1
> > -run_test ./ksm_tests -N -m 1
> > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 1
> >  # KSM test with 2 NUMA nodes and merge_across_nodes = 0
> > -run_test ./ksm_tests -N -m 0
> > +CATEGORY="ksm_numa" run_test ./ksm_tests -N -m 0
> >
> >  # protection_keys tests
> >  if [ -x ./protection_keys_32 ]
> >  then
> > -       run_test ./protection_keys_32
> > +       CATEGORY="pkey" run_test ./protection_keys_32
> >  fi
> >
> >  if [ -x ./protection_keys_64 ]
> >  then
> > -       run_test ./protection_keys_64
> > +       CATEGORY="pkey" run_test ./protection_keys_64
> >  fi
> >
> >  exit $exitcode
> > --
> > 2.31.1
> >
>