Re: [PATCH bpf-next v5 5/5] error-injection: Support fault injection framework

[Akinobu Mita]

2018-01-13 2:56 GMT+09:00 Masami Hiramatsu <mhiramat@xxxxxxxxxx>:
> Support in-kernel fault-injection framework via debugfs.
> This allows you to inject a conditional error to specified
> function using debugfs interfaces.
>
> Here is the result of test script described in
> Documentation/fault-injection/fault-injection.txt
>
>   ===========
>   # ./test_fail_function.sh
>   1+0 records in
>   1+0 records out
>   1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227404 s, 46.1 MB/s
>   btrfs-progs v4.4
>   See http://btrfs.wiki.kernel.org for more information.
>
>   Label:              (null)
>   UUID:               bfa96010-12e9-4360-aed0-42eec7af5798
>   Node size:          16384
>   Sector size:        4096
>   Filesystem size:    1001.00MiB
>   Block group profiles:
>     Data:             single            8.00MiB
>     Metadata:         DUP              58.00MiB
>     System:           DUP              12.00MiB
>   SSD detected:       no
>   Incompat features:  extref, skinny-metadata
>   Number of devices:  1
>   Devices:
>      ID        SIZE  PATH
>       1  1001.00MiB  /dev/loop2
>
>   mount: mount /dev/loop2 on /opt/tmpmnt failed: Cannot allocate memory
>   SUCCESS!
>   ===========
>
>
> Signed-off-by: Masami Hiramatsu <mhiramat@xxxxxxxxxx>
> Reviewed-by: Josef Bacik <jbacik@xxxxxx>
> ---
>   Changes in v3:
>    - Check and adjust error value for each target function
>    - Clear kporbe flag for reuse
>    - Add more documents and example
>   Changes in v5:
>    - Support multi-function error injection
> ---
>  Documentation/fault-injection/fault-injection.txt |   68 ++++
>  kernel/Makefile                                   |    1
>  kernel/fail_function.c                            |  349 +++++++++++++++++++++
>  lib/Kconfig.debug                                 |   10 +
>  4 files changed, 428 insertions(+)
>  create mode 100644 kernel/fail_function.c
>
> diff --git a/Documentation/fault-injection/fault-injection.txt b/Documentation/fault-injection/fault-injection.txt
> index 918972babcd8..f4a32463ca48 100644
> --- a/Documentation/fault-injection/fault-injection.txt
> +++ b/Documentation/fault-injection/fault-injection.txt
> @@ -30,6 +30,12 @@ o fail_mmc_request
>    injects MMC data errors on devices permitted by setting
>    debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
>
> +o fail_function
> +
> +  injects error return on specific functions, which are marked by
> +  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
> +  under /sys/kernel/debug/fail_function. No boot option supported.
> +
>  Configure fault-injection capabilities behavior
>  -----------------------------------------------
>
> @@ -123,6 +129,29 @@ configuration of fault-injection capabilities.
>         default is 'N', setting it to 'Y' will disable failure injections
>         when dealing with private (address space) futexes.
>
> +- /sys/kernel/debug/fail_function/inject:
> +
> +       Format: { 'function-name' | '!function-name' | '' }
> +       specifies the target function of error injection by name.
> +       If the function name leads '!' prefix, given function is
> +       removed from injection list. If nothing specified ('')
> +       injection list is cleared.
> +
> +- /sys/kernel/debug/fail_function/injectable:
> +
> +       (read only) shows error injectable functions and what type of
> +       error values can be specified. The error type will be one of
> +       below;
> +       - NULL: retval must be 0.
> +       - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
> +       - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
> +
> +- /sys/kernel/debug/fail_function/<functiuon-name>/retval:
> +
> +       specifies the "error" return value to inject to the given
> +       function for given function. This will be created when
> +       user specifies new injection entry.
> +
>  o Boot option
>
>  In order to inject faults while debugfs is not available (early boot time),
> @@ -268,6 +297,45 @@ trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
>  echo "Injecting errors into the module $module... (interrupt to stop)"
>  sleep 1000000
>
> +------------------------------------------------------------------------------
> +
> +o Inject open_ctree error while btrfs mount
> +
> +#!/bin/bash
> +
> +rm -f testfile.img
> +dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
> +DEVICE=$(losetup --show -f testfile.img)
> +mkfs.btrfs -f $DEVICE
> +mkdir -p tmpmnt
> +
> +FAILTYPE=fail_function
> +FAILFUNC=open_ctree
> +echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
> +echo -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
> +echo N > /sys/kernel/debug/$FAILTYPE/task-filter
> +echo 100 > /sys/kernel/debug/$FAILTYPE/probability
> +echo 0 > /sys/kernel/debug/$FAILTYPE/interval
> +echo -1 > /sys/kernel/debug/$FAILTYPE/times
> +echo 0 > /sys/kernel/debug/$FAILTYPE/space
> +echo 1 > /sys/kernel/debug/$FAILTYPE/verbose

I expected that the fault_attr is created for each target function.
(i.e. /sys/kernel/debug/fail_function/<func>/ directory contains not
only 'retval', but also 'probability', 'interval', ...)

Or is there any good reason to share a single fault_attr for all
error injected functions?