Re: [PATCH v3 next 04/17] selftests/nolibc: Improve reporting of vfprintf() errors

[Thomas Weißschuh]

On 2026-02-23 10:17:22+0000, david.laight.linux@xxxxxxxxx wrote:
> From: David Laight <david.laight.linux@xxxxxxxxx>
> 
> Check the string matches before checking the returned length.
> Only print the string once when it matches.
> Allow for skipping tests that check non-standard behaviour.
> 
> Since the 'return value' from snprintf() is really the output buffer
> there is no reason for the test cases to specify the numeric return value
> (the length of the extected output) as well as the expected output.
> The expected output needs to be the 'untruncated' output for tests that
> generate output that gets truncatedb because expect_vfprintf() uses
> a short buffer.
> This puts all the knowledge of any truncation to expect_vfprintf().
> It also easier to change the maximum size, and saves counting
> the length of the expected string by hand when adding tests.
> 
> Increase the size limit from 20 to 25 characters, changing the tests to
> match. This is needed to test octal conversions later on.

Then please do this right before the addition of the octal conversion.

> Append a '+' to the printed output (after the final ") when the output
> is truncated.
> 
> Additionally check that nothing beyond the end is written.
> The "width_trunc" test (#14) now fails because e90ce42e81381
> ("tools/nolibc: implement width padding in printf()") doesn't
> correctly update the space in the buffer when adding pad characters.
> This will be addressed in a later patch.

The build bots will yell at us for this.
Instead mark the test as skipped until it is fixed.

> Also correctly return 1 (the number of errors) when strcmp()
> fails rather than the return value from strncmp() which is the
> signed difference between the mismatching characters.

Please split these different steps into dedicated commits.
Yes, the testcases are going to be rewritten a bunch of times,
but that does not matter.

> Signed-off-by: David Laight <david.laight.linux@xxxxxxxxx>
> ---
> 
> For v3:
> - Patch 9 in v2, patch 5 in v1..
> - Increase the size limit to 25 in preparation for testing octal.
> - Change the way truncated fprintf() are handled.
> - Allow for tests being skipped.
> - Use a fixed value (0xa5) for the canary when detecting overwrites.
> 
>  tools/testing/selftests/nolibc/nolibc-test.c | 91 ++++++++++++++------
>  1 file changed, 64 insertions(+), 27 deletions(-)
> 
> diff --git a/tools/testing/selftests/nolibc/nolibc-test.c b/tools/testing/selftests/nolibc/nolibc-test.c
> index 0e8b3b9a86ef..029ed63e1ae4 100644
> --- a/tools/testing/selftests/nolibc/nolibc-test.c
> +++ b/tools/testing/selftests/nolibc/nolibc-test.c
> @@ -1660,33 +1660,70 @@ int run_stdlib(int min, int max)
>  	return ret;
>  }
>  
> -#define EXPECT_VFPRINTF(c, expected, fmt, ...)				\
> -	ret += expect_vfprintf(llen, c, expected, fmt, ##__VA_ARGS__)
> +#define EXPECT_VFPRINTF(cond, expected, fmt, ...)				\
> +	ret += expect_vfprintf(llen, cond, expected, fmt, ##__VA_ARGS__)
>  
> -static int expect_vfprintf(int llen, int c, const char *expected, const char *fmt, ...)
> +#define VFPRINTF_LEN 25

This is only used within expect_vfprintf(), so it can be a local variable.

> +
> +static int expect_vfprintf(int llen, int cond, const char *expected, const char *fmt, ...)
>  {
> -	char buf[100];
> +	ssize_t written, expected_len;
> +	char buf[VFPRINTF_LEN + 80];
> +	unsigned int cmp_len;
>  	va_list args;
> -	ssize_t w;
> -	int ret;
>  
> +	if (!cond) {
> +		result(llen, SKIPPED);
> +		return 0;
> +	}

The other EXPECT_*() macros evaluate the condition in the macro, not the
corresponding function. I'm not entirely sure why that is, but please
keep it consistent.

> +
> +	/* Fill and terminate buffer to check for overlong/absent writes */
> +	memset(buf, 0xa5, sizeof(buf) - 1);
> +	buf[sizeof(buf) - 1] = 0;
>  
>  	va_start(args, fmt);
> -	/* Only allow writing 21 bytes, to test truncation */
> -	w = vsnprintf(buf, 21, fmt, args);
> +	/* Limit the buffer length to test truncation */
> +	written = vsnprintf(buf, VFPRINTF_LEN + 1, fmt, args);
>  	va_end(args);
>  
> -	if (w != c) {
> -		llen += printf(" written(%d) != %d", (int)w, c);
> +	llen += printf(" \"%s\"", buf);
> +
> +	/* If the expected output is long it will have been truncated. */
> +	expected_len = strlen(expected);
> +	if (expected_len > VFPRINTF_LEN) {
> +		/* Indicate truncated in test output */
> +		llen += printf("+");
> +		cmp_len = VFPRINTF_LEN;
> +	} else {
> +		cmp_len = expected_len;
> +	}
> +
> +	if (memcmp(expected, buf, cmp_len) || buf[cmp_len]) {
> +		/* Copy and truncate until "%.*s" supported */
> +		memcpy(buf, expected, cmp_len);
> +		buf[cmp_len] = 0;
> +		llen += printf(" should be \"%s\"", buf);
>  		result(llen, FAIL);
>  		return 1;
>  	}
>  
> -	llen += printf(" \"%s\" = \"%s\"", expected, buf);
> -	ret = strncmp(expected, buf, c);
> +	if (written != expected_len) {
> +		llen += printf(" written(%d) != %d", (int)written, (int)expected_len);
> +		result(llen, FAIL);
> +		return 1;
> +	}

(...)