[PATCH 0/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() does not check its size arithmetic for overflow, so
malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX) return
a single page instead of NULL, and slightly smaller sizes fail with
EINVAL instead of ENOMEM.

Patch 1 fixes it. Patch 2 adds nolibc-test cases for huge sizes, which
would have caught it.

Tested on top of nolibc/for-next 0e1c44b472e1, on x86_64 (GCC and
clang) and i386, and on arm, arm64 and sparc64 under qemu-user:

- nolibc-test, all tests: no failures with the series. The only
  difference from before is the three new tests, which pass. They also
  pass against glibc (make libc-test), which builds without warnings.

- Without patch 1, all three new tests fail on every build:
  malloc(SIZE_MAX) and calloc(SIZE_MAX, 1) return a pointer, and
  malloc(SIZE_MAX - 4096) fails with EINVAL.

- On x86_64, a separate program shows realloc(ptr, SIZE_MAX) returning
  a pointer before patch 1 and NULL with ENOMEM after it.

Danish Khateeb (2):
  tools/nolibc: check for overflow in malloc()
  selftests/nolibc: test malloc() and calloc() with huge sizes

 tools/include/nolibc/stdlib.h                | 12 +++++++++---
 tools/testing/selftests/nolibc/nolibc-test.c | 11 +++++++++++
 2 files changed, 20 insertions(+), 3 deletions(-)


base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5
-- 
2.55.0