[PATCH v2 0/2] tools/nolibc: check for overflow in malloc()

[Danish Khateeb]

malloc() does not check the addition of its header size for overflow,
so malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX)
return a single page instead of NULL.

As suggested by Thomas, patch 1 drops the rounding to 4096 bytes, and
patch 2 adds the overflow check on top.

Tested on x86_64 (GCC and clang), i386, and on arm, arm64 and sparc64
under qemu-user:

- nolibc-test: no failures, with patch 1 alone and with the series.

- A separate program: malloc(), calloc() and realloc() with sizes close
  to SIZE_MAX fail with ENOMEM, and free() still unmaps every page.

Changes in v2:
- Drop the rounding in a separate patch, instead of checking it for
  overflow (Thomas)
- Drop the test patch (Thomas)
- Link to v1: https://lore.kernel.org/r/20260926134332.58184-1-danishkhateeb03@xxxxxxxxx

Danish Khateeb (2):
  tools/nolibc: stop rounding malloc() sizes up to 4096 bytes
  tools/nolibc: check for overflow in malloc()

 tools/include/nolibc/stdlib.h | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)


base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5
-- 
2.55.0