[PATCH v2 0/2] binfmt_elf: return the real error when program headers cannot be loaded

[Konstantin Khorenko]

load_elf_phdrs() returns NULL on any failure, and load_elf_binary()
turns that into -ENOEXEC for the executable and -ELIBBAD for its
interpreter. A failed allocation or read of the program header table
then looks like a bad file, and glibc's execvp() runs the ELF with
/bin/sh.

v2 changes the errno of those two failures and nothing else. Patch 2
adds a selftest for both what stays and what changes.

What changes
============

  +--------------------------------------+----------+------------+
  | the program header table             | before   | after      |
  +--------------------------------------+----------+------------+
  | has a bad e_phentsize or e_phnum     | ENOEXEC  | ENOEXEC    |
  | has an e_phoff past LLONG_MAX - size | ENOEXEC  | ENOEXEC    |
  | starts past the end of the file      | ENOEXEC  | ENOEXEC    |
  | gives a short read                   | ENOEXEC  | ENOEXEC    |
  +--------------------------------------+----------+------------+
  | cannot be allocated                  | ENOEXEC  | ENOMEM     |
  | cannot be read: kernel_read() error  | ENOEXEC  | that error |
  +--------------------------------------+----------+------------+

For the interpreter's table, ELIBBAD stands in place of every ENOEXEC
above; ENOMEM and the read error come back as they are. A read that
fails after some bytes were copied comes back from the page cache as a
short count, so it still gives ENOEXEC.

In the kernel, search_binary_handler() now stops at the ELF handler for
these two errors instead of trying the formats after it. All of those
decline a file with the ELF magic of this machine, so the loop ended
with -ENOEXEC before anyway.

Userspace
=========

The patch adds no errno that execve() cannot return today. Other reads
of the same files already pass the kernel_read() error through:
prepare_binprm() for the first BINPRM_BUF_SIZE bytes of the binary,
and elf_read() for the PT_INTERP path and the interpreter's ELF
header, while parse_elf_properties() reports a failed read as -EIO.
The program header table was the only read of an ELF exec whose error
reached userspace as ENOEXEC.

The execvp() family acts on ENOEXEC: glibc runs the file with /bin/sh,
that exec succeeds, and the shell replaces the caller and exits with an
error such as bash's "cannot execute binary file". glibc's
posix_spawnp() does so only through its compat symbol from before 2.15,
and musl's execvp() never does.

Shells check ENOEXEC themselves when they exec a command: bash reports
"cannot execute binary file", dash runs the file as a script. With the
patch they report the ENOMEM or the read error instead.

QEMU's cpr-exec live update re-execs itself with execvp() and resumes
the guest when that fails; on ENOEXEC the guest is killed instead. With
the patch these callers get the ENOMEM or the read error back.

Testing
=======

x86_64 KVM guest, mainline f66469d9905b with CONFIG_FAILSLAB and
CONFIG_FAULT_INJECTION_DEBUG_FS, selftests/exec:

  +------------------------------+---------------------+---------------+
  |                              | without patch 1     | with patch 1  |
  +------------------------------+---------------------+---------------+
  | load_elf_phdrs, bad tables   | pass                | pass          |
  +------------------------------+---------------------+---------------+
  | load_elf_phdrs, allocation   | fails: ENOEXEC for  | passes        |
  | failures                     | the executable,     |               |
  |                              | ELIBBAD for ld.so   |               |
  +------------------------------+---------------------+---------------+
  | the other exec selftests     | pass                | pass          |
  +------------------------------+---------------------+---------------+

Unprivileged, the allocation case is skipped. A kernel_read() error is
not covered by the selftest, as it needs a filesystem that fails the
read.

Assisted-by: LLM
Signed-off-by: Konstantin Khorenko <khorenko@xxxxxxxxxxxxx>
---
Changes in v2:
- Dropped the read loop: a short read gives -ENOEXEC as before, and
  only the errno changes (Kees).
- Patch 2: a selftest for the errors of the table.
- Dropped the Fixes: tags on 6a8d38945cf4 and a9d9ef133f44 (v3.19):
  the errno has been this way since then, and the change should not go
  to stable.
- Rebased on current mainline, f66469d9905b (after v7.4-rc1).
- The Assisted-by line moved from the patch to this letter.
- Link to v1: https://patch.msgid.link/20261014-b4-prep-vstor-149847-binfmt_elf-enomem-load-elf-phdrs-v1-1-f39c74d55e42@xxxxxxxxxxxxx

---
Konstantin Khorenko (2):
      binfmt_elf: return the real error when program headers cannot be loaded
      selftests/exec: check the errors of an unloadable program header table

 fs/binfmt_elf.c                               |  38 ++-
 tools/testing/selftests/exec/.gitignore       |   2 +
 tools/testing/selftests/exec/Makefile         |   4 +
 tools/testing/selftests/exec/load_elf_phdrs.c | 399 ++++++++++++++++++++++++++
 4 files changed, 428 insertions(+), 15 deletions(-)
---
base-commit: f66469d9905b39d2c260a15d7d6db450f4db67c1
change-id: 20261014-b4-prep-vstor-149847-binfmt_elf-enomem-load-elf-phdrs-395ffbfc33e2

Best regards,
--  
Konstantin Khorenko,
Virtuozzo Linux Kernel Team