Re: [PATCH] x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

[Ting-Han Hou]

On Tue, Sep 29, 2026 at 12:33:45PM -0500, Danish Khateeb wrote:
> x86/entry: Fail invalid syscall numbers other than -1 with -ENOSYS again

I tested this on top of current mainline a5ebb76233b7
(v7.3-rc6-386-ga5ebb76233b7): x86_64 defconfig + kvm_guest.config,
booted in QEMU/KVM (-cpu host), with the x86 selftests built from the
same tree.

selftests/x86/syscall_numbering_64:

  without the patch:
    [FAIL]  A total of 37800 system calls had incorrect behavior
  (12600 each in the "modifying the default return", "clobbering the
  top 32 bits" and "sign-extending the syscall number" ptrace passes,
  e.g. "x32 syscall 0:-64 returned -9999, but it should have returned
  -ENOSYS")

  with the patch:
    [OK]    All system calls succeeded or failed as expected

It also passes with CONFIG_X86_X32_ABI=y ("x32 is supported").

The selftest does not cover the syscall_32.c hunk, so I also used a
small tracer that sets the return register to -9999 at the
syscall-entry stop of int $0x80, from both a 32-bit and a 64-bit
process. Without the patch, numbers 1000, 0x7fffffff, -2 and -1000 all
return -9999. With the patch they return -ENOSYS. -1 returns -9999 in
both cases, as expected.

ptrace_syscall_{32,64}, syscall_nt_{32,64}, syscall_arg_fault_{32,64},
single_step_syscall_{32,64}, check_initial_reg_state_{32,64},
sysret_rip_64, sigreturn_64 and test_syscall_vdso_32 pass with and
without the patch. No warnings in dmesg on the patched kernels. I did
not test a native 32-bit kernel.

The kernel test robot report for this is
https://lore.kernel.org/all/202609210251.4f7bf339-lkp@xxxxxxxxx/ ,
which asked for:

Reported-by: kernel test robot <yi1.lai@xxxxxxxxx>
Closes: https://lore.kernel.org/oe-lkp/202609210251.4f7bf339-lkp@xxxxxxxxx

I used an AI assistant (Claude Code) to help build and run these tests
and to read through the change; I checked the results myself.

Tested-by: Ting-Han Hou <ue081723@xxxxxxxxx>
Reviewed-by: Ting-Han Hou <ue081723@xxxxxxxxx>