[PATCH RFC POC 00/50] file: handle files on syscall exit
From: Christian Brauner
Date: Tue Sep 15 2026 - 07:43:39 EST
Hey,
The idea in this series is old-ish and really never let go of me and so
I wanted to at least dump it onto the list once even if it's just for
illustrative purposes.
I still had parts of an implementation laying around that I started
montsh ago alongside FD_PREPARE()/FD_ADD(). I sat down and finished it.
We've spoken about this idea a few times over the years that we could
reserve fds and files on a task and then install or a clean them up on
syscall success or error and get rid of most of the complicated cleanup
dance that we have in a lot of code. In particular drm. It last came up
during the Rust file descriptor reservation discussion quite some time
ago.
TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
allocates a descriptor like get_unused_fd_flags() does and records it in
a slot on the task. fd_stage() attaches the file to that slot and
returns the number.
When the syscall returns success the exit path installs every staged
file. When it returns an error it drops the descriptors and the files.
So a caller reserves, hands the number to userspace whenever it wants,
creates the file, stages it and returns errors without unwinding
anything.
fd_prepare() returns the slot itself, as a const pointer. The
preexisting fd_prepare_fd() and fd_prepare_file() give access to the fd
and file.
get_unused_fd_flags() and fd_install() don't change. A descriptor is
only reserved where a caller asks for it. And open(), dup() and all
other syscalls that maximize speed simply use FD_ADD().
The task keeps two slots inline. For SCM_RIGHTS and multi-descriptor
ioctls a spill array is added. It stick with the task.
Reservations belong to the thread and the syscall that made them. A
child of fork() starts without any. A thread can't unshare its fdtagble
with outstanding reservations. Kernel threads never return to userspace
so nothing would commit. Anything left at exit is a bug and gets warned
about and dropped.
Christian
Signed-off-by: Christian Brauner (Amutable) <brauner@xxxxxxxxxx>
---
Christian Brauner (50):
file: install files on syscall exit
entry: commit fds on syscall exit
alpha: commit fds on syscall exit
ARC: commit fds on syscall exit
ARM: commit fds on syscall exit
arm64: commit fds on syscall exit
csky: commit fds on syscall exit
hexagon: commit fds on syscall exit
m68k: commit fds on syscall exit
microblaze: commit fds on syscall exit
MIPS: commit fds on syscall exit
nios2: commit fds on syscall exit
openrisc: commit fds on syscall exit
parisc: commit fds on syscall exit
sh: commit fds on syscall exit
sparc: commit fds on syscall exit
um: commit fds on syscall exit
xtensa: commit fds on syscall exit
file: require the syscall exit hook from every architecture
file: warn when the descriptor table is unshared with slots
io_uring: commit fds per request
net: install SCM_RIGHTS descriptors when recvmsg() returns
file: open-code receive_fd()'s immediate install
file: make FD_ADD() a standalone immediate install
file: reimplement FD_PREPARE() on the deferred fd_prepare() path
dma-buf: stop unwinding sync file descriptors by hand
drm/amdkfd: stop collecting CRIU dma-buf descriptors for a final install
drm/msm: install the out-fence descriptor when the ioctl returns
drm/virtio: install the out-fence descriptor when the ioctl returns
drm/vmwgfx: install the out-fence descriptor when the ioctl returns
vfio: install the migration data descriptor when the ioctl returns
liveupdate: install the session descriptors when the ioctl returns
io_uring/zcrx: install the exported descriptor when the request returns
sctp: install the peeloff descriptor when the syscall returns
ALSA: compress: install the task descriptors when the ioctl returns
nitro_enclaves: install the enclave descriptor when the ioctl returns
tpm: vtpm_proxy: install the server descriptor when the ioctl returns
perf: stop putting the event descriptor back on failure
seccomp: stop putting the listener descriptor back on failure
KVM: stop putting descriptors back on failure
KVM: guest_memfd: stop putting the descriptor back on failure
drm: stop unwinding descriptors by hand
drm/amdgpu: stop unwinding the fence descriptor by hand
drm/etnaviv: install the out-fence descriptor when the ioctl returns
accel/habanalabs: stop putting the dma-buf descriptor back on failure
xen/gntdev-dmabuf: stop putting the descriptor back on failure
iio: buffer: install the buffer descriptor when the ioctl returns
misc: fastrpc: install the dma-buf descriptor when the ioctl returns
iommufd: stop putting descriptors back on failure
Drivers: hv: mshv: stop putting descriptors back on failure
arch/alpha/include/asm/thread_info.h | 2 +
arch/alpha/kernel/entry.S | 24 ++-
arch/alpha/kernel/ptrace.c | 3 +
arch/arc/include/asm/thread_info.h | 2 +
arch/arc/kernel/entry.S | 5 +
arch/arc/kernel/ptrace.c | 4 +
arch/arm/include/asm/thread_info.h | 2 +
arch/arm/kernel/entry-common.S | 2 +
arch/arm/kernel/ptrace.c | 4 +
arch/arm64/include/asm/thread_info.h | 4 +-
arch/arm64/kernel/ptrace.c | 4 +
arch/csky/include/asm/thread_info.h | 2 +
arch/csky/kernel/entry.S | 9 +
arch/csky/kernel/ptrace.c | 4 +
arch/hexagon/include/asm/thread_info.h | 2 +
arch/hexagon/kernel/traps.c | 4 +
arch/m68k/68000/entry.S | 10 ++
arch/m68k/coldfire/entry.S | 12 ++
arch/m68k/include/asm/thread_info.h | 2 +
arch/m68k/kernel/entry.S | 2 +
arch/m68k/kernel/ptrace.c | 3 +
arch/microblaze/include/asm/thread_info.h | 5 +-
arch/microblaze/kernel/ptrace.c | 4 +
arch/mips/include/asm/thread_info.h | 5 +-
arch/mips/kernel/ptrace.c | 4 +
arch/nios2/include/asm/thread_info.h | 2 +
arch/nios2/kernel/entry.S | 5 +
arch/nios2/kernel/ptrace.c | 3 +
arch/openrisc/include/asm/thread_info.h | 5 +-
arch/openrisc/kernel/entry.S | 5 +-
arch/openrisc/kernel/ptrace.c | 4 +
arch/parisc/include/asm/thread_info.h | 2 +
arch/parisc/kernel/entry.S | 12 ++
arch/parisc/kernel/ptrace.c | 4 +
arch/powerpc/platforms/cell/spufs/inode.c | 12 +-
arch/sh/include/asm/thread_info.h | 10 +-
arch/sh/kernel/entry-common.S | 8 +-
arch/sh/kernel/ptrace_32.c | 4 +
arch/sparc/include/asm/thread_info_32.h | 2 +
arch/sparc/include/asm/thread_info_64.h | 9 +-
arch/sparc/kernel/entry.S | 2 +-
arch/sparc/kernel/ptrace_32.c | 4 +
arch/sparc/kernel/ptrace_64.c | 4 +
arch/sparc/kernel/syscalls.S | 4 +-
arch/um/include/asm/thread_info.h | 2 +
arch/um/kernel/ptrace.c | 4 +
arch/xtensa/include/asm/thread_info.h | 2 +
arch/xtensa/kernel/entry.S | 5 +
arch/xtensa/kernel/ptrace.c | 4 +
drivers/accel/habanalabs/common/memory.c | 13 +-
drivers/char/tpm/tpm_vtpm_proxy.c | 30 +---
drivers/dma-buf/dma-buf.c | 33 ++--
drivers/dma-buf/sw_sync.c | 40 ++---
drivers/dma-buf/sync_file.c | 52 ++----
drivers/gpio/gpiolib-cdev.c | 18 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c | 16 +-
drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 67 ++------
drivers/gpu/drm/drm_lease.c | 20 +--
drivers/gpu/drm/drm_prime.c | 13 +-
drivers/gpu/drm/drm_syncobj.c | 44 ++---
drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c | 14 +-
drivers/gpu/drm/msm/msm_gem_submit.c | 22 +--
drivers/gpu/drm/msm/msm_gem_vma.c | 22 +--
drivers/gpu/drm/msm/msm_perfcntr.c | 6 +-
drivers/gpu/drm/virtio/virtgpu_submit.c | 32 ++--
drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c | 29 ++--
drivers/hv/mshv_root_main.c | 27 +--
drivers/hv/mshv_vtl_main.c | 20 +--
drivers/iio/industrialio-buffer.c | 26 +--
drivers/iommu/iommufd/eventq.c | 39 ++---
drivers/media/mc/mc-request.c | 6 +-
drivers/misc/fastrpc.c | 19 +--
drivers/misc/ntsync.c | 6 +-
drivers/vfio/vfio_main.c | 25 +--
drivers/virt/nitro_enclaves/ne_misc_dev.c | 33 ++--
drivers/xen/gntdev-dmabuf.c | 14 +-
fs/eventfd.c | 6 +-
fs/eventpoll.c | 6 +-
fs/exec.c | 3 +
fs/file.c | 245 +++++++++++++++++++++++++--
fs/namespace.c | 12 +-
fs/nsfs.c | 6 +-
fs/xfs/xfs_handle.c | 6 +-
include/linux/entry-common.h | 8 +-
include/linux/file.h | 160 +++++------------
include/linux/sched.h | 21 +++
include/linux/thread_info.h | 2 +
io_uring/io_uring.c | 27 +++
io_uring/mock_file.c | 5 +-
io_uring/zcrx.c | 25 +--
kernel/bpf/bpf_iter.c | 6 +-
kernel/bpf/token.c | 6 +-
kernel/events/core.c | 31 ++--
kernel/exit.c | 1 +
kernel/fork.c | 3 +
kernel/liveupdate/luo_core.c | 48 ++----
kernel/liveupdate/luo_session.c | 24 +--
kernel/seccomp.c | 13 +-
mm/userfaultfd.c | 6 +-
net/core/scm.c | 13 +-
net/handshake/netlink.c | 20 ++-
net/kcm/kcmsock.c | 5 +-
net/sctp/socket.c | 78 ++++-----
sound/core/compress_offload.c | 24 ++-
virt/kvm/guest_memfd.c | 20 +--
virt/kvm/kvm_main.c | 65 +++----
106 files changed, 962 insertions(+), 861 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260915-work-fd-reserve-unify-folded-b89d3154b3d0