[BUG] tcp: NULL f_inode in listening_get_first() leaves listener spinlock held

[Ville Poutiainen]

Hi,

I encountered a kernel NULL-pointer dereference in the TCP proc iterator
on Fedora 44, followed by a persistent listener-bucket spinlock lockup
when another process later read /proc/net/tcp.

Fedora bug with sanitized logs, exact kernel configuration, and decoded
vmlinux analysis:

https://bugzilla.redhat.com/show_bug.cgi?id=2527805

Affected kernel:
  7.1.10-200.fc44.x86_64

Architecture:
  x86_64

Initial failure:

  BUG: kernel NULL pointer dereference, address: 0000000000000230
  #PF: supervisor read access in kernel mode
  Oops: 0000 [#1] SMP NOPTI
  CPU: 1 UID: 1000 PID: 1634 Comm: pasta.avx2
  RIP: listening_get_first+0xc0/0x120

Call path:

  listening_get_first
  tcp_get_idx
  tcp_seq_start
  seq_read_iter
  seq_read
  proc_reg_read
  vfs_read
  ksys_read

The userspace syscall was read(fd=16, ...). The process was pasta,
performing automatic port discovery by reading /proc/net/tcp.

I obtained the exact Fedora debuginfo/vmlinux matching the affected
kernel. Its Build ID is:

  4c9470ef910bac97b4554b740c84f337cada5950

GDB against that exact binary shows:

  struct seq_file.file     = 0x60
  struct file.f_inode      = 0x20
  struct inode.i_private   = 0x230

The relevant instructions in listening_get_first are:

  +0x8f: call _raw_spin_lock
  ...
  +0xb8: mov 0x60(%rbp),%rdx
  +0xbc: mov 0x20(%rdx),%rdx
  +0xc0: mov 0x230(%rdx),%rdx
  ...
  +0xff: call _raw_spin_unlock

At the fault:

  RDX = 0
  CR2 = 0x230

Thus, as far as I can tell from the exact binary and register state,
seq->file->f_inode was NULL while the active /proc/net/tcp read was in
progress. The fault occurred when the code then accessed
inode->i_private.

The listener hash-bucket spinlock had already been acquired and the
Oops occurred before the normal unlock path.

The kernel continued after the Oops and reported:

  note: pasta.avx2[1634] exited with irqs disabled
  note: pasta.avx2[1634] exited with preempt_count 1

A few minutes later, "ss" became permanently stuck at 100% system CPU:

  native_queued_spin_lock_slowpath
  _raw_spin_lock
  listening_get_first+0x94
  tcp_get_idx
  tcp_seq_next
  seq_read_iter

The listener-bucket lock address in the original pasta Oops was:

  ffff8f0d05894ec0

The later ss soft-lockup was attempting to acquire the same address:

  ffff8f0d05894ec0

The watchdog subsequently reported the task stuck for 26, 52, 85, 112,
138, 164, 190, 216, and at least 253 seconds, together with RCU stalls.
The machine ultimately became unusable and required a forced reboot.

Crash-time userspace configuration:

  passt/pasta:  0^20260728.gf8df3f1-2.fc44
  RootlessKit:  3.1.0-1.fc44
  moby-engine:  29.7.2-1.fc44

RootlessKit was running with:

  --net=pasta
  --port-driver=implicit

which caused pasta to use its automatic TCP/UDP port scanning path.

The first Oops was tainted G OE due to the NVIDIA out-of-tree/unsigned
modules which were loaded on the system. I have not reproduced this on
an untainted or vanilla upstream kernel.

This failure has occurred once. I have not deliberately reproduced it,
because once the Oops occurs the leaked spinlock can lead to a
system-wide lockup.

The machine is now running Fedora kernel 7.1.12-200.fc44.x86_64, but I
also changed RootlessKit to the pesto port driver. The current pasta
instance uses --tcp-ports=none --udp-ports=none, so the original scan
path is no longer being exercised. Therefore the absence of another
failure on 7.1.12 does not establish that the kernel issue is fixed.

I have not established whether the underlying cause of the NULL f_inode
is a struct file lifetime/reuse problem, other memory corruption, or
something else.

The Fedora bug linked above contains:

  kernel-crash-public.txt
  kernel-config.txt
  vmlinux-analysis-public.txt

The first file contains the complete initial Oops and subsequent
soft-lockup/RCU traces. The vmlinux analysis contains the exact
disassembly and structure layouts from the matching Fedora debuginfo.

I currently do not have enough spare compute resources to run a
controlled reproduction on current mainline, but I can provide
additional targeted information or perform a specific test later if it
would help identify the cause.

Regards,
Ville Poutiainen