kthread permanently stuck in TASK_RUNNING but not on any runqueue, wakeups have no effect (6.12.73)

[Stepan Sigut]

Hi,

We started seeing stuck wireguard tunnels on our routers using newer kernel.
Encrypted packets are arriving from the peer but nothing ever comes out of
the wireguard interface. WireGuard handshakes run fine but data packets are
lost. We originally assumed it was a WireGuard issue and wanted to report it
there, but after further debugging it seems to be scheduler related.

On the affected machine, the threaded NAPI poller of the WireGuard peer
(napi/wg-en01-wan1-0, PID 528) was stuck in TASK_RUNNING while not being on
any runqueue. Waking it had no effect. It stayed in this state for days until
I rebooted the machine. I cannot reproduce it on demand but I collected state
from the live instance before recovering. Encrypted packets still arrived and
were enqueued for the peer (its rx ring was at capacity), but the poller
never ran again, so nothing was delivered and no error counters increased.

Environment:
- Debian 13 kernel, 6.12.73+deb13-amd64, x86_64, 4 cores
- system mostly idle, no RT tasks running, no isolcpus/nohz_full,
  no cpuset/cgroup cpu limits on the thread, affinity 0-3
- sched features are the defaults, DELAY_DEQUEUE is enabled

State of the task:

/proc/528/status said R (running). /proc/528/sched was frozen; two samples
minutes apart were identical:

  se.exec_start        :    1637904237.874372
  se.vruntime          :       3267544.298282
  se.sum_exec_runtime  :      10277208.164664
  nr_switches          :            131465615

`perf record -t 528` collected zero samples.

/sys/kernel/debug/sched/debug showed the task listed on CPU0 in state R while
CPU0's runqueue was empty at the same moment:

  cpu#0:
    .nr_running   : 0
    .curr->pid    : 0
  cfs_rq[0]:/
    .h_nr_queued  : 0
    .h_nr_delayed : 0
  runnable tasks:
   R napi/wg-en01-wa 528  3267544.298282 E 3267546.350642 ... 10277208.164664 131465615 120

Its vruntime was ~111000 behind the cfs_rq zero_vruntime (3378864.824232). A
sibling napi kthread on the same CPU (PID 543) tracked zero_vruntime exactly.
Comparing se.exec_start with the clock in the same dump, the task last ran
about 3 hours before the dump was taken which coincides with our monitoring
detecting the tunnel being down.

The napi_struct of the affected peer, read via BTF:

  napi.state = 0x333   (SCHED | MISSED | LISTED | NO_BUSY_POLL |
                        THREADED | SCHED_THREADED)
  rx ring count = 1024 (full)

SIGSTOP/SIGCONT on the task had no effect. I did not find any way to recover
the thread from userspace. `ip link del` on the interface hung and the ip
process could not be killed with SIGKILL; I recovered with a reboot.

History: the same symptom (tunnel rx dead, handshakes fine, but unlike in the
reported case I was able to recover by recreating the interface) happened at
least on two other routers over the past 3 weeks always on the busiest peer of
the router (~350 GB rx, ~130M napi wakeups on the thread's counters). I only
have detailed state for this instance, so I can't say whether the earlier ones
were the same. All machines were running the same kernel.

For transparency: I debugged this together with an AI assistant (Claude). All
commands were run by me on the machines and the outputs above are verbatim.

I can provide full sched debug dump, /proc/<pid>/sched samples, and other
debugging outputs I collected if needed.

Regards,
Stepan