Re: [BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[铮]

Hi Stefan,

Thank you for the patch
("tpm: tpm_vtpm_proxy: Set STATE_OPENED_FLAG before inode is created").
We applied and tested it, and we have also reproduced the boot warning
Stefan reported and identified a sufficient condition for
it (running without KVM acceleration). Two findings first; "Where to
look" at the end points to the exact files in our repository.

1. Your patch does not change the outcome of the hang

A/B test, same session, same QEMU layout and command line, two-step
reproduction (49-program state-construction sequence; the hang appears
when its executor process exits and releases the vTPM fds):

  patched kernel  (7.3.0-kernsight-kasan #3, your patch applied;
                   drivers/char/tpm/tpm_vtpm_proxy.c is the only change):
    hang reproduced, 1/1 run
    t=414s: INFO: task syz.0.26:10773 blocked for more than 143 seconds.
            (t=434s: 163 seconds), blocked in
            vtpm_proxy_fops_release() -> __flush_work()
    t=460s sysrq-t: the chip-register work is sleeping in
            tpm_transmit() <- vtpm_proxy_request_locality(), with
            vtpm_proxy_tpm_req_canceled() present as the transmit wait's
            cancel callback; workqueue tpm-vtpm backlog
            "inactive: 64*vtpm_proxy_work" (plus 4+58 on a second pwq);
            serial repeats "tpm tpmN: Operation Timed out".

  unpatched control (7.3.0-kernsight-kasan #1, same session):
    hang reproduced, 1/1 run, t=387s, identical blocked stack.

So the udev / STATE_OPENED_FLAG race does not explain this stall. This
fits the shape of the reproducer: it creates vTPM proxy devices via
/dev/vtpmx and exits; nothing else ever opens the server-side anon-inode
fd, so there is no udev open/close racing the flag. What happens is:
fput() of the server fd -> vtpm_proxy_fops_release() ->
vtpm_proxy_work_stop() -> flush_work(), while the chip-register work
keeps taking TPM-timeout-sized naps on a client that never responds and
dozens of such works are queued behind it.

v7.4-rc1 is affected as well: drivers/char/tpm/tpm_vtpm_proxy.c is
unchanged between v7.3.0 and v7.4-rc1 (empty diff), and the same
reproduction hangs there 1/1 (t=402s, same signature).

Given that vtpm_proxy_tpm_req_canceled() is registered as the transmit
wait's cancel callback, which direction would you prefer for a fix?
  (a) make teardown terminate the in-flight register work promptly
      (e.g. the transmit path honors the closed/torn-down state so
      tpm_transmit() aborts instead of timing out), or
  (b) a lifecycle-safe teardown approach you consider appropriate for
      the release path and the in-flight register work (we defer to your
      preferred design).
We are happy to build and test whichever you prefer; the environment is
scripted, a build + run cycle takes about 15 minutes.

2. Your boot warning reproduces under TCG (no KVM acceleration)

You reported (2026-11-01) that the suggested QEMU command line fails
with:

  sched: CPU #2's llc-sibling CPU #0 is not on the same node!
         [node: 1 != 0]. Ignoring dependency.
  WARNING: arch/x86/kernel/smpboot.c:329 at topology_sane.isra.0

With KVM acceleration we could NOT reproduce it on our hosts, in any
combination we tried: QEMU 6.2 and QEMU 10.2.1 (Ubuntu 26.04,
seabios 1.17.0-1ubuntu1 — the BIOS string matches his report), full
layout, the original suggested command line, and even forcing the exact
cross-node-LLC shape via "-smp 4,sockets=1,cores=4,threads=1": clean
boots every time.

Without KVM acceleration it reproduces byte-for-byte: booting the same
QEMU 10.2.1 stack with "-accel tcg,thread=multi -cpu max" (same layout,
same command line, same unpatched kernel) produces exactly the warning
from your report (topology_sane.isra.0+0xa3/0xb0, same Hardware
name string) and then panics via panic_on_warn=1 — i.e. the boot fails
as he described.

Mechanism: with KVM + "-cpu host" the cache-topology CPUID leaves mostly
pass through our host CPU (Intel Xeon Gold 6326), so the guest never
sees LLC siblings across the NUMA nodes of the suggested layout; under
TCG QEMU synthesizes the topology itself, LLC siblings span both nodes,
and topology_sane() fires. Whether your machine ran TCG (e.g. no
usable /dev/kvm) or a host CPU whose KVM enumeration differs cannot be
determined from your report, but the condition is now reproducible and
checkable on our side. Note the two runs differ in both the accelerator
and the CPU model (KVM runs used "-cpu host", the TCG run "-cpu max"),
so the experiment isolates the pair, not the accelerator alone.

The mitigation works in both modes: "-smp 4,sockets=2,cores=2,threads=1"
keeps each socket's LLC within one NUMA node — the TCG boot then
proceeds with no warning, and under KVM the hang still reproduces with
the fixed topology, so the one-line change makes the boot robust
regardless of the accelerator while the bug stays testable.

3. Where to look in the repository [1]

  tcg-topology-20261109/            <- the TCG finding
    README.md                       KVM vs TCG comparison, mechanism
    serial-tcg-hit.log              boot under TCG: the warning + panic
                                    (byte-identical to your report)
    serial-tcg-fixed-topology.log   same TCG boot with the -smp
                                    mitigation: clean boot
    boot-tcg.sh / boot-tcg-fixed.sh exact QEMU wrappers of both runs

  patch-ab-test-20261107/           <- the patch A/B and v7.4-rc1
    README.md                       results table and build details
    0001-tpm-tpm_vtpm_proxy-Set-STATE_OPENED_FLAG-before-inod.patch
                                    the tested patch
    serial-7.3-patched.log          patched run (two hung-task reports)
    sysrq-t-7.3-patched-excerpts.txt  worker stack + workqueue backlog
    serial-7.3-unpatched-ctrl.txt   unpatched control run
    sysrq-t-7.3-unpatched-ctrl.txt  control sysrq-t dump
    serial-7.4-rc1.log              v7.4-rc1 run (hung-task report)
    sysrq-t-7.4-rc1.txt             v7.4-rc1 sysrq-t dump

  repro/                            state-construction sequence + testcase
  cmdline/                          guest kernel command lines, QEMU args
  kernel/kernel-7.3.config          kernel configuration (KASAN/KCOV)
  logs/                             original syzkaller campaign evidence

4. Caveat

In every log the window ends while the release task is still blocked, so
we have not captured whether the release eventually completes once the
backlog drains; we are not claiming a permanent deadlock.

Happy to test any proposed patch in either direction, and to provide any
file inline.

Best regards,
Zheng Zheng

[1] https://github.com/Zheng-zheng-codes/vtpm-proxy-release-hang-repro

On Sat, Nov 7, 2026 at 3:11 AM Stefan Berger <stefanb@xxxxxxxxxxxxx> wrote:
>
>
>
> On 11/1/26 12:36 PM, Jarkko Sakkinen wrote:
> > On Sun, Nov 01, 2026 at 09:22:30AM +0800, 铮 wrote:
> >> Hi Peter, Jarkko and Jason,
> > >>
> >> Question
> >> --------
> >> vtpm_proxy_tpm_req_canceled() already exists and the driver marks the
> >> device as being torn down before flush_work(); from the captured dumps
> >> I cannot tell why the wait nevertheless persists for minutes. Should
> >> the release path be bounded, or should the in-flight chip-register work
> >> be ended more promptly once the proxy is being torn down? I am happy to
> >> test proposed patches.
>
> I have tried to reproduce this now but cannot. The kernel fails for me
> like this when booting with the suggested QEMU command line:
>
> [    3.224399][    T1] Timer migration: 2 hierarchy levels; 8 children
> per group; 1 crossnode level
> [    3.239948][    T1] smp: Bringing up secondary CPUs ...
> [    3.243980][    T1] smpboot: x86: Booting SMP configuration:
> [    3.246035][    T1] .... node  #0, CPUs:      #1
> [    3.252255][    T1] .... node  #1, CPUs:   #2 #3
> [    0.076393][    T0] ------------[ cut here ]------------
> [    0.076393][    T0] sched: CPU #2's llc-sibling CPU #0 is not on the
> same node! [node: 1 != 0]. Ignoring dependency.
> [    0.076393][    T0] WARNING: arch/x86/kernel/smpboot.c:329 at
> topology_sane.isra.0+0x97/0xb0, CPU#2: swapper/2/0
> [    0.076393][    T0] Modules linked in:
> [    0.076393][    T0] CPU: 2 UID: 0 PID: 0 Comm: swapper/2 Not tainted
> 7.3.0-kernsight-kasan #1 PREEMPT(full)
> [    0.076393][    T0] Hardware name: QEMU Ubuntu 26.04 PC (i440FX +
> PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1ubuntu1 04/01/2014
> [    0.076393][    T0] RIP: 0010:topology_sane.isra.0+0xa3/0xb0
> [    0.076393][    T0] Code: 07 00 44 89 ef e8 ed fa 07 00 44 89 e7 89
> c3 e8 e3 fa 07 00 41 89 c0 48 8d 3d 79 4e 3e 10 41 89 d9 44 89 e9 4c 89
> fa 44 89 e6 <67> 48 0f b9 3a eb a5 66 0f 1f 44 00 00 90 90 90 90 90 90
> 90 90 90
> [    0.076393][    T0] RSP: 0000:ffffc900001afde0 EFLAGS: 00010092
>
> The first thing I was going to try is whether this has something to do
> with the 'STATE_OPENED_FLAG' flag. There's one location where this is
> set and that's after the anon_inode_getfile() was called. For a server
> side device to be able to access the device, udev would have to run.
> Maybe udev does run between anon_inode_getfile() and the server side
> opens the device and the immediately closes it and resets the flag
> BEFORE current vtpm_proxy_fops_open() sets the flag and makes the client
> think that there still is an server-side app responding to requests. So,
> let's reverse the order and set the flag before the inode gets created
> and the flag can be reset - patch also as attachment:
>
>
> diff --git a/drivers/char/tpm/tpm_vtpm_proxy.c
> b/drivers/char/tpm/tpm_vtpm_proxy.c
> index b81fd2a537df..245a8f959bc7 100644
> --- a/drivers/char/tpm/tpm_vtpm_proxy.c
> +++ b/drivers/char/tpm/tpm_vtpm_proxy.c
> @@ -197,10 +197,8 @@ static __poll_t vtpm_proxy_fops_poll(struct file
> *filp, poll_table *wait)
>    *
>    * Called when setting up the anonymous file descriptor
>    */
> -static void vtpm_proxy_fops_open(struct file *filp)
> +static void vtpm_proxy_fops_open(struct proxy_dev *proxy_dev)
>   {
> -       struct proxy_dev *proxy_dev = filp->private_data;
> -
>         proxy_dev->state |= STATE_OPENED_FLAG;
>   }
>
> @@ -550,6 +548,9 @@ static struct file *vtpm_proxy_create_device(
>                 goto err_delete_proxy_dev;
>         }
>
> +       /* simulate an open() on the server side */
> +       vtpm_proxy_fops_open(proxy_dev);
> +
>         file = anon_inode_getfile("[vtpms]", &vtpm_proxy_fops, proxy_dev,
>                                   O_RDWR);
>         if (IS_ERR(file)) {
> @@ -558,8 +559,6 @@ static struct file *vtpm_proxy_create_device(
>         }
>
>         /* from now on we can unwind with put_unused_fd() + fput() */
> -       /* simulate an open() on the server side */
> -       vtpm_proxy_fops_open(file);
>
>         if (proxy_dev->flags & VTPM_PROXY_FLAG_TPM2)
>                 proxy_dev->chip->flags |= TPM_CHIP_FLAG_TPM2;
>
>
>