Re: [BUG] tpm: prolonged stall in vtpm_proxy_fops_release() on Linux 7.3

[铮]

Hi Stefan,

To make the reproduction evidence easier to inspect, the repository
(https://github.com/Zheng-zheng-codes/vtpm-proxy-release-hang-repro)
contains five positive reproductions in total:

- Four replays of the 50-program single-file reproducer (one automated
  and three manual).
- One separate two-step reproduction (a 49-program state-construction
  run, with the hang occurring when its executor exits).

The serial logs show the hung-task reports, with the release path
blocked in flush_work(). The accompanying sysrq-t dumps provide
snapshots of the tpm-vtpm workqueue. The first captures the full
running chip-register work stack, the blocked release, and the
workqueue backlog. The other dumps also show the blocked release and
the in-flight work/backlog state, although the full worker stack is
not available in those captures.

The files are:

Serial logs (5 positive reproductions):

  logs/serial-7.3-auto-repro.log    -- automated replay
  logs/serial-7.3-manual-repro.log  -- manual replay 1
  logs/serial-verify3.log           -- manual replay 2
  logs/serial-verify4.log           -- manual replay 3 (with
                                       syz-execprog stdout in
                                       logs/execprog-verify4.out)
  logs/serial-twostep.log           -- two-step replay (with
                                       syz-execprog stdout in
                                       logs/execprog-twostep-step1.out)

Sysrq-t task dumps (4):

  logs/vtpm-sysrq-t-all-tasks.txt   -- replay 1: full kworker stack,
                                       blocked release, and backlog
  logs/dmesg-sysrq-verify3.txt      -- replay 2: backlog and blocked
                                       release
  logs/dmesg-sysrq-verify4.txt      -- replay 3: backlog and blocked
                                       release
  logs/dmesg-sysrq-twostep.txt      -- two-step replay: in-flight work,
                                       backlog, and blocked release

Campaign instances:

  logs/report-7.3-*.txt
  logs/log-7.3-*.txt

Answering your questions in order:

> Four runs out of four?

Yes. Those four were the negative controls, not the positive
reproductions. The sequence was run four times on four freshly booted
bare VMs, and no hang was recorded in any of them:

  logs/serial-7.3-bare-no-trigger-1..4.log

> Does this mean that the 'scratch/pmem/NUMA layout' is required to
> reproduce the hang?

That was not established by the original comparison. The bare-VM runs
differed from the positive runs in two variables at once: the kernel
command line and the QEMU device layout. Therefore, the original
comparison could not isolate the effect of either factor.

To investigate this, I ran a controlled 2x2 test matrix using the same
50-program sequence, the same kernel, hung_task_panic=0, and a fresh VM
for each run.

The results were:

  Full command line + full layout    4/4 hangs (one automated and
                                     three manual replays)
  Full command line + bare layout    0/2
  Minimal command line + full layout 0/2
  Minimal command line + bare layout 0/4

In this test matrix, the hang was reproduced only when both the full
kernel command line and the full device layout were used together.
Neither one alone was sufficient to reproduce it in the tested runs.

The 4/4 result counts the four 50-program single-file reproductions
and does not include the separate two-step reproduction.

My current hypothesis is that the combination affects guest execution
timing. Possible contributors include page_owner overhead,
initialization of the additional devices, and NUMA-related I/O. This
may increase the likelihood that chip-register work is still running
or queued when the closing task reaches flush_work(). However, the
exact mechanism has not yet been established.

The logs and per-run kernel command lines are available in the
repository under logs/controlled/ and cmdline/.

> What is 'device layout'?

By device layout, I mean the virtual hardware configuration and
topology presented to the guest by QEMU.

The full layout includes, in addition to the root IDE disk:

  - 8 scratch virtio disks (2 x 16 GiB and 6 x 12 GiB, created with
    `qemu-img create -f qcow2 ...` by the QEMU wrapper for each VM;
    the wrapper excerpt, the sizes and a `qemu-img info` output are in
    cmdline/qemu-wrapper-scratch-create.txt);
  - 1 additional virtio disk used as a write log;
  - 2 NVDIMM/pmem devices (1 GiB memory-backend files each);
  - 2 NUMA nodes (2 vCPUs and 3 GiB of memory per node).

The bare layout has only the root IDE disk as a block device, with no
additional virtio disks or NVDIMM/pmem devices, and a single NUMA node.

The repository contains a captured QEMU invocation for the standard
run (cmdline/cmdline-7.3.txt, with hung_task_panic=1), the kernel
command line used for live retesting
(cmdline/cmdline-7.3-retest.txt, with hung_task_panic=0), and
sanitized QEMU argument templates for the full and bare layouts:

  cmdline/qemu-args-full-layout.txt
  cmdline/qemu-args-bare-layout.txt

The controlled retest uses hung_task_panic=0; the per-configuration
command lines are also available in cmdline/.

I hope this clarifies the reproduction conditions and the distinction
between the kernel command line and the QEMU device layout.

Best regards,
Zheng Zheng