Re: [PATCH v4 0/2] tpm: Stop retrying TPMs that have stopped working
From: Jarkko Sakkinen
Date: Fri Nov 20 2026 - 16:30:02 EST
On Fri, Nov 20, 2026 at 12:47:06PM -0800, Breno Leitao wrote:
> On Meta's datacenter, given the size, TPMs that stop working write about
> 54 million lines a day to the kernel log, because nothing in the kernel
> gives up on them.
>
> Two cases make up nearly all of it.
>
> Some TPMs stop responding. Every command then fails only after the
> transport timeout, while the hwrng thread and userspace keep sending
> more.
>
> Other TPMs get reset under the running OS, and answer every command with
> TPM_RC_INITIALIZE (256) from then on, while the hwrng thread retries
> every 10 seconds:
>
> tpm tpm0: A TPM error (256) occurred attempting get random
>
> The goal is to retire TPMs that are not properly working, similarly to
> what was done for EFI in commit 4b2c033b5bc971 ("efi/runtime-wrappers:
> retire the worker if a wedged call ever returns")
>
> Patch 1 disables the chip once two commands in a row have timed out,
> instead of keeping it in this desolate state. Patch 2, from Jarkko,
> disables it once it answers TPM_RC_INITIALIZE to anything but
> TPM2_Startup.
>
> Disabling the chip makes the failure explicit: every later command
> fails at once instead of timing out.
>
> A health-check daemon, such as machine checker, can then detect
> the broken hardware and send the machine to repair.
>
> Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
> ---
> Changes in v4:
> - Replace the count of timeouts and the tpm.max_missed_responses
> parameter with TPM_CHIP_FLAG_MISSED_RESPONSE: one timeout is
> tolerated, and a second one in a row disables the TPM (Jarkko)
> - Say in patch 1 when TPMs stop responding (Jarkko), and drop the
> claim that some of them recover
> - Fix sashiko findings:
> - A timeout while a signal is pending no longer counts, since some
> drivers return -ETIME when a signal interrupts the wait
> - tpm1_do_selftest() clears the flag on the timeouts it expects from
> TPM 1.2 chips that block during the self test
> - Link to v3: https://patch.msgid.link/20261113-tpm_unresponsive-v3-0-8138c8e283b8@xxxxxxxxxx
>
> Changes in v3:
> - Rename tpm.max_unanswered to tpm.max_missed_responses and
> nr_unanswered to nr_missed_responses, and drop a redundant comment
> (Jarkko)
> - Document tpm.max_missed_responses in kernel-parameters.txt (Jarkko)
> - Check the limit with a modulo (Jarkko), guarded so that 0 still
> means never
> - Link to v2: https://patch.msgid.link/20261112-tpm_unresponsive-v2-0-7454d2c899ef@xxxxxxxxxx
>
> Changes in v2:
> - Disable the TPM after 60 commands in a row time out, instead of after
> ten minutes of timeouts, and make the limit the tpm.max_unanswered
> boot parameter (Jarkko)
> - Open-code the check in tpm_transmit() and drop the "timeout" field
> names (Jarkko)
> - Replace the hwrng failure counter with Jarkko's patch, which disables
> the TPM on TPM_RC_INITIALIZE
> - In that patch, move the check ahead of the RETRY/TESTING break, where
> it never ran, and skip TPM2_Startup, which a TPM that is already
> started fails with TPM_RC_INITIALIZE
> - Fix sashiko findings:
> - Two timeouts ten minutes apart no longer disable an idle TPM
> - Link to v1: https://patch.msgid.link/20261111-tpm_unresponsive-v1-0-42ac1b965eb9@xxxxxxxxxx
>
> ---
> Breno Leitao (1):
> tpm: Disable a TPM that stops responding
>
> Jarkko Sakkinen (1):
> tpm: Disable TPM on TPM_RC_INITIALIZE
>
> drivers/char/tpm/tpm-interface.c | 20 ++++++++++++++++++++
> drivers/char/tpm/tpm1-cmd.c | 1 +
> drivers/char/tpm/tpm2-cmd.c | 2 ++
> include/linux/tpm.h | 1 +
> 4 files changed, 24 insertions(+)
> ---
> base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47
> change-id: 20261110-tpm_unresponsive-a5f71c1db026
>
> Best regards,
> --
> Breno Leitao <leitao@xxxxxxxxxx>
>
Reviewed-by: Jarkko Sakkinen <jarkko@xxxxxxxxxx>
Br, Jarkko