Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup

[Hannes Reinecke]

On 10/30/26 10:25 PM, Surabhi Gogte (she/her) wrote:
> On Mon, Oct 26, 2026 at 3:57 AM Martin Belanger
> <martin.nvme.dev@xxxxxxxxx> wrote:
> > On 9/22/26 16:07, Surabhi Gogte wrote:
> >   > Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
> >   > allocation and startup"), refactor nvme tcp I/O queue setup to use async
> >   > API, combining allocation and startup into a single parallel operation
> >   > per queue. This reduces connection and reconnection setup time when
> >   > there are delays in establishing connections, which is especially
> >   > important for high-core-count hosts.
> > [...]
> >
> > Hi Surabhi,
> >
> > This patch breaks NVMe/TCP connects with TLS. This is a different bug
> > from the pf_cache_lock one that Shin'ichiro reported.
> >
> > Test: one VM with 4 CPUs, nvmet-tcp on 127.0.0.1 with tsas tls1.3, a
> > connect with --tls and 4 I/O queues, and tlshd at upstream commit
> > a0bd51d.
> >
> > - Linus master 6b4b543fa187: 14 of 20 connects fail.
> > - Same tree with this patch reverted: 20 of 20 connects pass.
> > - Same tree with Daniel's pf_cache_lock fix [1]: 22 of 40 and 17 of 40
> >     connects fail.
> >
> > On each failed connect, one I/O queue logs:
> >
> >     nvme nvme0: queue 4: TLS handshake complete, error -110
> >
> > The tlshd that runs for the target logs:
> >
> >     failed to search 'psk' key 'NVMe0R01 <hostnqn> <subnqn>'
> >     gnutls: Error in password/key file. (-31)
> >
> > and closes the connection. The tlshd that runs for the host then gets
> > GNUTLS_E_PREMATURE_TERMINATION, which it reports to the kernel as
> > ETIMEDOUT. The other queues use the same PSK, and their handshakes
> > succeed. blktests nvme/062 and nvme/063 fail the same way.
> >
> > I can share the reproducer script.
>
> Hey Martin, can you try the tests once with tlshd using commit 3a6658e
> ("tlshd: Link the NFS keyrings per handshake instead of at startup")
> or ktls-utils 1.5. blktests nvme/062 and nvme/063 passed with this
> commit. I believe that change fixed the keyring race in tlshd
> independently. If the issue persists, let me know the reproducer
> script so I can also take a look. I saw the same errors while running
> blktests with an older tlshd version from ktls-utils 1.3.1 and
> updating tlshd worked.
Ouch. That will be tricky for distros. If there is a specific commit
which requires an update to ktlshd we need to have a way to identify
that; otherwise we'll see lots of regressions where customers have
updated only one side.
Please modify the patch to work with older versions of ktlshd.

Cheers,

Hannes
--
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@xxxxxxx                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich