Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup

[Surabhi Gogte (she/her)]

On Sun, Nov 1, 2026 at 11:15 PM Hannes Reinecke <hare@xxxxxxx> wrote:
>
> On 10/30/26 10:25 PM, Surabhi Gogte (she/her) wrote:
> > On Mon, Oct 26, 2026 at 3:57 AM Martin Belanger
> > <martin.nvme.dev@xxxxxxxxx> wrote:
> >>
> >> On 9/22/26 16:07, Surabhi Gogte wrote:
> >>   > Similar to commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
> >>   > allocation and startup"), refactor nvme tcp I/O queue setup to use async
> >>   > API, combining allocation and startup into a single parallel operation
> >>   > per queue. This reduces connection and reconnection setup time when
> >>   > there are delays in establishing connections, which is especially
> >>   > important for high-core-count hosts.
> >> [...]
> >>
> >> Hi Surabhi,
> >>
> >> This patch breaks NVMe/TCP connects with TLS. This is a different bug
> >> from the pf_cache_lock one that Shin'ichiro reported.
> >>
> >> Test: one VM with 4 CPUs, nvmet-tcp on 127.0.0.1 with tsas tls1.3, a
> >> connect with --tls and 4 I/O queues, and tlshd at upstream commit
> >> a0bd51d.
> >>
> >> - Linus master 6b4b543fa187: 14 of 20 connects fail.
> >> - Same tree with this patch reverted: 20 of 20 connects pass.
> >> - Same tree with Daniel's pf_cache_lock fix [1]: 22 of 40 and 17 of 40
> >>     connects fail.
> >>
> >> On each failed connect, one I/O queue logs:
> >>
> >>     nvme nvme0: queue 4: TLS handshake complete, error -110
> >>
> >> The tlshd that runs for the target logs:
> >>
> >>     failed to search 'psk' key 'NVMe0R01 <hostnqn> <subnqn>'
> >>     gnutls: Error in password/key file. (-31)
> >>
> >> and closes the connection. The tlshd that runs for the host then gets
> >> GNUTLS_E_PREMATURE_TERMINATION, which it reports to the kernel as
> >> ETIMEDOUT. The other queues use the same PSK, and their handshakes
> >> succeed. blktests nvme/062 and nvme/063 fail the same way.
> >>
> >> I can share the reproducer script.
> >
> > Hey Martin, can you try the tests once with tlshd using commit 3a6658e
> > ("tlshd: Link the NFS keyrings per handshake instead of at startup")
> > or ktls-utils 1.5. blktests nvme/062 and nvme/063 passed with this
> > commit. I believe that change fixed the keyring race in tlshd
> > independently. If the issue persists, let me know the reproducer
> > script so I can also take a look. I saw the same errors while running
> > blktests with an older tlshd version from ktls-utils 1.3.1 and
> > updating tlshd worked.
> >
> Ouch. That will be tricky for distros. If there is a specific commit
> which requires an update to ktlshd we need to have a way to identify
> that; otherwise we'll see lots of regressions where customers have
> updated only one side.
> Please modify the patch to work with older versions of ktlshd.
>

I'm not sure how to do this well and would like your advice.

The race is in the target's tlshd (keyring link/unlink per handshake,
fixed by ktls-utils commit 3a6658e in 1.5.0), not in the kernel.

Two things that I faced, getthing in the way on trying make the host
backwards compatible:
1. Nothing tells the host which tlshd the target runs.
2. Serializing only the handshake doesn't work: nvmet-tcp blocks in
   its accept work peeking the first bytes of each connection, so a
   socket that waits for a lock before its ClientHello stalls the
   accepts behind it. Serializing the whole queue setup works but
   gives up the parallelism for TLS entirely.

Can you suggest how to structure this to avoid the race while keeping the
parallel setup? Should we fallback to serial io queue setup in case of tls
timeout errors?

Thanks,
Surabhi