Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup

From: Surabhi Gogte (she/her)

Date: Mon Nov 16 2026 - 12:00:01 EST


On Sun, Nov 15, 2026 at 3:36 AM Shin'ichiro Kawasaki
<shinichiro.kawasaki@xxxxxxx> wrote:
>
> On Oct 20, 2026 / 13:48, Surabhi Gogte (she/her) wrote:
> > On Sun, Oct 18, 2026 at 5:26 AM Shin'ichiro Kawasaki
> > <shinichiro.kawasaki@xxxxxxx> wrote:
> > >
> > > On Sep 22, 2026 / 14:07, Surabhi Gogte wrote:
> > >
> > > Recently, blktests-CI runs for the kernel at block/for-next branch started
> > > reporting failures of the test case nvme/004 (with tcp transport) and md/001.
> > > The kernel reported a WARN related to lock->magic, and it caused the failures.
> > > FYI, here I share the failure run log at md/001 [1] and dmesg at the failure.
> > >
> > > The md/001 confirms a bug fix in md driver. The bug was triggered when a
> > > nvme device with tcp transport is used as its backend. Then the test case
> > > exercises the nvme-tcp driver, which this patch touches.
> > >
> > > I bisected and found that this commit is the trigger of the failures. When I
> > > revert this commit from the block/for-next branch tip with git hash
> > > 34fbfb9d8d1f, the failure goes away.
> > >
> > > Surabhi, could you take a look in the failure?
> > >
> >
> > Yes, there is a bug in commit 61b0ed3be81b ("nvme-tcp: parallelize I/O
> > queue allocation and startup"). To know the queue range before the
> > queues are set up, it moved the I/O tag set allocation ahead of I/O
> > queue allocation. blk_mq_alloc_tag_set() calls
> > nvme_tcp_init_request() for every request, which takes pf_cache_lock
> > of the request's queue, but nvme_tcp_alloc_queue() only initializes
> > that mutex later:
> > nvme_tcp_configure_io_queues()
> > nvme_alloc_io_tag_set()
> > blk_mq_alloc_tag_set()
> > blk_mq_alloc_rqs()
> > nvme_tcp_init_request() mutex_lock(pf_cache_lock) <- WARN
> > nvme_tcp_setup_io_queues()
> > nvme_tcp_alloc_queue() mutex_init(pf_cache_lock)
> >
> > So with CONFIG_DEBUG_MUTEXES the first connect trips
> > DEBUG_LOCKS_WARN_ON(lock->magic != lock) in __mutex_lock().
> > That matches the call trace in your dmesg.
> >
> > Daniel Wagner posted a fix for this as patch 1 of [PATCH 0/5]
> > nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs:
> > lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-0-355156be27b2@xxxxxxxxxx
> >
> > It initializes pf_cache_lock for all queues in nvme_tcp_alloc_ctrl(),
> > right after the queues array is allocated, and drops the
> > mutex_destroy() calls so the lock stays valid for the lifetime of the
> > controller.
> >
> > I ran blktests with his fix on top of block/for-next (34fbfb9d8d1f, the
> > same tip you bisected on) and md/001 passes with no WARN related to
> > lock->magic. Without the fix, the same tree and config fails md/001
> > with the WARN at kernel/locking/mutex.c:625.
> >
> > With the fix : nvme-tcp: initialize pf_cache_lock when allocating the controller
> > lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-1-355156be27b2@xxxxxxxxxx/
> >
> > ~/blktests$ sudo ./check md/001
> > md/001 (Raid with bitmap on tcp nvmet with opt-io-size over bitmap
> > size) [passed]
> > runtime 1.779s ... 1.814s
>
> Surabhi, thanks for the confirmation. It's great taht The md/001 failure is
> gone. Said that, now I face failrues of other test cases: nvme/062 and 063.
>
> While I was running blktests on v7.4-rc1 and v7.4-rc2 kernel, I observed the two
> test cases nvme/062 and 063 fail for tcp transport [*]. I bisected and found
> that this patch as the commit 61b0ed3be81b is the trigger. I observed the
> failure with nvme-cli versions 3.0, 3.1 and 3.2. ktls-utils version is 1.4.0.
>
> When I reported blktests failures for v7.4-rc1 kernel [#], the failure was
> not observed by some reason. But now the failure is observed in stable manner.
>
> [#] https://lore.kernel.org/linux-block/avBz7uOGYo2y426_@shinmob/
>
> It looks like nvme connect can fail for tls connection with multiple I/O
> queues. I created a blktests patch to limit number of I/O quees to 1 [**],
> and observed the failure goes away. But I'm not sure if this is just hiding
> the failure.
>
> Are the failures observed in your environment?
>

Hi Shin'ichiro, I ran bltkests with ktls 1.5.0. It contains the tlshd
commit 3a6658e
("tlshd: Link the NFS keyrings per handshake instead of at startup")
which fixes the keyring race during the handshake. Both tests 62 and
63 pass with it.
Can you retest with that version to see if the issue persists? Can you
also share the kernel logs of the failure.

Ref: https://lore.kernel.org/linux-nvme/CAMof9cU5s0ck20fcSAYSvK-xpcP5_Hv2BmN20Bh3btpLum7THg@xxxxxxxxxxxxxx/

Regards,
Surabhi.

>
> [*] nvme/062 and nvme/063 failure
>
> nvme/062 (tr=tcp) (Create TLS-encrypted connections) [failed]
> runtime 4.678s ... 13.619s
> --- tests/nvme/062.out 2026-11-09 20:12:01.056000000 +0900
> +++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad 2026-11-15 20:21:09.213000000 +0900
> @@ -1,6 +1,10 @@
> Running nvme/062
> Test unencrypted connection w/ tls not required
> Test encrypted connection w/ tls not required
> +FAIL: nvme connect return error code
> +WARNING: connection is not encrypted
> Test unencrypted connection w/ tls required (should fail)
> Test encrypted connection w/ tls required
> ...
> (Run 'diff -u tests/nvme/062.out /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad' to see the entire diff)
> nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation)
> runtime 3.123s ...
> WARNING: Test did not clean up port: 0
> nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation) [failed]
> runtime 3.123s ... 7.313sost: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
> --- tests/nvme/063.out 2026-11-09 20:12:01.060000000 +0900
> +++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/063.out.bad 2026-11-15 20:21:30.207000000 +0900
> @@ -1,5 +1,4 @@
> Running nvme/063
> Test secure concatenation with SHA256
> -Reset controller
> -Test secure concatenation with SHA384
> -Test complete
> +FAIL: nvme connect return error code
> +WARNING: connection failed
>
>
> [**] trial blktests patch
>
> diff --git a/tests/nvme/062 b/tests/nvme/062
> index eabe2d9..69a05b6 100755
> --- a/tests/nvme/062
> +++ b/tests/nvme/062
> @@ -55,7 +55,7 @@ test() {
>
> # Test encrypted connection
> echo "Test encrypted connection w/ tls not required"
> - _nvme_connect_subsys --tls
> + _nvme_connect_subsys --tls --nr-io-queues 1
>
> ctrl=$(_find_nvme_dev "${def_subsysnqn}")
> if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then
> @@ -77,7 +77,7 @@ test() {
>
> # Test encrypted connection
> echo "Test encrypted connection w/ tls required"
> - _nvme_connect_subsys --tls
> + _nvme_connect_subsys --tls --nr-io-queues 1
>
> ctrl=$(_find_nvme_dev "${def_subsysnqn}")
> if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null; then
> diff --git a/tests/nvme/063 b/tests/nvme/063
> index feb07bf..900bb9f 100755
> --- a/tests/nvme/063
> +++ b/tests/nvme/063
> @@ -46,7 +46,7 @@ test() {
> _set_nvmet_dhgroup "${def_hostnqn}" "ffdhe2048"
>
> echo "Test secure concatenation with SHA256"
> - _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
> + _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
>
> ctrl=$(_find_nvme_dev "${def_subsysnqn}")
> if [[ -z "$ctrl" ]]; then
> @@ -88,7 +88,7 @@ test() {
> _set_nvmet_dhgroup "${def_hostnqn}" "ffdhe3072"
>
> echo "Test secure concatenation with SHA384"
> - _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
> + _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
>
> ctrl=$(_find_nvme_dev "${def_subsysnqn}")
> if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then