Re: [PATCH v4 3/3] nvme-tcp: parallelize I/O queue allocation and startup
From: Shin'ichiro Kawasaki
Date: Sun Nov 15 2026 - 06:37:05 EST
On Oct 20, 2026 / 13:48, Surabhi Gogte (she/her) wrote:
> On Sun, Oct 18, 2026 at 5:26 AM Shin'ichiro Kawasaki
> <shinichiro.kawasaki@xxxxxxx> wrote:
> >
> > On Sep 22, 2026 / 14:07, Surabhi Gogte wrote:
> >
> > Recently, blktests-CI runs for the kernel at block/for-next branch started
> > reporting failures of the test case nvme/004 (with tcp transport) and md/001.
> > The kernel reported a WARN related to lock->magic, and it caused the failures.
> > FYI, here I share the failure run log at md/001 [1] and dmesg at the failure.
> >
> > The md/001 confirms a bug fix in md driver. The bug was triggered when a
> > nvme device with tcp transport is used as its backend. Then the test case
> > exercises the nvme-tcp driver, which this patch touches.
> >
> > I bisected and found that this commit is the trigger of the failures. When I
> > revert this commit from the block/for-next branch tip with git hash
> > 34fbfb9d8d1f, the failure goes away.
> >
> > Surabhi, could you take a look in the failure?
> >
>
> Yes, there is a bug in commit 61b0ed3be81b ("nvme-tcp: parallelize I/O
> queue allocation and startup"). To know the queue range before the
> queues are set up, it moved the I/O tag set allocation ahead of I/O
> queue allocation. blk_mq_alloc_tag_set() calls
> nvme_tcp_init_request() for every request, which takes pf_cache_lock
> of the request's queue, but nvme_tcp_alloc_queue() only initializes
> that mutex later:
> nvme_tcp_configure_io_queues()
> nvme_alloc_io_tag_set()
> blk_mq_alloc_tag_set()
> blk_mq_alloc_rqs()
> nvme_tcp_init_request() mutex_lock(pf_cache_lock) <- WARN
> nvme_tcp_setup_io_queues()
> nvme_tcp_alloc_queue() mutex_init(pf_cache_lock)
>
> So with CONFIG_DEBUG_MUTEXES the first connect trips
> DEBUG_LOCKS_WARN_ON(lock->magic != lock) in __mutex_lock().
> That matches the call trace in your dmesg.
>
> Daniel Wagner posted a fix for this as patch 1 of [PATCH 0/5]
> nvme-fabrics: fix connects with isolated CPUs and queue mapping bugs:
> lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-0-355156be27b2@xxxxxxxxxx
>
> It initializes pf_cache_lock for all queues in nvme_tcp_alloc_ctrl(),
> right after the queues array is allocated, and drops the
> mutex_destroy() calls so the lock stays valid for the lifetime of the
> controller.
>
> I ran blktests with his fix on top of block/for-next (34fbfb9d8d1f, the
> same tip you bisected on) and md/001 passes with no WARN related to
> lock->magic. Without the fix, the same tree and config fails md/001
> with the WARN at kernel/locking/mutex.c:625.
>
> With the fix : nvme-tcp: initialize pf_cache_lock when allocating the controller
> lore : https://lore.kernel.org/all/20261014-nvme-fabrics-isolcpu-v1-1-355156be27b2@xxxxxxxxxx/
>
> ~/blktests$ sudo ./check md/001
> md/001 (Raid with bitmap on tcp nvmet with opt-io-size over bitmap
> size) [passed]
> runtime 1.779s ... 1.814s
Surabhi, thanks for the confirmation. It's great taht The md/001 failure is
gone. Said that, now I face failrues of other test cases: nvme/062 and 063.
While I was running blktests on v7.4-rc1 and v7.4-rc2 kernel, I observed the two
test cases nvme/062 and 063 fail for tcp transport [*]. I bisected and found
that this patch as the commit 61b0ed3be81b is the trigger. I observed the
failure with nvme-cli versions 3.0, 3.1 and 3.2. ktls-utils version is 1.4.0.
When I reported blktests failures for v7.4-rc1 kernel [#], the failure was
not observed by some reason. But now the failure is observed in stable manner.
[#] https://lore.kernel.org/linux-block/avBz7uOGYo2y426_@shinmob/
It looks like nvme connect can fail for tls connection with multiple I/O
queues. I created a blktests patch to limit number of I/O quees to 1 [**],
and observed the failure goes away. But I'm not sure if this is just hiding
the failure.
Are the failures observed in your environment?
[*] nvme/062 and nvme/063 failure
nvme/062 (tr=tcp) (Create TLS-encrypted connections) [failed]
runtime 4.678s ... 13.619s
--- tests/nvme/062.out 2026-11-09 20:12:01.056000000 +0900
+++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad 2026-11-15 20:21:09.213000000 +0900
@@ -1,6 +1,10 @@
Running nvme/062
Test unencrypted connection w/ tls not required
Test encrypted connection w/ tls not required
+FAIL: nvme connect return error code
+WARNING: connection is not encrypted
Test unencrypted connection w/ tls required (should fail)
Test encrypted connection w/ tls required
...
(Run 'diff -u tests/nvme/062.out /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/062.out.bad' to see the entire diff)
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation)
runtime 3.123s ...
WARNING: Test did not clean up port: 0
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation) [failed]
runtime 3.123s ... 7.313sost: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
--- tests/nvme/063.out 2026-11-09 20:12:01.060000000 +0900
+++ /home/shin/Blktests/blktests/results/nodev_tr_tcp/nvme/063.out.bad 2026-11-15 20:21:30.207000000 +0900
@@ -1,5 +1,4 @@
Running nvme/063
Test secure concatenation with SHA256
-Reset controller
-Test secure concatenation with SHA384
-Test complete
+FAIL: nvme connect return error code
+WARNING: connection failed
[**] trial blktests patch
diff --git a/tests/nvme/062 b/tests/nvme/062
index eabe2d9..69a05b6 100755
--- a/tests/nvme/062
+++ b/tests/nvme/062
@@ -55,7 +55,7 @@ test() {
# Test encrypted connection
echo "Test encrypted connection w/ tls not required"
- _nvme_connect_subsys --tls
+ _nvme_connect_subsys --tls --nr-io-queues 1
ctrl=$(_find_nvme_dev "${def_subsysnqn}")
if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then
@@ -77,7 +77,7 @@ test() {
# Test encrypted connection
echo "Test encrypted connection w/ tls required"
- _nvme_connect_subsys --tls
+ _nvme_connect_subsys --tls --nr-io-queues 1
ctrl=$(_find_nvme_dev "${def_subsysnqn}")
if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null; then
diff --git a/tests/nvme/063 b/tests/nvme/063
index feb07bf..900bb9f 100755
--- a/tests/nvme/063
+++ b/tests/nvme/063
@@ -46,7 +46,7 @@ test() {
_set_nvmet_dhgroup "${def_hostnqn}" "ffdhe2048"
echo "Test secure concatenation with SHA256"
- _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
+ _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
ctrl=$(_find_nvme_dev "${def_subsysnqn}")
if [[ -z "$ctrl" ]]; then
@@ -88,7 +88,7 @@ test() {
_set_nvmet_dhgroup "${def_hostnqn}" "ffdhe3072"
echo "Test secure concatenation with SHA384"
- _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat
+ _nvme_connect_subsys --dhchap-secret "${hostkey}" --concat --nr-io-queues 1
ctrl=$(_find_nvme_dev "${def_subsysnqn}")
if ! _nvme_ctrl_tls_key "$ctrl" > /dev/null ; then