[RFC PATCH 0/2] nvme-tcp: remove the direct send path
From: Seiji Nishikawa
Date: Sat Nov 21 2026 - 20:19:44 EST
Hannes suggested removing the direct send path in
nvme_tcp_queue_request() while reviewing commit 5db358f3a7f9 ("nvme-tcp:
fix io_work priority inversion when send_mutex is held"). This series
does that, on top of nvme-7.4.
Patch 1 removes the direct send path, so io_work is the only sender.
Patch 2 removes send_mutex, which has nothing left to protect after
patch 1.
The direct send path was added to cut latency when the queue is idle,
so I measured it on two machines with 1Gb NICs. The target is nvmet-tcp
backed by null_blk (irqmode=0). The host runs fio like this, 60 seconds
per run and 3 runs per case:
fio --name=qd1 --filename=/dev/nvme0n1 --direct=1 --ioengine=libaio \
--bs=4k --rw=randread --iodepth=1 --numjobs=1 --cpus_allowed=2 \
--time_based --runtime=60 --ramp_time=5
Average of the 3 runs at queue depth 1, latency in usec, "patched" is
this series:
IOPS slat clat lat clat p99
randread base 4153 7.1 232.3 239.3 277
patched 4154 4.7 234.7 239.4 280
randwrite base 2472 24.5 375.6 400.1 553
patched 2460 14.3 387.6 402.0 570
The send time moves from slat to clat. With the direct send path,
io_submit() itself sends the command on the socket, so that time is
counted in slat. Without it, io_submit() only queues the request and
returns, and io_work sends it, so the same time is counted in clat. lat
is the sum of the two, and it changes by 0.1 usec for randread and 1.9
usec (+0.5%) for randwrite.
At queue depth 32 with 4 jobs, both kernels reach the 1Gb link limit
(28.2k IOPS), so there is no difference there.
On the same two machines, I also ran 5 controller resets during a 120
second randrw fio run on the patched kernel, and fio finished with no
errors. Separately, blktests nvme over tcp on a VM with an nvmet tcp
loopback target gives the same results with and without this series.
On a 1Gb network one I/O takes about 240 us, so a difference of a few
microseconds is easy to miss. Numbers from faster networks would show
whether it matters there. I am posting this as RFC so the performance
side can be discussed on the list.
Seiji Nishikawa (2):
nvme-tcp: remove the direct send path
nvme-tcp: remove send_mutex
drivers/nvme/host/tcp.c | 60 +++++++----------------------------------
1 file changed, 9 insertions(+), 51 deletions(-)
base-commit: 68099cf27253a8c89d703a282d3d811446d9453b
--
2.56.1