[RFC PATCH 1/2] nvme-tcp: remove the direct send path
From: Seiji Nishikawa
Date: Sat Nov 21 2026 - 20:20:09 EST
nvme_tcp_queue_request() sends the request directly from the submitting
context when it runs on the io_cpu of the queue, the queue is empty and
it can take send_mutex. Commit db5ad6b7f8cd ("nvme-tcp: try to send
request in queue_rq context") added this to skip the workqueue hop when
the queue is idle.
Since then this path has needed a series of fixes for races and
priority inversions between it and io_work. These are
commit 5c11f7d9f843 ("nvme-tcp: Fix possible race of io_work and direct
send"), commit a0fdd1418007 ("nvme-tcp: rerun io_work if req_list is
not empty"), commit 70f437fb4395 ("nvme-tcp: fix io_work priority
inversion"), commit 08acb54b063a ("nvme-tcp: defer TLS inline send to
io_work") and commit 5db358f3a7f9 ("nvme-tcp: fix io_work priority
inversion when send_mutex is held").
All of them come from having 2 contexts that send on the same queue and
coordinate through send_mutex, while io_work runs on a high priority
workqueue and the submitting task does not.
Remove the direct send path and always send from io_work. TLS queues
already work this way since 08acb54b063a. With a single sender, io_work
has to be queued whenever the last request of a batch is added, so the
has_pending check in nvme_tcp_queue_request() goes away too, and
nvme_tcp_send_all() has no caller left.
Suggested-by: Hannes Reinecke <hare@xxxxxxx>
Assisted-by: LLM
Signed-off-by: Seiji Nishikawa <snishika@xxxxxxxxxx>
---
drivers/nvme/host/tcp.c | 33 ++-------------------------------
1 file changed, 2 insertions(+), 31 deletions(-)
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 62a594b8ab9b..2b832ac3068a 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -213,7 +213,6 @@ struct nvme_tcp_ctrl {
static struct workqueue_struct *nvme_tcp_wq;
static const struct blk_mq_ops nvme_tcp_mq_ops;
static const struct blk_mq_ops nvme_tcp_admin_mq_ops;
-static int nvme_tcp_try_send(struct nvme_tcp_queue *queue);
#ifdef CONFIG_DEBUG_LOCK_ALLOC
/* lockdep can detect a circular dependency of the form
@@ -420,16 +419,6 @@ static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req,
}
}
-static inline void nvme_tcp_send_all(struct nvme_tcp_queue *queue)
-{
- int ret;
-
- /* drain the send queue as much as we can... */
- do {
- ret = nvme_tcp_try_send(queue);
- } while (ret > 0);
-}
-
static inline bool nvme_tcp_queue_has_pending(struct nvme_tcp_queue *queue)
{
return !list_empty(&queue->send_list) ||
@@ -446,28 +435,10 @@ static inline void nvme_tcp_queue_request(struct nvme_tcp_request *req,
bool last)
{
struct nvme_tcp_queue *queue = req->queue;
- bool empty;
- empty = llist_add(&req->lentry, &queue->req_list) &&
- list_empty(&queue->send_list) && !queue->request;
-
- /*
- * if we're the first on the send_list and we can try to send
- * directly, otherwise queue io_work. Also, only do that if we
- * are on the same cpu, so we don't introduce contention.
- *
- * TLS kTLS send takes ctx->tx_lock while blk_mq holds set->srcu.
- * lockdep reports circular locking via elevator_lock. Defer TLS
- * sends to the io workqueue instead of inline from this path.
- */
- if (queue->io_cpu == raw_smp_processor_id() &&
- !nvme_tcp_queue_tls(queue) &&
- empty && mutex_trylock(&queue->send_mutex)) {
- nvme_tcp_send_all(queue);
- mutex_unlock(&queue->send_mutex);
- }
+ llist_add(&req->lentry, &queue->req_list);
- if (last && nvme_tcp_queue_has_pending(queue))
+ if (last)
queue_work_on(queue->io_cpu, nvme_tcp_wq, &queue->io_work);
}
--
2.56.1