Re: [PATCH v4] fuse: virtio-fs: fix lost wakeup on a full virtqueue

From: Li Wang

Date: Tue Nov 17 2026 - 07:19:27 EST


Hi Hang,

On 13/11/2026 17:17, Qihang wrote:
> When virtio_fs_enqueue_req() returns -ENOSPC, completion processing
> can run before the caller adds the request to queued_reqs. If no
> further completions arrive, dispatch is never scheduled and the
> request remains counted in in_flight, hanging queue draining.
>
> Track completion progress under fsvq->lock and sample it when
> submission fails with -ENOSPC. After queueing, schedule dispatch if
> the sequence changed; otherwise completion processing will see the
> queued request and schedule it. Each failed submission keeps its own
> snapshot, so concurrent submitters cannot consume each other's wakeup.
>
> Fixes: 106e4df1206b ("virtio-fs: improved request latencies when Virtio queue is full")
> Cc: stable@xxxxxxxxxxxxxxx # 6.11+
> Link: https://lore.kernel.org/all/sashiko-outbox-204365@xxxxxxxxxx/
> Signed-off-by: Qihang <q.h.hack.winter@xxxxxxxxx>
> ---
> Changes since v3:
>
> - Only the lost-wakeup fix is included here. v3 also carried teardown
> admission control; that is a separate change, not part of this patch,
> and nothing here depends on it.
> - Add a completion sequence sampled per failed submission to close the
> window between -ENOSPC and queue insertion.
>
> Runtime tests used an arm64 kernel with the separate teardown
> admission change also applied. The request scenarios ran before
> admission was stopped, with fsvq->aborting remaining false. The
> standalone patch was not run on its own.
>
> - Filled the request virtqueue and held the submitter between -ENOSPC and
> queue insertion while real device completions ran. The unpatched
> kernel hung in virtio_fs_drain_queue(); this change finished the drain.
> - Held the dispatch worker in the same place. The unpatched kernel
> stranded the request; this change rescheduled dispatch once more and
> every request finished.
> - Kept the queue full without completions: dispatch did not reschedule
> itself in a loop.
>
> fs/fuse/virtio_fs.c | 29 ++++++++++++++++++++++++-----
> 1 file changed, 24 insertions(+), 5 deletions(-)
>
> diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c
> index e445398..68aeaa6 100644
> --- a/fs/fuse/virtio_fs.c
> +++ b/fs/fuse/virtio_fs.c
> @@ -55,6 +55,7 @@ struct virtio_fs_vq {
> struct work_struct dispatch_work;
> struct fuse_dev *fud;
> bool connected;
> + u64 completion_seq; /* completion progress under lock */
> long in_flight;
> struct completion in_flight_zero; /* No inflight requests */
> struct kobject *kobj;
> @@ -99,7 +100,7 @@ struct virtio_fs_req_work {
>
> static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq,
> struct fuse_req *req, bool in_flight,
> - gfp_t gfp);
> + gfp_t gfp, u64 *completion_seq);
>
> static const struct constant_table dax_param_enums[] = {
> {"always", FUSE_VDAX_ALWAYS },
> @@ -348,6 +349,7 @@ static void virtio_fs_start_all_queues(struct virtio_fs *fs)
> fsvq = &fs->vqs[i];
> spin_lock(&fsvq->lock);
> fsvq->connected = true;
> + fsvq->completion_seq = 0;
> spin_unlock(&fsvq->lock);
> }
> }
> @@ -576,6 +578,7 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work)
> /* Dispatch pending requests */
> while (1) {
> unsigned int flags;
> + u64 completion_seq;
>
> spin_lock(&fsvq->lock);
> req = list_first_entry_or_null(&fsvq->queued_reqs,
> @@ -588,12 +591,15 @@ static void virtio_fs_request_dispatch_work(struct work_struct *work)
> spin_unlock(&fsvq->lock);
>
> flags = memalloc_nofs_save();
> - ret = virtio_fs_enqueue_req(fsvq, req, true, GFP_KERNEL);
> + ret = virtio_fs_enqueue_req(fsvq, req, true, GFP_KERNEL,
> + &completion_seq);
> memalloc_nofs_restore(flags);
> if (ret < 0) {
> if (ret == -ENOSPC) {
> spin_lock(&fsvq->lock);
> list_add_tail(&req->list, &fsvq->queued_reqs);
> + if (completion_seq != fsvq->completion_seq)
> + schedule_work(&fsvq->dispatch_work);
> spin_unlock(&fsvq->lock);
> return;
> }
> @@ -902,6 +908,8 @@ static void virtio_fs_requests_done_work(struct work_struct *work)
>
> /* Try to push previously queued requests, as the queue might no longer be full */
> spin_lock(&fsvq->lock);
> + /* Record progress for submitters not yet on queued_reqs. */
> + fsvq->completion_seq++;
> if (!list_empty(&fsvq->queued_reqs))
> schedule_work(&fsvq->dispatch_work);
> spin_unlock(&fsvq->lock);
> @@ -1418,10 +1426,15 @@ static unsigned int sg_init_fuse_args(struct scatterlist *sg,
> return total_sgs;
> }
>
> -/* Add a request to a virtqueue and kick the device */
> +/*
> + * Add a request to a virtqueue and kick the device. On -ENOSPC, return the
> + * completion sequence under the same lock as the failed submission. If it
> + * changes before the caller queues the request, the caller must schedule
> + * dispatch itself: done_work may have run before the request was queued.
> + */
> static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq,
> struct fuse_req *req, bool in_flight,
> - gfp_t gfp)
> + gfp_t gfp, u64 *completion_seq)
> {
> /* requests need at least 4 elements */
> struct scatterlist *stack_sgs[6];
> @@ -1488,6 +1501,8 @@ static int virtio_fs_enqueue_req(struct virtio_fs_vq *fsvq,
> vq = fsvq->vq;
> ret = virtqueue_add_sgs(vq, sgs, out_sgs, in_sgs, req, GFP_ATOMIC);
> if (ret < 0) {
> + if (ret == -ENOSPC)
> + *completion_seq = fsvq->completion_seq;
> spin_unlock(&fsvq->lock);
> goto out;
> }
> @@ -1529,6 +1544,7 @@ static void virtio_fs_send_req(struct fuse_iqueue *fiq, struct fuse_req *req)
> unsigned int queue_id;
> struct virtio_fs *fs;
> struct virtio_fs_vq *fsvq;
> + u64 completion_seq;
> int ret;
>
> fuse_request_assign_unique(fiq, req);
> @@ -1545,7 +1561,8 @@ static void virtio_fs_send_req(struct fuse_iqueue *fiq, struct fuse_req *req)
> queue_id);
>
> fsvq = &fs->vqs[queue_id];
> - ret = virtio_fs_enqueue_req(fsvq, req, false, GFP_ATOMIC);
> + ret = virtio_fs_enqueue_req(fsvq, req, false, GFP_ATOMIC,
> + &completion_seq);
> if (ret < 0) {
> if (ret == -ENOSPC) {
> /*
> @@ -1555,6 +1572,8 @@ static void virtio_fs_send_req(struct fuse_iqueue *fiq, struct fuse_req *req)
> spin_lock(&fsvq->lock);
> list_add_tail(&req->list, &fsvq->queued_reqs);
> inc_in_flight_req(fsvq);
> + if (completion_seq != fsvq->completion_seq)
> + schedule_work(&fsvq->dispatch_work);
> spin_unlock(&fsvq->lock);
> return;
> }

Regarding handling -ENOSPC, it looks like we can simplify things similar to
how send_forget_request() does it:

If -ENOSPC is encountered while holding fsvq->lock, we can simply use
list_add_tail() to queue it into queued_reqs. For newly submitted requests,
we increment via inc_in_flight_req().

virtio_fs_send_req() and virtio_fs_request_dispatch_work() can just return
immediately upon seeing -ENOSPC, avoiding a second lock acquisition and re-queuing.

With this approach, we won't need completion_seq, per-call sequence snapshots,
or an extra scheduling step from the submitter side.

Thanks,
Li Wang