Re: [PATCH] nvme: reject completions for requests that are not in flight
From: Mauricio Faria de Oliveira
Date: Tue Nov 03 2026 - 14:42:11 EST
On 2026-11-03 16:21, K wrote:
> Thanks, pls let me take a look, it looks like all AI generated. May I
AI has not been used for analysis or the commit message; just the
reproducer, as mentioned.
Please check git log for previous contribution history.
If you're not comfortable with the commit message, I can certainly
change the authorship.
Please just let me know your preference (e.g., Originally-by,
Co-developed-by, name in the commit message, or none).
> know what kind of test you run or LLM run?
Sure. It's a synthetic reproducer that runs the completion handler
again.
Code: https://termbin.com/40tl
In a VM with 1 vCPU, (so nvme_handle_cqe() -> nvme_try_complete_req() ->
blk_mq_complete_request_remote() returns false),
echo mq-deadline >/sys/block/nvme0n1/queue/scheduler # (so
nvme_handle_cqe() -> blk_mq_add_to_batch() also returns false),
# with both returning false, nvme_handle_cqe() ->
nvme_pci_complete_rq() is taken, reaching __blk_mq_free_request(), which
does rq->mq_hctx = NULL
echo 1 >/sys/module/nvme/parameters/repro_arm # prepare reproducer
dd if=/dev/nvme0n1 of=/dev/null bs=4k count=1 iflag=direct # oops
cheers,
>
> Regards,
> Chao
>
> On Tue, Nov 3, 2026 at 2:17 PM Mauricio Faria de Oliveira
> <mfo@xxxxxxxxxx> wrote:
>>
>> From: Chao Shi <coshi036@xxxxxxxxx>
>>
>> There are a few kernel crash reports from the Steam Deck where incorrect
>> completions received by the nvme driver have a command id/tag associated
>> with a freed request (rq->mq_hctx = NULL), dereferencing a NULL pointer.
>>
>> nvme_handle_cqe()
>> -> nvme_try_complete_req(req, ...)
>> -> blk_mq_complete_request_remote(req)
>> -> if ((rq->mq_hctx->nr_ctx == 1 && /* Oops! */
>>
>> Although the usual suspect is the nvme device firmware, one report shows
>> incorrect completions also received by the ath11k wireless driver at the
>> same time, which suggests the underlying issue might be something else.
>>
>> [ 2585.223155] ath11k_pci 0000:03:00.0: tx completion for unknown msdu_id 0
>> [ 2585.223163] ath11k_pci 0000:03:00.0: tx completion for unknown msdu_id 0
>> ...
>> [ 2585.228077] nvme nvme0: request 0xec genctr mismatch (got 0x2 expected 0xa)
>> [ 2585.228079] nvme nvme0: invalid id 8428 completed on queue 3
>> [ 2585.228205] nvme nvme0: request 0xe2 genctr mismatch (got 0xe expected 0x2)
>> [ 2585.228208] nvme nvme0: invalid id 57570 completed on queue 3
>> ...
>> [ 2585.228726] BUG: kernel NULL pointer dereference, address: 00000000000000fe
>> ...
>> [ 2585.228752] RIP: 0010:blk_mq_complete_request_remote+0x1c/0x160
>> ...
>> [ 2585.228787] Call Trace:
>> [ 2585.228790] <IRQ>
>> [ 2585.228828] nvme_poll_cq+0x14b/0x300
>> [ 2585.228838] nvme_irq+0x41/0x80
>> [ 2585.228845] __handle_irq_event_percpu+0x4a/0x180
>> [ 2585.228850] handle_irq_event+0x38/0x90
>> [ 2585.228853] handle_edge_irq+0x8b/0x230
>> [ 2585.228858] __common_interrupt+0x3e/0x90
>> [ 2585.228862] common_interrupt+0x80/0xa0
>> [ 2585.228866] </IRQ>
>> [ 2585.228868] <TASK>
>> ...
>>
>> However, even when the nvme device firmware is at fault and need an update,
>> it is interesting for the kernel not to crash, so that it is even possible
>> to run the firmware update where the issue happens frequently or early on,
>> and while waiting for the update to be made available by the device vendor.
>>
>> The fix is to check that a request is actually in-flight before completing
>> it. This has been previously reported and proposed by Chao Shi, found with
>> fuzzing, but without further feedback since July. I happened to learn of it
>> only after root-cause analysis and reaching a similar conclusion, checking
>> for prior work/feedback, thus keeping them as Author/Signed-off-by, adding
>> myself as Co-developed-by for the commit message and root cause analysis.
>>
>> In Chao Shi's patch, there's been feedback/question of whether this should
>> be done in blk-mq, as this is not specific to nvme. I'd say no, at first:
>> - Some blk-mq calls for completion have no return value (e.g., completion
>> on other CPUs), which suggests maybe an API change and updates to many
>> callers, all for an edge case.
>> - An alternative available currently might be blk_mq_ops.complete(), but
>> that is handled in the driver anyway, with risk of hitting an error in
>> blk-mq (like this) as the code changes, and seems semantically wrong,
>> i.e., completing a request that should not have been completed.
>> - Keeping in mind a small fix for stable, for now, should this be merged.
>> - But happy to take suggestions if blk-mq is indeed the way to go.
>>
>> This has been tested with a synthetic reproducer produced by an LLM.
>>
>> Originally-by: Chao Shi <coshi036@xxxxxxxxx>
>> Signed-off-by: Chao Shi <coshi036@xxxxxxxxx>
>> Link: https://lore.kernel.org/linux-nvme/20260522153034.2168862-1-coshi036@xxxxxxxxx/
>> Link: https://lore.kernel.org/linux-nvme/20260715005105.1115958-1-coshi036@xxxxxxxxx/
>> Co-developed-by: Mauricio Faria de Oliveira <mfo@xxxxxxxxxx>
>> Signed-off-by: Mauricio Faria de Oliveira <mfo@xxxxxxxxxx>
>> ---
>> drivers/nvme/host/nvme.h | 5 +++++
>> 1 file changed, 5 insertions(+)
>>
>> diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
>> index 2d8f681079e85601a92624d196e77f1ba3ace34a..3ba0666da7be6a3de2e5f0fd50155af115b4e1f6 100644
>> --- a/drivers/nvme/host/nvme.h
>> +++ b/drivers/nvme/host/nvme.h
>> @@ -708,6 +708,11 @@ static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,
>> tag, genctr, nvme_genctr_mask(nvme_req(rq)->genctr));
>> return NULL;
>> }
>> + if (unlikely(blk_mq_rq_state(rq) != MQ_RQ_IN_FLIGHT)) {
>> + dev_err(nvme_req(rq)->ctrl->device,
>> + "completion for request %#x not in flight\n", tag);
>> + return NULL;
>> + }
>> return rq;
>> }
>>
>>
>> ---
>> base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
>> change-id: 20261102-nvme-double-cmpl-uaf-blk-mq-e3d28f74cdaf
>>
>> Best regards,
>> --
>> Mauricio Faria de Oliveira <mfo@xxxxxxxxxx>
>>
--
Mauricio