Re: [BUG] RCU hang with io_uring nvme polling

[Ben Carey]

On Sat, Jul 4, 2026 at 1:01 PM Keith Busch <kbusch@xxxxxxxxxx> wrote:
>
> What exactly do you have in mind? Shouldn't you expect to spend most of
> your CPU time in the polling loop? As long as you keep the queues busy,
> there's something to poll, so blk_hctx_poll is exactly where you want to
> see the software be in a perf report. Seeing a high poll CPU utilization
> means the software is efficient compared to the hardware. If we spend
> very little time in the polling loop, then either you have incredibly
> quick hardware, and let's face it, Optane SSDs are EOL and a generation
> behind on link speeds so that's not gonna get there anymore, or our
> software dispatch stack has an inefficiency somewhere.

This is a fair point and I also think exposes some flaws with using perf
runtime reports for justifying a fix.

I'm most definitely not qualified to suggest this as a passable alternative,
but when polling a tagset, is there a way to check if the tagset's been
completed by another thread? Maybe break out if, for each polled request,
request->state == MQ_RQ_COMPLETE? I'm unsure how to translate the parameters in
blk_hctx_poll into the set of requests being waited on.

> If you have many pollers competing against a very low utilized queue,
> then I think you have an application level problem mismatched to the
> feature.

You're right, the test case above doesn't give a fair representation of the
issue.

> The only thing the jiffie timeout may show a problem is when you stop
> dispatching, which should only affect the time to close the ring when it
> lost the polling race with a peer on the last IO it is looking for, but
> should not affect individual IO latency.

We measured the disk latency usage with a simple kernel patch and confirmed
that individual IO latency is not impacted by the timeout issue.

We've seen, however, that the timeout can occur a large number of times even
with high queue saturation. When running the fio job below we observed the
timeout 132757 times, which I'm concerned could negatively impact bandwidth.

fio --bs=128K --direct=1 --iodepth=256 --runtime=200 --rw=randread \
    --time_based \
  --ioengine=io_uring --hipri=1 --fixedbufs=0 --registerfiles=0 \
    --sqthread_poll=0 \
  --numjobs=32 --name=job0 --output-format=json --clocksource=clock_gettime \
  --filename=/dev/nvme0n1