Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling

[changfengnan]

> From: "Anuj Gupta"<anuj1072538@xxxxxxxxx>
> Date:  Thu, Aug 13, 2026, 01:50
> Subject:  Re: [RESEND RFC PATCH v2] nvme-pci: add adaptive interrupt polling
> To: "Fengnan Chang"<changfengnan@xxxxxxxxxxxxx>
> Cc: <kbusch@xxxxxxxxxx>, <axboe@xxxxxxxxx>, <hch@xxxxxx>, <sagi@xxxxxxxxxxx>, <linux-nvme@xxxxxxxxxxxxxxxxxxx>, <linux-kernel@xxxxxxxxxxxxxxx>, "Guzebing"<guzebing@xxxxxxxxxxxxx>
> On Thu, Aug 06, 2026 at 11:10:58AM +0800, Fengnan Chang wrote:
> > Each Solidigm SB5PH27X038T device used for testing can deliver about 3.2M
> > 4 KiB random-read IOPS.  Four of them should be good for about 12.8M IOPS,
> > but interrupt-driven completion tops out at 5.59M, only about 44% of that.
> > 
> > Polling gets rid of that cost, but polling every queue all the time burns
> > CPU and hurts the sparse or bursty queues that interrupts handle just fine.
> > So instead of a global switch, let each queue make the call on its own,
> > from how fast it has been completing lately, and re-check often enough that
> > the decision tracks the workload rather than a fixed tunable.
> > 
> > Each queue runs a small loop with three stages.  First it samples its
> > completion rate while still on interrupts.  Only if that rate is high
> > enough to fill a small batch inside a bounded latency window does it mask
> > its own IRQ and start draining the CQ from a high-resolution timer, with
> > each wait sized to collect roughly one batch.  It keeps polling as long as
> > it keeps up with that rate; the moment it stalls or slows down it turns the
> > IRQ back on and backs off, waiting longer the further behind it fell.
> > A queue that doesn't benefit drops back quickly and only gets retried once
> > in a while, so polling stays on the queues that are actually
> > interrupt-bound and everything else keeps running on the untouched IRQ
> > path.
> > 
> > Measured with 4 KiB random reads on Solidigm SB5PH27X038T, adaptive on
> > versus off:
> > 
> >                                    QD32      QD64      QD128
> >   one device, one job             +18.44%   +24.35%   +26.38%
> >   four devices, eight jobs        +83.76%   +99.02%   +96.26%
> >
> 
> Did a quick test on single NVMe with QD64, one job, 4K random reads via
> io_uring, I see ~23% improvement: 569K -> 701K IOPS.
> 
> Do you expect the fixed batch size, sample size, maximum delay, episode
> length, and backoff multiplier to work across devices with different
> latency profiles and across different workloads, or should these values
> somehow adapt to observed queue behavior?

Thank you for sharing the data. My goal is for these parameters to be
applicable to all disk and workload scenarios, but more testing is needed
in practice.
I’ve made some adjustments to the parameters and logic in the version
I’m currently developing, and it looks pretty good so far.

Attached is the version I'm currently developing. If you have time to take
a look or test it and offer some feedback, I'd really appreciate it.

> 
> > +/*
> > + * Stop polling and turn the queue's IRQ back on.  @elapsed is how long the
> > + * episode ran after it started falling behind, or 0 if it ended cleanly.
> > + * The bigger @elapsed is, the more completions we missed, and the longer we
> > + * wait before sampling this queue again, so a queue that polling doesn't
> > + * help is left alone most of the time.
> > + */
> 
> @elapsed is the total polling-episode duration, not the time since the
> queue fell behind. It is then used with the total completion count to
> estimate cumulative deficit. Could the comment be reworded accordingly?

Sorry, there are some issues with the comments in this version. I'll fix them
in the next version.

> 
> > +/*
> > + * Called from the IRQ handler after a reap that found something.  If we're
> > + * still in backoff, just count it down.  Otherwise time how long
> > + * NVME_ADAPTIVE_SAMPLE_CQES completions take to get the average gap between
> > + * them.  If that looks worth polling (see the filter below) mask the IRQ and
> > + * switch to poll mode; if not, leave the queue on interrupts.
> > + */
> > +static void nvme_adaptive_sample(struct nvme_queue *nvmeq,
> > +                                 unsigned int completions)
> > +{
> > +        struct nvme_adaptive_poll *adaptive = nvmeq->adaptive;
> > +        unsigned int sample;
> > +        unsigned long flags;
> > +        u64 delta, interval, now;
> > +
> > +        if (adaptive->retry_completions) {
> > +                if (adaptive->retry_completions != U64_MAX)
> > +                        adaptive->retry_completions -= min_t(u64, completions,
> > +                                                         adaptive->retry_completions);
> > +                return;
> > +        }
> > +        if (!adaptive->start_ns) {
> > +                adaptive->start_ns = ktime_get_ns();
> > +                return;
> > +        }
> > +        adaptive->completions += completions;
> > +        if (adaptive->completions < NVME_ADAPTIVE_SAMPLE_CQES)
> > +                return;
> > +
> > +        now = ktime_get_ns();
> > +        delta = now - adaptive->start_ns;
> > +        sample = adaptive->completions;
> > +        adaptive->start_ns = now;
> > +        adaptive->completions = 0;
> > +        if (!delta || delta > (u64)NVME_ADAPTIVE_SAMPLE_CQES *
> > +                               NVME_ADAPTIVE_MAX_DELAY_NS)
> > +                return;
> > +        /*
> > +         * Is this rate worth polling?  The 100/99 factor trims 1% off the gap so
> > +         * a queue sitting right on the threshold isn't pulled in.  Skip it if
> > +         * it's too slow to fill a batch within MAX_DELAY, and also skip it if
> > +         * it's already fast enough to batch by itself.  The hardware's own
> > +         * coalescing already handles that case, so leave it on interrupts.
> > +         */
> > +        interval = div64_u64(delta * 100, sample * 99);
> > +        if (!interval || interval > NVME_ADAPTIVE_MAX_DELAY_NS ||
> 
> The comment says this rejects queues that cannot fill a target batch
> within MAX_DELAY, but this condition only checks whether one completion
> interval exceeds MAX_DELAY. Should this instead account for TARGET_BATCH
> or this comment is describing a different policy?

TARGET_BATCH will no longer be used in the next version.

> 
> > +            (delta > NSEC_PER_MSEC &&
> > +             interval <= NVME_ADAPTIVE_MAX_DELAY_NS /
> > +                         NVME_ADAPTIVE_TARGET_BATCH))
> > +                return;
> > +
> 
> For a normal 256-CQE sample with an interval of 2us or less, delta will
> be about 512us or less, so the 1ms condition prevents this exclusion
> from firing. Is this intended to detect only samples that substantially
> overshoot 256 CQEs? If so, could the comment make that narrower intent
> explicit?

I did it that way before because I thought that if the interrupt mode could
process a CQE in 2 μs, that would mean the interrupt mode was already very fast.
 I’ve removed that check from the version I’m currently developing.
>

Attachment: 0001-nvme-pci-add-adaptive-interrupt-polling.patch
Description: Binary data