[PATCH v2 0/2] nvme-pci: add adaptive interrupt polling

[Fengnan Chang]

At high IOPS, interrupt handling can limit throughput, especially when
multiple drives contend for CPU time.

Add an opt-in adaptive policy for I/O queues that use a dedicated MSI-X
vector and do not use threaded interrupts. Each queue independently
switches between interrupt and poll mode based on its recent completion
rate.

In interrupt mode, wait for at least 8192 completions. If they arrive
no more than 10 us apart on average, try polling. While polling, an
hrtimer runs irq_poll to drain the completion queue. Before a window
is full, polling may fall 20 us behind the interrupt-mode pace. After
8192 completions, polling must be strictly faster than interrupt mode.

If a try fails, go back to interrupt mode and measure again. After
three failures in a row, stay in interrupt mode for 64 windows of
completions before measuring again. If polling works, keep it for at
most 64 windows, then go back to interrupt mode and measure again.

The module parameter is off by default and only sets the initial
mode for new controllers. A per-controller sysfs file can change
the mode later. The change freezes the namespace queues and waits
for in-flight I/O to finish. Queues that use threaded, MSI, or
legacy interrupts keep the regular interrupt handler.

Measured with 4 KiB random reads on Solidigm SB5PH27X076T, adaptive
polling on versus off:
                                     QD32      QD64      QD128
one device, one job                 -0.08%    +31.56%   +30.95%
fifteen devices, eight jobs/device  +188.03%  +236.46%  +231.38%

The fifteen-device aggregate with eight jobs per device increases
from 13.48M / 13.47M / 13.47M IOPS to 38.83M / 45.31M / 44.64M IOPS.

Changes since v1:
- Split the nvme_poll_cq() return-count change into a separate prep patch
  (patch 1/2) so the functional change stands on its own.
- Factor out nvme_irq_complete_batch() so the regular and adaptive
  interrupt handlers share one batch-completion path.
- Only call irq_poll_complete() when completions < budget, matching the
  irq_poll contract; the previous boundary case could return the poll
  object to the softirq list after it was already removed.
- Clarify comments: document the NVMEQ_ADAPTIVE_* flags and the
  dual-purpose retry_completions field, and reword the constant rationale.
- Minor style/indentation fixes per review.


Fengnan Chang (2):
  nvme-pci: return completion count from nvme_poll_cq
  nvme-pci: add adaptive interrupt polling

 Documentation/ABI/testing/sysfs-nvme |  17 +
 drivers/nvme/host/Kconfig            |   1 +
 drivers/nvme/host/pci.c              | 542 +++++++++++++++++++++++++--
 3 files changed, 536 insertions(+), 24 deletions(-)

-- 
2.39.5 (Apple Git-154)