Re: [PATCH v1 1/1] nvme-pci: adaptive interrupt coalescing

[Fengnan Chang]

Hi all,

I'd like to add some information. A few months ago, when I was
testing Zeng's solution, I encountered some issues, primarily that
enabling or disabling interrupt aggregation on certain disks could
cause I/O latency of several tens of milliseconds. 

Inspired by the design of network NAPI, I came up with a similar idea
and implemented a patch, which performed well in testing.
If you're interested in this approach, I can continue to improve it.
The reason I didn't move forward with this approach is that our
initial goal was to solve the issues encountered with QEMU, but
disabling interrupts causes problems in VFIO+QEMU environments, and
this approach only works in physical machine scenarios. 

The patch estimates how busy a queue is from the submission queue tail
and the SQ head reported in completion entries.  When the number of
pending commands crosses a high watermark, the hard IRQ handles the
current completions, masks the queue's interrupt and wakes the IRQ
thread.  The IRQ thread then polls for more completions.  It switches
back to interrupts when the pending count falls below a low watermark,
or when a bounded number of polling loops is reached.  Using separate
high and low watermarks avoids switching back and forth too often.

This is only enabled when the controller has separate MSI-X vectors, so
masking one queue does not block completions from other queues.

I tested it with four Solidigm NVMe devices using 4K random reads.  fio
used regular io_uring, without hipri or SQ polling.  These are the
median results from three runs on a 7.2-rc1 based kernel:

                         baseline    patched
  IOPS                   7.61M       11.41M  (+50%)
  mean latency           267 us      177 us  (-34%)
  hard IRQs per I/O      0.52        0.21    (-60%)

Without the patch, three of the four devices were limited to about
1.49M IOPS each.  With the patch, all four devices reached around
2.8M-2.9M IOPS.

The patch is included below. 

Thanks,
Fengnan