Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Dave Chinner]

On Sun, Aug 30, 2026 at 06:47:00PM -0600, Eric Peterson wrote:
> On Mon, Aug 31, 2026 at 07:26:52AM +1000, Dave Chinner wrote:
> > Hence this really doesn't seem like something we should be trying to
> > infer from indirect filesystem stats. Why can't you use the bdev
> > stats to get the actual filesystem wide queue depth information?
> 
> The block device measures the device queue, which is a different
> quantity than filesystem outstanding I/O - not just a lower-layer view
> of the same thing.
> 
> Below are three cases where filesystem queue depth is not what the block
> layer sees:

I do know the difference. Assume I understand what you are saying,
and that you don't need to explain how the IO stack works to me...

> To be clear about scope: I'm not proposing a queue-depth feature in
> the kernel. The change just adds read/write completion counters to pair
> with the existing call (submission) counters, so userspace can compute
> outstanding I/O and derive a response-time estimate itself. The kernel
> side is only exposing the complementary raw signal that's currently
> missing - calls are counted, completions are not.

I know, I just don't see how it can be used for a response time
metric that any way useful for behavioural correlation because of
the sampling method.

> Being upfront: what userspace derives from this is an instantaneous
> approximation, not a precise time-weighted queue length. It's meant as
> a cheap, always-on aggregate, not a replacement for accurate per-op
> tooling.

And that's exactly why I'm having trouble understanding how this
new metric means anything useful. Ignoring temporal sampling jitter
of multiple per-cpu counters, if you sample read + completions
it at some instant, all it tells you is what is happening at that
instant.

What happens the other 999.9ms of that second is not captured by
this new "in-flight" metric?  For example, if I sample read
submissions at 10Hz (annotated manually with rough deltas between
samples):

$ pmval -r -t 0.1 xfs.read

metric:    xfs.read
host:      devoid
semantics: cumulative counter
units:     count
samples:   all
  294454912
  294454912		S (0 IO in flight)
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294454912
  294455553	+650
  294455553		S (0 IO in flight)
  294455555	+2
  294455555
  294455555
  294455555
  294455555
  294455555
  294455555
  294455555
  294455559	+4
  294455559		S (0 IO in flight)
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559
  294455559		S (0 IO in flight)
  294455561	+2
  294455561
  294455561
  294455564
  294455564
  294456360	+800
  294457344	+1000
  294457344
  294457346	+2
  294457352	+6	S (at most 6 IO in flight)
  294457352
  294458065	+700
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285
  294458285		S (0 IO in flight)

You can see that there are some 100ms periods where nothing happens,
whilst others have 650-1000 buffered reads. In all the cases where
there are periods with no submission, the in-flight calculation will
be zero. In the busy periods, it will likely be some non-zero
number, but it won't give any indication of IO behaviour in that
entire period.

If we pick a 1s sample time (marked with "S" above), only one of
those sample points had any chance of there being IO in flight.
If I pick a sampling pattern that hits one of those high
IO periods, it gives an unrealisticly high in flight value for the
sampling period, given that for most of the rest of the second
around that burst there was almost no read activity.

Hence I don't see how sampling a point in time "in-flight" metric
slowly provides reliable insight into application behaviour. To
address that, one would need to sample and calculate the inflight
metric at high resolution to be able to catch the concurrency of IO
in those high IOPS bursts.

However, the faster you sample to catch bursts, the closer the read
submission rate approaches the in-flight IO rate. i.e. if I sample
at 1000Hz instead of 10Hz, it'll capture the fact that there are
bursts much faster bursts than 8-10 read IOs per millisecond, yet
the in-flight counter still won't reflect that - it might still not
register any IO being in flight at all because at the sample instant
there was no IO in flight....

Hence I'm asking how this new metric is supposed to be used and
correlated to observed/measured application behaviour. i.e. what
insight does it give you into application performance that can only
be derived from this point in time snapshot?

-Dave.
-- 
Dave Chinner
dgc@xxxxxxxxxx