Re: [PATCH] xfs: add per-mount read/write I/O completion counters

[Dave Chinner]

On Tue, Sep 01, 2026 at 11:32:30PM -0600, Eric Peterson wrote:
> On Mon, Aug 31, 2026 at 09:38 UTC, Dave Chinner wrote:
> > Hence I'm asking how this new metric is supposed to be used and
> > correlated to observed/measured application behaviour. i.e. what
> > insight does it give you into application performance that can only
> > be derived from this point in time snapshot?
> 
> My apologies - it wasn't my intention to come across as patronizing.
> I was unsure what background was or wasn't common ground, so I erred on
> the side of more detail.
> 
> You're right about the sampling limitation: a slowly-sampled
> point-in-time queue depth value cannot characterize bursty,
> sub-interval concurrency. If the goal is to resolve what happens inside
> a 10ms burst, this is the wrong tool - per-op tooling (tracepoints,
> histograms) is the right one, and this is not meant to replace it.
> 
> The important part is that this is a property of the sampling rate, not
> of the counters. Nyquist-Shannon says that to observe a phenomenon at
> timescale T you have to sample at >= 2/T; if you sample slower than the
> behavior you care about, it will be missed. This is true of any sampled
> counter, including the existing submission counter - in your 10Hz pmval
> example, xfs.read has exactly the same property. The sampling rate is a
> policy choice for the user to match to what they're trying to observe.

I know what nyquist sampling implies - that's exactly why I gave
that example to demonstrate how point in time sampling of
instantaneous values is not representative.

In more detail, the nyquist sampling theorem only holds when -every
sample is representative of the overall waveform-. You also need to
know what the highest frequency of the waveform is to be able to
reconstruct the behaviour.

The problem with using nyquist here is that the queue depth is an
instantaneous value metric - it is not a repeating waveform. Hence
nyquist says it is impossible to recover a valid signal from the
metric because the sample rate must be 2x instantaneous.

Something like a summing value (i.e. read count) does allow nyquist
theorem to be applied, because the delta between samples leads to a
meaningful waveform - the read rate per sample period. But you
cannot convert an instantaneous value calculated from summing
variables into a rate metric - it is always an instantaneous value
and that means the delta between samples is meaningless when placed
in a time series. i.e. there is no frequency component that nyquist
sampling theorem can recover from it.

Let's go further. Last email I said "ignoring per-cpu summing
jitter". Did you think about that at all?

Look at how the samples are presented to userspace:

	for each counter group {
		for each counter in group {
			for each cpu {
				sum counter
			}
			print counter val
		}
	}

Think about that for a moment. What happens when you have hundreds
of CPUs (just call it N)?

Yeah, summing each counter is an expensive operation, involving
accessing N cachelines for each counter. If each of those stats is
being actively modified whilst the sum is in progress, we take cache
miss on each CPU for each counter. Let's be charitable and call that
100ns per CPU. For a thousand CPUs, that means it takes 100us to
sample that counter.

Now, read completion was placed directly after reads, so there is a
window of 100us between the sampling of each per cpu value.

Consider that a buffered read could take as little as a few
microseconds to run. That means that across the sampling of read +
read completion, each CPU could start and complete multiple buffered
IOs.

In that situation, we have on a single CPU:

read 1 ....
sample read		X
read 1 complete
read 2 ....
read 2 complete
read 3 ....
read 3 complete
....
read N ....
sample read_complete	X + (N - 1)

In this situation, the "queue depth" on this CPU which is (read -
read_complete) returns -(N - 1). What does a -negative queue depth-
mean?

And what happens when there is sufficient other read/read_complete
differences on other CPUs that this negative sample cancels out all
the other positive "queue depths"?

IOWs, we can't even trust the instantaneous value of the queue depth
calculation to be a valid representation of the state of the
filesystem at a single point in time. Hence any downstream use of
the value (regardless of the sample rate) is not going to be any
more trustworthy than the instantaneous value....

> Answering your question, it lets userspace characterize filesystem
> queue depth over time. The places where this is useful are the ones
> where the desired signal persists across multiple sample periods,
> leading to a representative measurement:
....

All of these use cases are based on the assumption that the
underlying metric and the sampling method produces a valid and/or
meaningful representation of the current filesystem state.

I am not convinced that this is a valid assumption because of the
nature of the sampling - an instantaneous value sample cannot be
representative of overall behaviour, regardless of the sampling
rate.

However, if you really, really want this counter added then I won't
oppose it based on the fact I think it cannot be used the way you
want to use it - I'll just ignore it like I do all the other useless
stats we still keep around from the days of Irix for userspace
compatibility reasons.

Cheers,

Dave.
-- 
Dave Chinner
dgc@xxxxxxxxxx