[PATCH v2 0/4] ceph: fix the buffered read size metric and make the subvolume metrics per CPU

[Xiubo Li via B4 Relay]

Two problems in the client I/O metrics, reported by Dan van der Ster
while looking at IO500 runs on 144-core Grace clients.

Patch 1: metrics/size has reported 0 bytes for every page cache read
since buffered reads moved to CEPH_OSD_DATA_TYPE_ITER, because
finish_netfs_read() still reads osd_data->length, which only the page
vector type sets.

Patches 2-4: every OSD read and write completion updates the subvolume
metrics through state shared by the whole mount.  Even with subvolume
metrics disabled it bumped an atomic64 counter that nothing reads;
with them enabled it took a mount-wide spinlock and searched an rbtree.
Patch 2 drops the unused counters.  Patch 3 flushes the OSD completion
workqueue before the mds client and its metrics are freed.  Patch 4
accumulates the counters in a per-CPU slot and only folds them into the
tree when the slot changes subvolume or when the metrics are sent or
dumped.

Tested on a 12-vCPU x86_64 VM against a main MDS with SUBVOLUME_METRICS,
with KASAN, lockdep and kmemleak enabled:

- 12 fio jobs doing 4k direct I/O in one subvolume: the op and byte
  totals the MDS received over all metrics sends match what fio issued.
  Alternating between two subvolumes on every I/O, the MDS sees the
  same totals as with the unpatched client.
- The same with failslab failing half of the subvolume entry
  allocations: in one subvolume v1 lost about 9% of the ops and this
  version loses none; alternating between two subvolumes it drops 64
  of 240k I/Os, those whose direct insert failed as well.
- lock_stat over 20s of 12-job random reads in one subvolume:
  tracker->lock goes from 90k acquisitions and 78 contentions to 316
  and none; the per-CPU slot locks see no contention.
- umount -f with the OSDs paused and I/O in flight, and MDS failover
  and kill -9 under I/O: no KASAN, lockdep or kmemleak reports.  The
  race patch 3 closes was not hit there: all 3840 aborted requests
  completed before ceph_mdsc_destroy().

Throughput on that VM is unchanged within noise: at 12 vCPUs the
tracker lock is not a bottleneck, so the gain is only expected on
large clients like the ones in the report.

---
Changes in v2:
- Fold a per-CPU slot into the tree under the tracker lock and clear it
  only once its entry is in the tree, so its counters are kept when the
  entry allocation fails (Alex Markuze)
- New patch 3: flush osdc->completion_wq in ceph_mdsc_destroy() before
  the metrics are freed (Alex Markuze)
- Link to v1: https://patch.msgid.link/20261014-ceph-metrics-percpu-v1-0-38c5a24fec6e@xxxxxxxxx

To: Ilya Dryomov <idryomov@xxxxxxxxx>
To: Alex Markuze <amarkuze@xxxxxxxxxx>
To: Xiubo Li <xiubo.li@xxxxxxxxx>
To: Viacheslav Dubeyko <slava@xxxxxxxxxxx>
Cc: ceph-devel@xxxxxxxxxxxxxxx
Cc: linux-kernel@xxxxxxxxxxxxxxx

---
Xiubo Li (4):
      ceph: report the requested length in the netfs read metrics
      ceph: drop the unused subvolume metrics debug counters
      ceph: flush the OSD completions before destroying the mds client
      ceph: accumulate subvolume metrics per CPU

 fs/ceph/addr.c                  |   2 +-
 fs/ceph/mds_client.c            |   8 +-
 fs/ceph/subvolume_metrics.c     | 275 ++++++++++++++++++++++++++++------------
 fs/ceph/subvolume_metrics.h     |  26 +---
 include/linux/ceph/osd_client.h |   1 +
 net/ceph/osd_client.c           |  11 ++
 6 files changed, 217 insertions(+), 106 deletions(-)
---
base-commit: ed0502cb11f84fff29f374238e526ae94d9bdfde
change-id: 20261014-ceph-metrics-percpu-1cb5047f7150

Best regards,
--  
Xiubo Li <xiubo.li@xxxxxxxxx>