[PATCH 0/2] ceph: serialize O_APPEND writes across clients
From: Xiubo Li via B4 Relay
Date: Fri Nov 20 2026 - 20:21:27 EST
When two clients append to one file, the filelock is in MIX and both
hold Fw, so each appends at the end of file it has cached and may
overwrite what the other one appended. Commit d2a8d446a09c ("ceph:
revalidate ki_pos for O_APPEND writes after cap acquisition") does
not cover this case: neither client waits for caps, and the other
client's size only arrives after it flushed its caps.
The MDS side [1] lets O_APPEND writers take turns on Fx, in request
order, and sends the current size together with Fx. It is enabled by
the new CEPHFS_FEATURE_APPEND_EXCL feature bit; with an MDS that does
not have it, the client behaves as before.
Patch 1 fixes a reordering of the cap messages of an inode that this
work exposed: a later cap update could overtake the flush carrying the
new size, the MDS then finished its Fw gather early and gave Fx with
the old size to the other client.
Patch 2 is the O_APPEND change itself.
Tested on a vstart cluster with [1], KASAN and lockdep enabled, two
mounts (noshare, or ceph-fuse + kernel), 4 writers on each, without
any file locking; a lost record means one append overwrote another:
+--------------------------------------+--------+------------------+
| case | runs | lost records |
| | | base / patched |
+--------------------------------------+--------+------------------+
| buffered, kernel + kernel | 4 | ~3300-4000 / 0 |
| O_DIRECT, kernel + kernel | 10 | ~4000 / 0 |
| buffered/O_DIRECT, ceph-fuse+kernel | 6 | - / 0 |
| 2 active MDS, export every 3 s | 4 | - / 0 |
| MDS failover during the writes | 2 | - / 0 |
+--------------------------------------+--------+------------------+
With an MDS without the feature the patched client loses records like
the base one, and does not hang. xfstests -g quick shows the same 8
failures (ceph/006, fscrypt key tests) with and without the series.
[1] https://github.com/ceph/ceph/pull/70279
Signed-off-by: Xiubo Li <xiubo.li@xxxxxxxxx>
---
Xiubo Li (2):
ceph: send the cap messages of an inode in the order they were prepared
ceph: serialize O_APPEND writes across clients on Fx
fs/ceph/caps.c | 208 +++++++++++++++++++++++++++++++++++++++++++
fs/ceph/file.c | 33 +++++--
fs/ceph/inode.c | 6 ++
fs/ceph/mds_client.h | 4 +-
fs/ceph/super.h | 17 ++++
include/linux/ceph/ceph_fs.h | 1 +
6 files changed, 261 insertions(+), 8 deletions(-)
---
base-commit: 6a9830a8f414d3830c584782568d0233be969d09
change-id: 20261121-ceph-append-excl-64ce323d1a04
Best regards,
--
Xiubo Li <xiubo.li@xxxxxxxxx>