Re: [PATCH v3 0/4] ceph: support subvolume quarantine

[Xiubo Li]

On Sat, Oct 17, 2026 at 01:40:12PM +0000, Alex Markuze wrote:
> CephFS subvolume quarantine isolates a subvolume while allowing access
> by recovery clients with q/Q MDS auth caps.

Hi Alex,

I tested v3 on a KASAN/lockdep kernel against a vstart cluster with
MDS quarantine support.  Basic quarantine, held fds, disable/recovery,
MDS failover, umount while quarantined and fsstress with quarantine
cycling all behave as expected, with clean dmesg.  The v1 flock leak is
fixed.  A recovery client with "allow rwq path=<subvol>" works, and a
read-only "rq" client can read but not write.  I saw no hot-path
regression on unrelated directories, with or without a quarantine
active.

Problems:

1) Snapshot access now costs an MDS round trip per operation, even
   when no subvolume is quarantined.  __ceph_do_getattr() forces
   snapshot inodes, the snapdir path does a forced getattr on the
   parent, and __ceph_get_caps() calls ceph_snapshot_read_auth() on
   every acquisition.  Compared with the base modules:

     1000 x stat() of a file under .snap:    0.20s -> 20.07s,
                                             0 -> 6000 MDS requests
     1024 x 4k read() of a snapshot file:     0.51s -> 3.93s,
                                             8 -> 1041 MDS requests
     ls -l of a 52-entry snapshot dir:        0.64s -> 3.62s

   Backup workloads that read from .snap will notice.  Can this be
   limited to clients that actually have quarantined state, or driven
   by an MDS notification instead of per-operation requests?

2) Blocking read() and getdents() can fail with EAGAIN.  With one
   subvolume quarantined, a file outside it at depth 300 gave 24 EAGAIN
   out of 20000 preads and 16 out of 20000 listdirs, while another task
   was creating, renaming and listing files on the same mount.  The
   ancestry walk stops at 256 levels, the result is CEPH_Q_UNKNOWN, and
   ceph_quarantine_revalidate() returns -EAGAIN when a splice is in
   progress, the epoch is odd, or after three requests.  Callers pass
   that straight to userspace.  Disconnected aliases (NFS re-export,
   open_by_handle_at) should hit the same path.  These calls should
   wait or retry rather than fail.

3) Some changes apply to every mount whether or not the MDS supports
   quarantine: ACL_DONT_CACHE on all inodes, and quarantine_lock plus
   two atomics on every d_splice_alias()/d_instantiate()/cross-directory
   d_move().  Please gate them on the feature.


For reference, overlapping grants such as "allow rw fsname=a, allow rwq
fsname=a path=<subvol>" (in either order) deny quarantined access with
both the kernel client and ceph-fuse, so that one is on the MDS side,
as the cover letter says.

Thanks,
Xiubo

On Sat, 17 Oct 2026 at 16:48, Alex Markuze <amarkuze@xxxxxxxxxx> wrote:
>
> CephFS subvolume quarantine isolates a subvolume while allowing access
> by recovery clients with q/Q MDS auth caps. The kernel client currently
> does not advertise quarantine support, so the MDS can hold its requests
> until quarantine is lifted.
>
> This series handles MClientCaps v14 quarantine notifications, checks
> cached data and metadata access, and handles disable notifications.
> Fresh MDS replies allow recovery after a lost disable. It applies directly
> to the base below and has no fscrypt prerequisite.
>
> The first patch exports d_find_alias_rcu() to GPL modules. The remaining
> three patches add the wire definitions, record quarantine state and
> ancestry generations, then install the access checks before advertising
> the session feature.
>
> Changes since v2:
> - Add the VFS export prerequisite to fix the undefined d_find_alias_rcu
>   symbol reported by the kernel test robot with CONFIG_CEPH_FS=m.
>   The fix carries Reported-by and Closes tags.
> - Keep all three Ceph patches unchanged from v2.
> - Replace object-only compile validation with a complete W=1 kernel and
>   module build using GCC 14.2.0 and the robot's x86_64-rhel-9.4 config.
>
> Validation:
> A complete W=1 build with the kernel.org GCC 14.2.0 toolchain and the
> robot's x86_64-rhel-9.4 config after olddefconfig passes. It produces
> vmlinux, bzImage and all 2,233 configured modules, including ceph.ko and
> libceph.ko. MODPOST succeeds, and Module.symvers contains the GPL export
> of d_find_alias_rcu. Configuration changes are limited to olddefconfig's
> toolchain capability normalization; there are no manual config changes.
> W=1 warnings remain elsewhere in the kernel; none come from Ceph or
> dcache.c.
> All four patches pass strict checkpatch and apply in order to reproduce
> the tested source tree. Seven deterministic source-extraction regression
> runners pass; the ASan/UBSan/LSan VFS runner also demonstrates the remaining
> cached encrypted-readlink bypass.
>
> Known limitation: cached encrypted symlink targets can bypass quarantine
> checks. fscrypt populates inode->i_link, which the VFS can use for readlink
> or symlink traversal without calling Ceph's get_link operation. Uncached
> encrypted targets and plaintext symlinks still pass through Ceph's checks.
> Closing the encrypted-target cache gap needs the separate fscrypt support
> and Ceph opt-in follow-up.
>
> Live cluster, KASAN/lockdep, multi-MDS recovery and depth benchmarks have
> not been run for this version. Existing snapshot mappings need server
> notifications to revoke already installed PTEs. Disable messages carry
> no generation, so same-rank ordering relies on ordered MDS delivery.
> Recovery assumes the intended rwq/rwQ delegation; overlapping ordinary
> rw and read-only q/Q grants still need server-side cap filtering.
>
> v2: https://lore.kernel.org/r/cover.1792074165.git.amarkuze@xxxxxxxxxx
> Robot report:
> https://lore.kernel.org/oe-kbuild-all/202610171220.i77KPiw5-lkp@xxxxxxxxx/
>
> Alex Markuze (4):
>   dcache: export d_find_alias_rcu() for modular filesystems
>   ceph: add subvolume quarantine wire definitions
>   ceph: record subvolume quarantine from MDS cap updates
>   ceph: fail quarantined subvolume access with EACCES
>
>  fs/ceph/acl.c                |  28 +-
>  fs/ceph/addr.c               |  72 ++-
>  fs/ceph/caps.c               | 291 +++++++++++-
>  fs/ceph/debugfs.c            |   1 +
>  fs/ceph/dir.c                |  51 ++-
>  fs/ceph/file.c               | 117 +++--
>  fs/ceph/inode.c              | 841 ++++++++++++++++++++++++++++++++++-
>  fs/ceph/ioctl.c              |  13 +
>  fs/ceph/mds_client.c         |  23 +-
>  fs/ceph/mds_client.h         |  26 +-
>  fs/ceph/super.c              |   2 +
>  fs/ceph/super.h              |  80 +++-
>  fs/ceph/xattr.c              |  54 ++-
>  fs/dcache.c                  |   1 +
>  include/linux/ceph/ceph_fs.h |  13 +
>  net/ceph/ceph_common.c       |   2 +
>  16 files changed, 1482 insertions(+), 133 deletions(-)
>
>
> base-commit: ed0502cb11f84fff29f374238e526ae94d9bdfde
> --
> 2.34.1
>