fs/ceph: ceph_statfs() can dereference a freed MDS map (double evaluation of mdsc->mdsmap) (UAF)

From: 徐晗翔

Date: Tue Nov 17 2026 - 07:48:46 EST


ceph_statfs() evaluates fsc->mdsc->mdsmap twice in one expression pair
(super.c:68-69), without mdsc->mutex. ceph_mdsc_handle_mdsmap() replaces
mdsc->mdsmap and immediately frees the old struct ceph_mdsmap in the same
critical section (mds_client.c:7057-7061), and struct ceph_mdsmap has no
reference count and no RCU freeing. Any unprivileged user can call
statfs(2)/df on a ceph mount, so a statfs that overlaps an MDS map update can
read a freed map, or can observe a new map with m_num_data_pg_pools == 0 and
dereference m_data_pg_pools[0] where kcalloc(0) returned ZERO_SIZE_PTR
(0x10) - a near-NULL dereference. I reproduced both on 7.3: 9 KASAN
slab-use-after-free reports across the statfs read points (including the
second evaluation) and a null-ptr-deref at 0x10 with the same read shape.

The detailed description:

* Test environment / affected version

Linux 7.3.0 tree (Makefile: VERSION 7, PATCHLEVEL 3, SUBLEVEL 0) at
/home/xuhanxiang/project/race/linux, not a git checkout:

fs/ceph/super.c d06bb506183e26374582ee5d82322b8c
fs/ceph/mds_client.c 8678459253e5b1452667946bf4457c5a
fs/ceph/mdsmap.c 5cb2b54f0d3007961c36e68da1f6318c
fs/ceph/mdsmap.h 8effd15a2d508dc1f69b44b47a5ce3da

All line numbers are from this tree. Kernel: 7.3.0 #2 PREEMPT(lazy), gcc
13.3.0, KASAN enabled (CONFIG_KASAN=y, CONFIG_CEPH_FS=y, CONFIG_CEPH_LIB=y,
CONFIG_DEBUG_FS=y). QEMU/KVM guest (4 vCPU, 4G), busybox initramfs.

* Mechanism

ceph_statfs() (super.c:59-76):

u64 data_pool;

if (fsc->mdsc->mdsmap->m_num_data_pg_pools == 1) {
data_pool = fsc->mdsc->mdsmap->m_data_pg_pools[0];
} else {
data_pool = CEPH_NOPOOL;
}
err = ceph_monc_do_statfs(monc, data_pool, &st);

The compiler is free to load mdsc->mdsmap twice (and the source shows two
separate expressions), so:

A) T1 loads mdsc->mdsmap = M0 at line 68, compares M0->m_num_data_pg_pools
== 1; T2 publishes M1 and frees M0; T1 then loads
mdsc->mdsmap->m_data_pg_pools (line 69) - either re-reading the current
pointer (M1, usually fine) or, if it already loaded M0, dereferencing
freed memory.
B) Even without a re-read, a load of M0 that is followed by the free
before the m_data_pg_pools load is a UAF read.
C) If line 69 observes a new map with m_num_data_pg_pools == 0,
m_data_pg_pools is kcalloc(0), which returns ZERO_SIZE_PTR (0x10) on
this tree (mdsmap.c:282-292), and [0] dereferences 0x10.

The writer side is the same as in the debugfs report: mdsc->mdsmap is
replaced at mds_client.c:7059 and the old map destroyed at 7061; no
refcount/RCU; ceph_mdsmap_destroy() (mdsmap.c:443-455) frees the arrays
synchronously. Note that ioctl.c:41-58 (__validate_layout) reads the same
data-pool fields under mdsc->mutex - the lock protocol exists, ceph_statfs()
just does not use it.

* Trigger model

- Any local user with access to a mounted cephfs can call statfs(2)/df; no
privileges needed.
- A concurrent MDS map update is needed: MDS restart/failover, rank removal,
max_mds change, laggy set updates, etc. (mon connection work context.)
- The window is between the pointer loads and the m_data_pg_pools load.

* Reproducer (attached: ceph_mdsmap_race_harness.c + instrumentation.patch)
and results on 7.3

the harness builds a real struct ceph_mds_client (real ceph_mdsc_init()),
publishes a minimal MDS map through the real ceph_mdsc_handle_mdsmap(), and
runs readers that replay the exact ceph_statfs() read shape: load
mdsc->mdsmap (READ_ONCE) -> block -> after the map is freed, read
m_num_data_pg_pools and m_data_pg_pools[0]. The "replicas" plan runs five
readers; kinds 1 and 2 are the two statfs evaluation forms. The "mr02zero"
plan publishes a new map with m_num_data_pg_pools = 0.

Result (plan-replicas.log, plan-mr02zero.log):

BUG: KASAN: slab-use-after-free in rv_reader_fn+0x6c/0x80/0x90
Read of size 4/8 ... (m_num_data_pg_pools and m_data_pg_pools load)
Allocated by: ceph_mdsmap_decode+0xe1 <- ceph_mdsc_handle_mdsmap+0x288
Freed by: ceph_mdsc_handle_mdsmap+0x2c7
BUG: KASAN: null-ptr-deref in rv_reader_fn+0x132/0x170
Read of size 8 at addr 0000000000000010

Control run replicasctl (release readers before publishing the new map):
KASAN count 0, all readers return normal values.



* Suggested direction

Snapshot the fields once under mdsc->mutex:

u64 data_pool;
mutex_lock(&fsc->mdsc->mutex);
if (fsc->mdsc->mdsmap->m_num_data_pg_pools == 1)
data_pool = fsc->mdsc->mdsmap->m_data_pg_pools[0];
else
data_pool = CEPH_NOPOOL;
mutex_unlock(&fsc->mdsc->mutex);

Long term, RCU publication would let ceph_statfs() use a stable snapshot
without holding the mutex.

Please keep me in Cc; I am not subscribed to the lists.

Regards,
Hanxiang Xu
xuhx@xxxxxxxxxxx

Attachment: ceph_mdsmap_race_harness.h
Description: Binary data

// SPDX-License-Identifier: GPL-2.0
/*
* fs/ceph mdsmap lifetime race verification harness (TEST ONLY, not upstream).
*
* This file is compiled into ceph.o only when CONFIG_CEPH_RACE_VERIFY=y.
* It builds a fake (but minimally consistent) ceph client / fs_client /
* mds_client, publishes MDS maps into mdsc->mdsmap through the *real*
* ceph_mdsc_handle_mdsmap() writer, and lets readers (the real debugfs
* mdsmap_show() and line-faithful replicas of the audited read sites)
* dereference a map while a concurrent map update publishes a new map and
* ceph_mdsmap_destroy()s the old one.
*
* Only the test scaffolding and one stall hook are added; the writer, the
* free path, the debugfs show callback and ceph_mdsmap_is_cluster_available()
* are the unmodified production code under test.
*/
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/debugfs.h>
#include <linux/fs.h>
#include <linux/kthread.h>
#include <linux/delay.h>
#include <linux/completion.h>
#include <linux/atomic.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/uaccess.h>
#include <linux/ceph/decode.h>
#include <linux/ceph/libceph.h>

#include "super.h"
#include "mds_client.h"
#include "mdsmap.h"
#include "race_verify.h"

static struct ceph_options rv_copts; /* con_modes[0] == UNKNOWN -> msgr1 */
static struct ceph_client *rv_client;
static struct ceph_fs_client *rv_fsc;
static struct ceph_mount_options *rv_mopts;
static struct ceph_mds_client *rv_mdsc;
static struct dentry *rv_dir;
static u32 rv_cur_epoch;

static bool rv_pause_enabled;
static atomic_t rv_loaded = ATOMIC_INIT(0);
static atomic_t rv_released = ATOMIC_INIT(0);
static DECLARE_COMPLETION(rv_go);

#define RV_BUF_SZ 4096
#define RV_NMDS 1

/* ------------------------------------------------------------------ */
/* MDS map wire encoder: mdsmap_v=1 (legacy), one ACTIVE mds rank. */
/* ------------------------------------------------------------------ */

#define RV_MSG_HDR_SZ 24 /* ceph_fsid (16) + epoch (u32) + maplen (u32) */

static int rv_build_map(u8 *buf, size_t bufsz, u32 epoch, u32 npools, u32 in_n)
{
struct ceph_fsid fsid;
void *p = buf + RV_MSG_HDR_SZ;
void *end = buf + bufsz;
u32 maplen;
u32 i;
int j;

if (bufsz <= RV_MSG_HDR_SZ)
return -EINVAL;

ceph_encode_8(&p, 1); /* mdsmap_v */
ceph_encode_8(&p, 0); /* compat version byte */
ceph_encode_32(&p, epoch); /* m_epoch */
ceph_encode_32(&p, 0); /* m_client_epoch */
ceph_encode_32(&p, 0); /* m_last_failure */
ceph_encode_32(&p, 1); /* m_root */
ceph_encode_32(&p, 60); /* m_session_timeout */
ceph_encode_32(&p, 300); /* m_session_autoclose */
ceph_encode_64(&p, 1ULL << 40); /* m_max_file_size */
ceph_encode_32(&p, RV_NMDS); /* m_max_mds */
ceph_encode_32(&p, RV_NMDS); /* m_num_active_mds */

for (i = 0; i < RV_NMDS; i++) {
ceph_encode_64(&p, i + 1); /* global_id */
ceph_encode_8(&p, 1); /* info_v (no addrvec, no exports) */
ceph_encode_64(&p, 0); /* skipped field */
ceph_encode_32(&p, 0); /* namelen */
ceph_encode_32(&p, i); /* mds rank */
ceph_encode_32(&p, 0); /* inc */
ceph_encode_32(&p, CEPH_MDS_STATE_ACTIVE);
ceph_encode_64(&p, 0); /* state_seq */
/* legacy entity_addr: 1 marker byte + 3 type bytes + nonce */
ceph_encode_8(&p, CEPH_ENTITY_ADDR_TYPE_NONE);
ceph_encode_8(&p, 0);
ceph_encode_8(&p, 0);
ceph_encode_8(&p, 0);
ceph_encode_32(&p, 0); /* nonce */
for (j = 0; j < 16; j++) /* sockaddr_storage (128B) */
ceph_encode_64(&p, 0);
ceph_encode_64(&p, 0); /* laggy_since (ceph_timespec) */
ceph_encode_32(&p, 0); /* skipped field */
ceph_encode_32(&p, 0); /* namelen */
}

ceph_encode_32(&p, npools); /* m_num_data_pg_pools */
for (i = 0; i < npools; i++)
ceph_encode_64(&p, 1); /* data pool id */
ceph_encode_64(&p, 0); /* m_cas_pg_pool */

/* tail for mdsmap_ev == 1 */
ceph_encode_32(&p, 1); /* metadata_pool */
ceph_encode_64(&p, 0); /* created (ceph_timespec) */
ceph_encode_64(&p, 0); /* modified (ceph_timespec) */
ceph_encode_32(&p, 0); /* tableserver */
ceph_encode_32(&p, in_n); /* in set: in_n entries */
for (i = 0; i < in_n; i++)
ceph_encode_32(&p, i); /* rank i is in */
ceph_encode_32(&p, 0); /* inc map: 0 entries */
ceph_encode_32(&p, 0); /* up map: 0 entries */
ceph_encode_32(&p, 0); /* failed set: 0 entries */
ceph_encode_32(&p, 0); /* stopped set: 0 entries */

if (p > end)
return -EOVERFLOW;

maplen = p - (void *)(buf + RV_MSG_HDR_SZ);

/* message header: ceph_fsid + map epoch + map length */
memset(&fsid, 0, sizeof(fsid));
fsid.fsid[0] = 0xce;
fsid.fsid[15] = 0x71;
p = buf;
ceph_encode_copy(&p, &fsid, sizeof(fsid));
ceph_encode_32(&p, epoch);
ceph_encode_32(&p, maplen);
if (p != buf + RV_MSG_HDR_SZ)
return -EINVAL;
return (buf + RV_MSG_HDR_SZ + maplen) - buf;
}

static int rv_deliver_map_ex(u32 epoch, u32 npools, u32 in_n)
{
struct ceph_msg msg = { };
u8 buf[RV_BUF_SZ];
int len;

len = rv_build_map(buf, sizeof(buf), epoch, npools, in_n);
if (len < 0) {
pr_err("CEPHV: map encode failed %d\n", len);
return len;
}
msg.front.iov_base = buf;
msg.front.iov_len = len;
ceph_mdsc_handle_mdsmap(rv_mdsc, &msg);
rv_cur_epoch = epoch;
return 0;
}

static int rv_deliver_map(u32 epoch, u32 npools)
{
return rv_deliver_map_ex(epoch, npools, RV_NMDS);
}

/* ------------------------------------------------------------------ */
/* Setup: fake client / fs_client / mds_client, then first map epoch 2 */
/* ------------------------------------------------------------------ */

static int rv_setup(void)
{
int ret;

if (rv_mdsc)
return -EALREADY;

rv_client = kzalloc(sizeof(*rv_client), GFP_KERNEL);
rv_fsc = kzalloc(sizeof(*rv_fsc), GFP_KERNEL);
rv_mopts = kzalloc(sizeof(*rv_mopts), GFP_KERNEL);
if (!rv_client || !rv_fsc || !rv_mopts)
return -ENOMEM;

rv_client->options = &rv_copts;
mutex_init(&rv_client->monc.mutex);
rv_client->monc.client = rv_client;

rv_fsc->client = rv_client;
rv_fsc->mount_options = rv_mopts;
INIT_LIST_HEAD(&rv_fsc->metric_wakeup);

ret = ceph_mdsc_init(rv_fsc);
if (ret) {
pr_err("CEPHV: ceph_mdsc_init failed %d\n", ret);
return ret;
}
rv_mdsc = rv_fsc->mdsc;

/* Neutralize the delayed_work timer before the first map delivery,
* so the work item early-returns instead of running cap machinery.
*/
rv_mdsc->stopping = CEPH_MDSC_STOPPING_FLUSHED;

ret = rv_deliver_map(2, 1);
if (ret)
return ret;
flush_delayed_work(&rv_mdsc->delayed_work);

/* The real debugfs show callback, using the real mdsmap_fops. */
ceph_race_create_mdsmap_file(rv_fsc, rv_dir);

pr_info("CEPHV: setup done, epoch=%u stopping=%d\n",
rv_cur_epoch, (int)rv_mdsc->stopping);
return 0;
}

/* ------------------------------------------------------------------ */
/* Stall hook called from the real mdsmap_show() after it snapshots */
/* fsc->mdsc->mdsmap (debugfs.c:66 in the unmodified tree). */
/* ------------------------------------------------------------------ */

void ceph_race_verify_pause(struct ceph_mds_client *mdsc,
struct ceph_mdsmap *mdsmap)
{
if (!READ_ONCE(rv_pause_enabled) || !rv_mdsc)
return;
atomic_inc(&rv_loaded);
wait_for_completion(&rv_go);
atomic_set(&rv_loaded, 0);
}

/* ------------------------------------------------------------------ */
/* Kernel-thread readers: faithful replicas of the audited read sites */
/* ------------------------------------------------------------------ */

enum rv_reader_kind {
RV_R_STATFS = 1, /* super.c:68-69 (ceph_statfs) */
RV_R_TOCTOU, /* super.c:69 (double load) */
RV_R_XATTR, /* xattr.c:1237 (__ceph_setxattr) */
RV_R_RENEW, /* mds_client.c:2130/2179 */
RV_R_PEER, /* mds_client.c:5227 */
};

static int rv_reader_fn(void *arg)
{
long kind = (long)arg;
struct ceph_mdsmap *m, *m2;
u64 val = 0;
u32 u32v = 0;
int st;

m = READ_ONCE(rv_mdsc->mdsmap);
atomic_inc(&rv_loaded);
wait_for_completion(&rv_go);

switch (kind) {
case RV_R_STATFS:
u32v = m->m_num_data_pg_pools;
if (u32v == 1) {
u64 *p = m->m_data_pg_pools;
val = p[0];
}
break;
case RV_R_TOCTOU:
u32v = m->m_num_data_pg_pools;
m2 = READ_ONCE(rv_mdsc->mdsmap);
if (u32v == 1) {
u64 *p = m2->m_data_pg_pools;
val = p[0];
}
break;
case RV_R_XATTR:
val = m->m_max_xattr_size;
break;
case RV_R_RENEW:
u32v = m->m_session_timeout;
break;
case RV_R_PEER:
st = ceph_mdsmap_get_state(m, 0);
{
struct ceph_entity_addr addr;
struct ceph_entity_addr *a =
ceph_mdsmap_get_addr(m, 0);
if (a)
memcpy(&addr, a, sizeof(addr));
pr_info("CEPHV: peer replica state=%d addr=%p\n",
st, a);
}
return 0;
default:
return -EINVAL;
}

pr_info("CEPHV: reader kind=%ld val=%llu u32=%u\n", kind,
(unsigned long long)val, u32v);
return 0;
}

/* Real production reader: ceph_mdsmap_is_cluster_available() on a
* previously snapshotted map (the ceph_get_tree() out_splat pattern).
*/
static int rv_reader3_fn(void *arg)
{
struct ceph_mdsmap *m = READ_ONCE(rv_mdsc->mdsmap);
bool avail;

atomic_inc(&rv_loaded);
wait_for_completion(&rv_go);
avail = ceph_mdsmap_is_cluster_available(m);
pr_info("CEPHV: reader3 avail=%d\n", avail);
return 0;
}

static int rv_spawn(int (*fn)(void *), long kind, const char *name,
struct task_struct **out)
{
struct task_struct *t = kthread_run(fn, (void *)kind, "%s", name);

if (IS_ERR(t))
return PTR_ERR(t);
if (out)
*out = t;
return 0;
}

static void rv_release(void)
{
atomic_set(&rv_released, 1);
complete_all(&rv_go);
}

/* ------------------------------------------------------------------ */
/* control file */
/* ------------------------------------------------------------------ */

static ssize_t rv_ctl_write(struct file *f, const char __user *ubuf,
size_t len, loff_t *off)
{
struct task_struct *t;
char buf[32];
int ret = 0;

if (len >= sizeof(buf))
return -EINVAL;
if (copy_from_user(buf, ubuf, len))
return -EFAULT;
buf[len] = 0;
strim(buf);

if (!strcmp(buf, "setup")) {
ret = rv_setup();
} else if (!strcmp(buf, "arm")) {
rv_pause_enabled = true;
reinit_completion(&rv_go);
atomic_set(&rv_loaded, 0);
atomic_set(&rv_released, 0);
pr_info("CEPHV: armed (pause hook enabled)\n");
} else if (!strcmp(buf, "trigger")) {
ret = rv_deliver_map(rv_cur_epoch + 1, 1);
pr_info("CEPHV: trigger epoch=%u pools=1\n", rv_cur_epoch);
} else if (!strcmp(buf, "trigger0")) {
ret = rv_deliver_map_ex(rv_cur_epoch + 1, 0, 0);
pr_info("CEPHV: trigger epoch=%u pools=0\n", rv_cur_epoch);
} else if (!strcmp(buf, "release")) {
rv_release();
pr_info("CEPHV: released\n");
} else if (!strcmp(buf, "reader3")) {
ret = rv_spawn(rv_reader3_fn, 0, "cephrv3", &t);
} else if (!strcmp(buf, "rep")) {
ret = rv_spawn(rv_reader_fn, RV_R_STATFS, "cephrv-s", NULL);
if (!ret)
ret = rv_spawn(rv_reader_fn, RV_R_TOCTOU, "cephrv-t", NULL);
if (!ret)
ret = rv_spawn(rv_reader_fn, RV_R_XATTR, "cephrv-x", NULL);
if (!ret)
ret = rv_spawn(rv_reader_fn, RV_R_RENEW, "cephrv-r", NULL);
if (!ret)
ret = rv_spawn(rv_reader_fn, RV_R_PEER, "cephrv-p", NULL);
} else if (!strcmp(buf, "rep0")) {
ret = rv_spawn(rv_reader_fn, RV_R_TOCTOU, "cephrv-t0", NULL);
} else if (!strcmp(buf, "noop")) {
ret = 0;
} else {
ret = -EINVAL;
}

if (ret)
return ret;
return len;
}

static const struct file_operations rv_ctl_fops = {
.write = rv_ctl_write,
};

static int rv_state_show(struct seq_file *s, void *unused)
{
seq_printf(s, "mdsc=%d epoch=%u loaded=%d released=%d pause=%d stopping=%d\n",
rv_mdsc ? 1 : 0, rv_cur_epoch,
atomic_read(&rv_loaded), atomic_read(&rv_released),
rv_pause_enabled ? 1 : 0,
rv_mdsc ? (int)rv_mdsc->stopping : -1);
return 0;
}
DEFINE_SHOW_ATTRIBUTE(rv_state);

static int __init rv_init(void)
{
rv_dir = debugfs_create_dir("cephrace", NULL);
if (IS_ERR_OR_NULL(rv_dir))
return 0;
debugfs_create_file("control", 0200, rv_dir, NULL, &rv_ctl_fops);
debugfs_create_file("state", 0400, rv_dir, NULL, &rv_state_fops);
pr_info("CEPHV: harness loaded\n");
return 0;
}
late_initcall(rv_init);

Attachment: instrumentation.patch
Description: Binary data