Re: [PATCH] ceph: reconnect on peer reset even if the mdsmap says the MDS is active

[Xiubo Li]

On Wed, Oct 21, 2026 at 19:06, Sun Yuechi wrote:
> Since commit 39fe30315893 ("ceph: harden send_mds_reconnect and handle
> active-MDS peer reset"), mds_peer_reset() tears the session down instead
> of reconnecting when our mdsmap shows the rank past RECONNECT.
>
> That map can be stale.  An MDS killed and restarted on the same address
> resets our connection before the monitors report the rank as failed, so
> the session is torn down while the map still says active.

Agreed.  The local teardown trusts a map that the peer reset itself
says is out of date.  Once the session has been unregistered,
check_new_map() has nothing to reconnect, and the new rank 0 evicts us.

> Send the reconnect and let the MDS decide, as before that commit.  A
> daemon restarted on the old address has a new nonce, so the reconnect
> does not get through, and check_new_map() reconnects once a map carries
> the rank's new address.

I checked this path.  The session stays in RECONNECTING, but
check_new_map() compares the whole ceph_entity_addr, nonce included.
So both a standby taking over and the same daemon coming back on the
old ip:port end up in ceph_con_close(), RESTARTING and a fresh
send_mds_reconnect().  Nothing is left stuck in RECONNECTING.

> An MDS that really is active replies with
> CLOSE, and handle_session() unregisters the session so that new
> requests open a fresh one, which is what that commit intended.

Yes.  Server::handle_client_reconnect() denies a reconnect outside the
reconnect window, or for a closed session, by sending
CEPH_SESSION_CLOSE ("denied reconnect attempt").  A sessionless
reconnect gets CEPH_SESSION_REJECT.  So the "reject or ignore" premise
in 39fe30315893 does not hold for the ignore part.  The client side then
unregisters the session, removes its caps and kicks the waiting
requests.

A few nits, none of them blocking:

1. It would help to say in the changelog that this is a partial revert:
   only the mds_peer_reset() half of 39fe30315893 goes away, and the
   send_mds_reconnect() hardening stays.  Please also mention that the
   active MDS denies the reconnect with CLOSE.

2. The reproducer needs the client to send no MDS request between the
   peer reset and the new map.  If a request is issued in that window,
   it registers a new session.  check_new_map() then reconnects that
   session when the address changes, so the bug is hidden.  Even then,
   39fe30315893 has already dropped all of the client's caps with
   remove_session_caps(), while a real reconnect keeps them.  That seems
   worth a sentence too.

> -       if (rc)
> +       rc = send_mds_reconnect(mdsc, s);
> +       if (rc && rc != -ESTALE && rc != -ENOENT)

3. check_new_map() still warns on every non-zero return of
   send_mds_reconnect().  Its -ESTALE/-ENOENT cases are logged by
   send_mds_reconnect() itself, just like here.  Not for this patch,
   but the two callers could be made consistent in a follow-up.

> +        * Send the reconnect and let the MDS decide; one that is past
> +        * RECONNECT replies with CLOSE and handle_session() closes the

4. Nit: "replies with CLOSE (or REJECT if it has no session for us)".

Tested on x86_64 with a 7.3.0-rc1 based kernel (KASAN, lockdep,
kmemleak) against a vstart cluster from Ceph main: 1 MON, 3 OSDs, and
two MDSes with allow_standby_replay set (one active plus one
standby-replay).  Only ceph.ko was swapped between the runs; libceph.ko
was identical.

Each run did the following.  A writer keeps a file open and stops
issuing requests during the window.  iptables drops the kernel client's
established monitor connection.  The active MDS gets SIGKILL and is
restarted on the same address with --public_addrv.  Once the client logs
"closed our session", the block is lifted.  After the takeover the run
checks ls, a new write, a read and further writes on the held fd.

  stale map, MDS restarted on the old address:
    without the patch: 4 of 4 runs evicted ("session blocklisted",
                       "rejected session"), EACCES afterwards
    with the patch:    3 of 3 runs "reconnect start" ->
                       "reconnect success" -> "recovery completed",
                       no errors from the writer

  "client evict" by the active MDS, mds_session_blocklist_on_evict=false:
    without the patch: 2 of 2 recover with a new session
    with the patch:    2 of 2 "reconnect denied", then a new session,
                       I/O fine

No KASAN, lockdep or kmemleak reports.

Reviewed-by: Xiubo Li <xiubo.li@xxxxxxxxx>
Tested-by: Xiubo Li <xiubo.li@xxxxxxxxx>

Thanks,
- Xiubo

On Wed, 21 Oct 2026 at 19:06, Sun Yuechi <sunyuechi@xxxxxxxxxxx> wrote:
>
> Since commit 39fe30315893 ("ceph: harden send_mds_reconnect and handle
> active-MDS peer reset"), mds_peer_reset() tears the session down instead
> of reconnecting when our mdsmap shows the rank past RECONNECT.
>
> That map can be stale.  An MDS killed and restarted on the same address
> resets our connection before the monitors report the rank as failed, so
> the session is torn down while the map still says active.  When a
> standby then takes over the rank in up:reconnect, check_new_map() has
> no session left to reconnect, the new MDS evicts and blocklists us
> after mds_reconnect_timeout, and the mount returns EACCES until it is
> unmounted:
>
>   libceph: mds0 (2)10.89.20.12:6800 session reset
>   ceph: [...]: mds0 closed our session
>   (mds) evicting unresponsive client ceph-c1 (14900), after waiting
>         49.7175 seconds during MDS startup
>
> Send the reconnect and let the MDS decide, as before that commit.  A
> daemon restarted on the old address has a new nonce, so the reconnect
> does not get through, and check_new_map() reconnects once a map carries
> the rank's new address.  An MDS that really is active replies with
> CLOSE, and handle_session() unregisters the session so that new
> requests open a fresh one, which is what that commit intended.
>
> Skip the warning for -ESTALE and -ENOENT: send_mds_reconnect() already
> logs those, and the removed code was silent for closed and rejected
> sessions.
>
> Tested on riscv64 (7.2.6 kernel, Ceph 21, one active MDS with a
> standby-replay): SIGKILL the active MDS while dropping the client's
> monitor traffic until it logs "closed our session".  Evicted in 4 of 4
> runs without the patch, "reconnect success" in 3 of 3 with it.  Eviction
> by an active MDS without blocklist still recovers with a new session,
> 2 of 2 runs.
>
> Fixes: 39fe30315893 ("ceph: harden send_mds_reconnect and handle active-MDS peer reset")
> Cc: stable@xxxxxxxxxxxxxxx
> Signed-off-by: Sun Yuechi <sunyuechi@xxxxxxxxxxx>
> ---
>  fs/ceph/mds_client.c | 84 +++++---------------------------------------
>  1 file changed, 9 insertions(+), 75 deletions(-)
>
> diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
> index d4ef8a355a48..d23250af0750 100644
> --- a/fs/ceph/mds_client.c
> +++ b/fs/ceph/mds_client.c
> @@ -7236,7 +7236,7 @@ static void mds_peer_reset(struct ceph_connection *con)
>  {
>         struct ceph_mds_session *s = con->private;
>         struct ceph_mds_client *mdsc = s->s_mdsc;
> -       int session_state;
> +       int rc;
>
>         pr_warn_client(mdsc->fsc->client, "mds%d closed our session\n",
>                        s->s_mds);
> @@ -7246,82 +7246,16 @@ static void mds_peer_reset(struct ceph_connection *con)
>                 return;
>
>         /*
> -        * Only reconnect if MDS is in its RECONNECT phase.  An MDS past
> -        * RECONNECT (REJOIN, CLIENTREPLAY, ACTIVE) will reject reconnect
> -        * attempts, so those states fall through to session teardown below.
> +        * Our mdsmap may be stale: an MDS restarted on the same address
> +        * resets the connection before we learn that the rank failed.
> +        * Send the reconnect and let the MDS decide; one that is past
> +        * RECONNECT replies with CLOSE and handle_session() closes the
> +        * session so that new requests open a fresh one.
>          */
> -       if (ceph_mdsmap_get_state(mdsc->mdsmap, s->s_mds) == CEPH_MDS_STATE_RECONNECT) {
> -               int rc = send_mds_reconnect(mdsc, s);
> -
> -               if (rc)
> -                       pr_warn_client(mdsc->fsc->client,
> -                                      "mds%d reconnect failed: %d\n",
> -                                      s->s_mds, rc);
> -               return;
> -       }
> -
> -       /*
> -        * MDS is active (past RECONNECT).  It will not accept a
> -        * CLIENT_RECONNECT from us, so tear the session down locally
> -        * and let new requests re-open a fresh session.
> -        *
> -        * Snapshot session state with READ_ONCE, then revalidate under
> -        * mdsc->mutex before acting.  The subsequent mdsc->mutex
> -        * section rechecks s_state to catch concurrent transitions, so
> -        * the lockless snapshot here is safe.  s->s_mutex is taken
> -        * separately for cleanup after unregistration, which avoids
> -        * introducing a new s->s_mutex + mdsc->mutex nesting.
> -        */
> -       session_state = READ_ONCE(s->s_state);
> -
> -       switch (session_state) {
> -       case CEPH_MDS_SESSION_RESTARTING:
> -       case CEPH_MDS_SESSION_RECONNECTING:
> -       case CEPH_MDS_SESSION_CLOSING:
> -       case CEPH_MDS_SESSION_OPEN:
> -       case CEPH_MDS_SESSION_HUNG:
> -       case CEPH_MDS_SESSION_OPENING:
> -               mutex_lock(&mdsc->mutex);
> -               if (s->s_mds >= mdsc->max_sessions ||
> -                   mdsc->sessions[s->s_mds] != s ||
> -                   s->s_state != session_state) {
> -                       pr_info_client(mdsc->fsc->client,
> -                                      "mds%d state changed to %s during peer reset\n",
> -                                      s->s_mds,
> -                                      ceph_session_state_name(s->s_state));
> -                       mutex_unlock(&mdsc->mutex);
> -                       return;
> -               }
> -
> -               ceph_get_mds_session(s);
> -               s->s_state = CEPH_MDS_SESSION_CLOSED;
> -               __unregister_session(mdsc, s);
> -               __wake_requests(mdsc, &s->s_waiting);
> -               mutex_unlock(&mdsc->mutex);
> -
> -               mutex_lock(&s->s_mutex);
> -               cleanup_session_requests(mdsc, s);
> -               remove_session_caps(s);
> -               mutex_unlock(&s->s_mutex);
> -
> -               wake_up_all(&mdsc->session_close_wq);
> -
> -               mutex_lock(&mdsc->mutex);
> -               kick_requests(mdsc, s->s_mds);
> -               mutex_unlock(&mdsc->mutex);
> -
> -               ceph_put_mds_session(s);
> -               break;
> -       case CEPH_MDS_SESSION_CLOSED:
> -       case CEPH_MDS_SESSION_REJECTED:
> -               break;
> -       default:
> +       rc = send_mds_reconnect(mdsc, s);
> +       if (rc && rc != -ESTALE && rc != -ENOENT)
>                 pr_warn_client(mdsc->fsc->client,
> -                              "mds%d peer reset in unexpected state %s\n",
> -                              s->s_mds,
> -                              ceph_session_state_name(session_state));
> -               break;
> -       }
> +                              "mds%d reconnect failed: %d\n", s->s_mds, rc);
>  }
>
>  static void mds_dispatch(struct ceph_connection *con, struct ceph_msg *msg)
> --
> 2.56.0
>