[PATCH] ceph: reconnect on peer reset even if the mdsmap says the MDS is active

[Sun Yuechi]

Since commit 39fe30315893 ("ceph: harden send_mds_reconnect and handle
active-MDS peer reset"), mds_peer_reset() tears the session down instead
of reconnecting when our mdsmap shows the rank past RECONNECT.

That map can be stale.  An MDS killed and restarted on the same address
resets our connection before the monitors report the rank as failed, so
the session is torn down while the map still says active.  When a
standby then takes over the rank in up:reconnect, check_new_map() has
no session left to reconnect, the new MDS evicts and blocklists us
after mds_reconnect_timeout, and the mount returns EACCES until it is
unmounted:

  libceph: mds0 (2)10.89.20.12:6800 session reset
  ceph: [...]: mds0 closed our session
  (mds) evicting unresponsive client ceph-c1 (14900), after waiting
        49.7175 seconds during MDS startup

Send the reconnect and let the MDS decide, as before that commit.  A
daemon restarted on the old address has a new nonce, so the reconnect
does not get through, and check_new_map() reconnects once a map carries
the rank's new address.  An MDS that really is active replies with
CLOSE, and handle_session() unregisters the session so that new
requests open a fresh one, which is what that commit intended.

Skip the warning for -ESTALE and -ENOENT: send_mds_reconnect() already
logs those, and the removed code was silent for closed and rejected
sessions.

Tested on riscv64 (7.2.6 kernel, Ceph 21, one active MDS with a
standby-replay): SIGKILL the active MDS while dropping the client's
monitor traffic until it logs "closed our session".  Evicted in 4 of 4
runs without the patch, "reconnect success" in 3 of 3 with it.  Eviction
by an active MDS without blocklist still recovers with a new session,
2 of 2 runs.

Fixes: 39fe30315893 ("ceph: harden send_mds_reconnect and handle active-MDS peer reset")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Sun Yuechi <sunyuechi@xxxxxxxxxxx>
---
 fs/ceph/mds_client.c | 84 +++++---------------------------------------
 1 file changed, 9 insertions(+), 75 deletions(-)

diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index d4ef8a355a48..d23250af0750 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -7236,7 +7236,7 @@ static void mds_peer_reset(struct ceph_connection *con)
 {
 	struct ceph_mds_session *s = con->private;
 	struct ceph_mds_client *mdsc = s->s_mdsc;
-	int session_state;
+	int rc;
 
 	pr_warn_client(mdsc->fsc->client, "mds%d closed our session\n",
 		       s->s_mds);
@@ -7246,82 +7246,16 @@ static void mds_peer_reset(struct ceph_connection *con)
 		return;
 
 	/*
-	 * Only reconnect if MDS is in its RECONNECT phase.  An MDS past
-	 * RECONNECT (REJOIN, CLIENTREPLAY, ACTIVE) will reject reconnect
-	 * attempts, so those states fall through to session teardown below.
+	 * Our mdsmap may be stale: an MDS restarted on the same address
+	 * resets the connection before we learn that the rank failed.
+	 * Send the reconnect and let the MDS decide; one that is past
+	 * RECONNECT replies with CLOSE and handle_session() closes the
+	 * session so that new requests open a fresh one.
 	 */
-	if (ceph_mdsmap_get_state(mdsc->mdsmap, s->s_mds) == CEPH_MDS_STATE_RECONNECT) {
-		int rc = send_mds_reconnect(mdsc, s);
-
-		if (rc)
-			pr_warn_client(mdsc->fsc->client,
-				       "mds%d reconnect failed: %d\n",
-				       s->s_mds, rc);
-		return;
-	}
-
-	/*
-	 * MDS is active (past RECONNECT).  It will not accept a
-	 * CLIENT_RECONNECT from us, so tear the session down locally
-	 * and let new requests re-open a fresh session.
-	 *
-	 * Snapshot session state with READ_ONCE, then revalidate under
-	 * mdsc->mutex before acting.  The subsequent mdsc->mutex
-	 * section rechecks s_state to catch concurrent transitions, so
-	 * the lockless snapshot here is safe.  s->s_mutex is taken
-	 * separately for cleanup after unregistration, which avoids
-	 * introducing a new s->s_mutex + mdsc->mutex nesting.
-	 */
-	session_state = READ_ONCE(s->s_state);
-
-	switch (session_state) {
-	case CEPH_MDS_SESSION_RESTARTING:
-	case CEPH_MDS_SESSION_RECONNECTING:
-	case CEPH_MDS_SESSION_CLOSING:
-	case CEPH_MDS_SESSION_OPEN:
-	case CEPH_MDS_SESSION_HUNG:
-	case CEPH_MDS_SESSION_OPENING:
-		mutex_lock(&mdsc->mutex);
-		if (s->s_mds >= mdsc->max_sessions ||
-		    mdsc->sessions[s->s_mds] != s ||
-		    s->s_state != session_state) {
-			pr_info_client(mdsc->fsc->client,
-				       "mds%d state changed to %s during peer reset\n",
-				       s->s_mds,
-				       ceph_session_state_name(s->s_state));
-			mutex_unlock(&mdsc->mutex);
-			return;
-		}
-
-		ceph_get_mds_session(s);
-		s->s_state = CEPH_MDS_SESSION_CLOSED;
-		__unregister_session(mdsc, s);
-		__wake_requests(mdsc, &s->s_waiting);
-		mutex_unlock(&mdsc->mutex);
-
-		mutex_lock(&s->s_mutex);
-		cleanup_session_requests(mdsc, s);
-		remove_session_caps(s);
-		mutex_unlock(&s->s_mutex);
-
-		wake_up_all(&mdsc->session_close_wq);
-
-		mutex_lock(&mdsc->mutex);
-		kick_requests(mdsc, s->s_mds);
-		mutex_unlock(&mdsc->mutex);
-
-		ceph_put_mds_session(s);
-		break;
-	case CEPH_MDS_SESSION_CLOSED:
-	case CEPH_MDS_SESSION_REJECTED:
-		break;
-	default:
+	rc = send_mds_reconnect(mdsc, s);
+	if (rc && rc != -ESTALE && rc != -ENOENT)
 		pr_warn_client(mdsc->fsc->client,
-			       "mds%d peer reset in unexpected state %s\n",
-			       s->s_mds,
-			       ceph_session_state_name(session_state));
-		break;
-	}
+			       "mds%d reconnect failed: %d\n", s->s_mds, rc);
 }
 
 static void mds_dispatch(struct ceph_connection *con, struct ceph_msg *msg)
-- 
2.56.0