Re: [PATCH] libceph: reset OSD session when keepalive2 acks stop arriving

[Viacheslav Dubeyko]

On Tue, 2026-07-07 at 19:46 -0500, Chris J Arges wrote:
> handle_timeout() in osd_client.c sends CEPH_MSGR2_TAG_KEEPALIVE2 frames
> to OSDs with stalled requests, but libceph never verifies if the ACKs
> actually return.
> 
> Consequently, if an OSD messenger queue wedges while the underlying
> TCP socket remains ESTABLISHED, the client will block indefinitely in
> ceph_osdc_wait_request(), causing tasks to hang in D state.
> 
> Fix this by introducing a watchdog in handle_timeout() that checks
> ceph_con_keepalive_expired() against a new CEPH_OSD_PING_TIMEOUT (60s).
> On expiry, reset the sparse-read state, call reopen_osd(), and kick
> outstanding requests when the session is reopened.
> 
> Because OSD keepalives are only sent to OSDs with stalled requests,
> last_keepalive_ack can be stale on an otherwise healthy connection that
> has simply been idle. Track the start of each slow/probing episode per
> OSD and require the episode to last CEPH_OSD_PING_TIMEOUT before checking
> for an expired keepalive ack, so the watchdog only fires after we have
> been actively pinging.
> 
> Additionally, seed last_keepalive_ack to the current time in
> ceph_con_open() to prevent the watchdog from firing spuriously on fresh
> connections for both OSD and monitor clients.
> 
> Fixes: 8b9558aab853 ("libceph: use keepalive2 to verify the mon session is alive")
> Link: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issues_76202&d=DwICaQ&c=BSDicqBQBDjDI9RkVyTcHQ&r=q5bIm4AXMzc8NJu1_RGmnQ2fMWKq4Y4RAkElvUgSs00&m=8udRqH37aQXbOFlaT2eCFf9V5N2umZEpRHyjRWqPY9nOaaxnBb5-_kuP0k6YrT4O&s=4IAHOQNcSzsOsmpIm3nIsgqXGHCx-bxmgd1yDJ3m-VU&e= 
> Co-developed-by: Andrew DeMaria <ademaria@xxxxxxxxxxxxxx>
> Signed-off-by: Andrew DeMaria <ademaria@xxxxxxxxxxxxxx>
> Signed-off-by: Chris J Arges <carges@xxxxxxxxxxxxxx>
> ---
> This patch fixes an issue where the kernel rbd client and an OSD have an
> ESTABLISHED TCP connection, but keepalive2 ACKs stop returning from the
> OSD. When that happens, outstanding OSD requests can remain held by the
> client and callers can hang in D state. We were able to mitgiate this
> issue by using ss -K to kill the affected OSD TCP connection which
> reopened the OSD session.
> 
> We were able to reproduce this issue in production a few times, and
> synthetically by dropping OSD to rbd application frames while allowing
> TCP ACKs through.
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issues_76202&d=DwICaQ&c=BSDicqBQBDjDI9RkVyTcHQ&r=q5bIm4AXMzc8NJu1_RGmnQ2fMWKq4Y4RAkElvUgSs00&m=8udRqH37aQXbOFlaT2eCFf9V5N2umZEpRHyjRWqPY9nOaaxnBb5-_kuP0k6YrT4O&s=4IAHOQNcSzsOsmpIm3nIsgqXGHCx-bxmgd1yDJ3m-VU&e=  describes the same situation.
> 
> The following patch addresses this by creating a watchdog that resets the
> OSD session if this situation is detected.
> ---
>  include/linux/ceph/libceph.h    |  1 +
>  include/linux/ceph/osd_client.h |  1 +
>  net/ceph/messenger.c            |  3 +++
>  net/ceph/osd_client.c           | 23 ++++++++++++++++++++++-
>  4 files changed, 27 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/ceph/libceph.h b/include/linux/ceph/libceph.h
> index 63e0e2aa1ce9..1a117c5f1964 100644
> --- a/include/linux/ceph/libceph.h
> +++ b/include/linux/ceph/libceph.h
> @@ -74,6 +74,7 @@ struct ceph_options {
>   */
>  #define CEPH_MOUNT_TIMEOUT_DEFAULT	msecs_to_jiffies(60 * 1000)
>  #define CEPH_OSD_KEEPALIVE_DEFAULT	msecs_to_jiffies(5 * 1000)
> +#define CEPH_OSD_PING_TIMEOUT		msecs_to_jiffies(60 * 1000)

Could we reuse the user-configurable osd_keepalive_timeout (default 5s)? Why
exactly 60s has been selected?

Thanks,
Slava.

>  #define CEPH_OSD_IDLE_TTL_DEFAULT	msecs_to_jiffies(60 * 1000)
>  #define CEPH_OSD_REQUEST_TIMEOUT_DEFAULT 0  /* no timeout */
>  #define CEPH_READ_FROM_REPLICA_DEFAULT	0  /* read from primary */
> diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
> index 50b14a5661c7..52eb76e9d62a 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -94,6 +94,7 @@ struct ceph_osd {
>  	struct ceph_auth_handshake o_auth;
>  	unsigned long lru_ttl;
>  	struct list_head o_keepalive_item;
> +	unsigned long o_keepalive_stamp;
>  	struct mutex lock;
>  	struct ceph_sparse_read	o_sparse_read;
>  };
> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> index 34b3097b4c7b..f7776d83d506 100644
> --- a/net/ceph/messenger.c
> +++ b/net/ceph/messenger.c
> @@ -610,6 +610,9 @@ void ceph_con_open(struct ceph_connection *con,
>  
>  	memcpy(&con->peer_addr, addr, sizeof(*addr));
>  	con->delay = 0;      /* reset backoff memory */
> +
> +	ktime_get_real_ts64(&con->last_keepalive_ack);
> +
>  	mutex_unlock(&con->mutex);
>  	queue_con(con);
>  }
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 2ff00070c181..6d9530beb2e3 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -53,6 +53,7 @@ static void link_linger(struct ceph_osd *osd,
>  static void unlink_linger(struct ceph_osd *osd,
>  			  struct ceph_osd_linger_request *lreq);
>  static void clear_backoffs(struct ceph_osd *osd);
> +static void kick_osd_requests(struct ceph_osd *osd);
>  
>  #if 1
>  static inline bool rwsem_is_wrlocked(struct rw_semaphore *sem)
> @@ -3480,8 +3481,11 @@ static void handle_timeout(struct work_struct *work)
>  			mutex_unlock(&lreq->lock);
>  		}
>  
> -		if (found)
> +		if (found) {
>  			list_move_tail(&osd->o_keepalive_item, &slow_osds);
> +		} else {
> +			osd->o_keepalive_stamp = 0;
> +		}
>  	}
>  
>  	if (opts->osd_request_timeout) {
> @@ -3507,6 +3511,23 @@ static void handle_timeout(struct work_struct *work)
>  							struct ceph_osd,
>  							o_keepalive_item);
>  		list_del_init(&osd->o_keepalive_item);
> +
> +		/* Record start of ping timeout from the first slow tick. */
> +		if (!osd->o_keepalive_stamp) {
> +			osd->o_keepalive_stamp = jiffies;
> +		} else if (time_after_eq(jiffies,
> +				  osd->o_keepalive_stamp + CEPH_OSD_PING_TIMEOUT) &&
> +		    ceph_con_keepalive_expired(&osd->o_con,
> +					       CEPH_OSD_PING_TIMEOUT)) {
> +			pr_warn_ratelimited("osd%d not responding to keepalives, resetting session\n",
> +					    osd->o_osd);
> +			osd->o_sparse_op_idx = -1;
> +			ceph_init_sparse_read(&osd->o_sparse_read);
> +			if (!reopen_osd(osd))
> +				kick_osd_requests(osd);
> +			continue;
> +		}
> +
>  		ceph_con_keepalive(&osd->o_con);
>  	}
>  
> 
> ---
> base-commit: 0e35b9b6ec0ffcc5e23cbdec09f5c622ad532b53
> change-id: 20260707-fix-rbd-keepalives-664fe9266c35
> 
> Best regards,
> --  
> Chris J Arges <carges@xxxxxxxxxxxxxx>
>