[PATCH] NFSv4/pNFS: Preserve layoutreturn seqid while lsegs drain

[Tim Menninger]

Commit e20772cbdf46 ("NFSv4/pNFS: Fix a layoutget livelock loop")
changed pnfs_set_plh_return_info() so that a zero seq argument is
replaced with the current layout stateid seqid. This ensures that a
pending layout return has a nonzero plh_return_seq and does not
continually invalidate newly acquired layout segments.

pnfs_cache_lseg_for_layoutreturn() already passed zero to this helper.
Before e20772cbdf46, that call updated the pending return iomode and
NFS_LAYOUT_RETURN_REQUESTED state, but did not change plh_return_seq.
After e20772cbdf46, it also snapshots the current layout stateid seqid.

This is problematic when an lseg that was already selected for return
takes time to drain its outstanding references. For example, suppose a
return is established with plh_return_seq S. While an lseg covered by
that return is draining, successful LAYOUTGET operations can advance the
layout stateid to T, where T is newer than S.

When the lseg's final reference is released, the path is:

    pnfs_put_lseg()
    -> pnfs_cache_lseg_for_layoutreturn()
       -> pnfs_set_plh_return_info(seq=0) # advances plh_return_seq
    -> pnfs_put_layout_hdr()
       -> pnfs_layoutreturn_before_put_layout_hdr()
          -> pnfs_layout_need_return()
             -> pnfs_mark_layout_stateid_return(seq=lo->plh_return_seq)
                -> pnfs_mark_matching_lsegs_return(seq)

The zero argument causes pnfs_cache_lseg_for_layoutreturn() to advance
plh_return_seq from S to T. pnfs_layout_need_return() then uses T as the
upper bound when selecting lsegs for return, so layout segments acquired
after the original return was requested can also be invalidated and have
their I/O cancelled.

Under sustained I/O, those cancellations cause more lsegs to drain,
whose final puts can advance the return boundary again. This can form a
positive feedback loop in which newly acquired layouts are repeatedly
invalidated and cancelled.

This was observed on a one-client one-DS system under sustained
direct-read I/O over pNFS/RDMA after restarting the nfs service on the
DS, where the repeated layout invalidation and cancellation caused
severe throughput collapse.

When the draining lseg is already covered by plh_return_seq, pass the
existing return seqid to pnfs_set_plh_return_info() instead of zero so
that caching the lseg does not move the return boundary. If there is no
return seqid yet, or the lseg is newer than the recorded boundary,
retain the existing behavior and snapshot the current layout stateid.

Fixes: e20772cbdf46 ("NFSv4/pNFS: Fix a layoutget livelock loop")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Tim Menninger <tmenninger@xxxxxxxxxxxxxxxx>
---
 fs/nfs/pnfs.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 4f9c0f639014..69b085482921 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -596,9 +596,17 @@ static bool
 pnfs_cache_lseg_for_layoutreturn(struct pnfs_layout_hdr *lo,
 		struct pnfs_layout_segment *lseg)
 {
+	u32 seq = lo->plh_return_seq;
+
 	if (test_and_clear_bit(NFS_LSEG_LAYOUTRETURN, &lseg->pls_flags) &&
 	    pnfs_layout_is_valid(lo)) {
-		pnfs_set_plh_return_info(lo, lseg->pls_range.iomode, 0);
+		/*
+		 * Avoid resnapshotting the current layout stateid for an lseg that
+		 * is already covered by the pending return.
+		 */
+		if (!seq || pnfs_seqid_is_newer(lseg->pls_seq, seq))
+			seq = 0;
+		pnfs_set_plh_return_info(lo, lseg->pls_range.iomode, seq);
 		list_move_tail(&lseg->pls_list, &lo->plh_return_segs);
 		return true;
 	}
-- 
2.34.1