Re: [PATCH 0/6] nfsd: layout recall and lease break fixes

[Chuck Lever]

Hi Daejun,

An nfsd thread must not block for more than a few milliseconds, and
never on something a client controls. A client can tie up thread
after thread until the server has no more threads to respond to
LAYOUTRETURN or DELEGRETURN, and then the server deadlocks.

For delegations we are safe. nfsd_open_break_lease() passes
O_NONBLOCK and the operation returns NFS4ERR_DELAY.

For layouts we are not. A WRITE or SETATTR goes into XFS,
xfs_break_leased_layouts() calls break_layout() blocking, and the
nfsd thread sleeps in __break_lease() until the lease is dropped or
the breaker times out. That dates from f52792f484ba, but it was
bounded by fs.lease-break-time, and a Linux client from v6.8 on
rarely hit it. It responded OLD_STATEID, the callback failed, the
lease was dropped, and the thread was back within a round trip.

This series makes that wait the common case and then unbounded.

- Patch 3 polls on OLD_STATEID, so the thread sleeps for as long as
  the client takes to drain its I/O, up to lease-break-time.

- Patch 2 waits on the fence script with UMH_WAIT_PROC while
  lm_breaker_timedout() keeps returning false, so the thread waits
  lease-break-time plus however long the script takes to reach the
  client.

- Patch 5 extends the wait of patch 3 to a client past the seqid.

I think the fix has to come first. Before the WRITE or SETATTR, call
break_layout() non-blocking to start the recall, and return
NFS4ERR_DELAY on -EWOULDBLOCK, as we do for delegations. Then the
fence worker and the poll loop can take as long as they need and no
nfsd thread waits on them.

Give that a try and let me know if that works or needs improvement.


-- 
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)