Re: [PATCH v2] pNFS: deadlock in pnfs_send_layoutreturn

[Benjamin Coddington]

On 17 Apr 2026, at 3:17, Ben Roberts wrote:

> Hi Ben,
>
>> Did you reproduce and diagnose this problem on a recent upstream kernel
>> version?
>
> I'm not able to switch the production systems to a more recent kernel at
> this time, and don't have a reliable way to reproduce the issue in the
> wild without risking production systems back on an unpatched kernel. The
> best evidence I have that this patch is needed, is that we were seeing
> this deadlock occur repeatedly under high loads and memory pressure last
> year before applying this patch locally. It was rolled out to all
> production systems in early Jan and we have not seen a single reoccurrence
> since. The relevant code paths look similar between the modified EL9
> kernel and the current git HEAD.
>
> I've spent all this week trying to devise a precise reproduction (with a
> lot of help from an LLM since I'm not that familar with kernel
> development) on both 7.0.0-rc7 (9a9c8ce300cd) and 5.14.0-611.9.1 kernels
> to definitively prove this patch is needed, but without success. The
> initial analysis suggested the deadlock might be triggered from a single
> process via a recursive call. This theory has been ruled out; all calling
> paths triggered by a single process are guarded in such a way that a
> recursive call does not happen. Revised theory for why this patch helps
> is that the deadlock is subject to an inter-process race, where one
> process suffers a memory allocation failure, causing a second process to
> become a victim of bad state, leading to an unserviceable RPC request.
> The timing appears to be very sensitive and hard to reproduce on demand.
>
> The LLM-assisted analysis follows, and to my non-expert eyes seems
> compelling.
>
> Thanks,
> Ben

The kzalloc failure is definitely a rarely-used/tested path, so its possible
there's an issue there no one has seen yet, but from what I can see it looks
like every call to pnfs_send_layoutreturn() first calls
pnfs_prepare_layoutreturn(), which already clears
NFS_LAYOUT_RETURN_REQUESTED.  I don't see how you can end up with another
proccess seeing the flag.

There's at least one body of work in this area that your systems
don't yet have:
https://lore.kernel.org/linux-nfs/20240613050055.854323-1-trond.myklebust@xxxxxxxxxxxxxxx/

Again, I strongly recommend engaging RH support here.
Ben