Re: [Bug report] NULL pointer dereference in frwr_unmap_sync()

[Dai Ngo]

On 3/9/25 12:40 PM, Dai Ngo wrote:
> Hi Nan,
>
> Can you try the attached patch with 6.14-rc4?
>
> This patch adds a spinlock to protect the rl_free_mrs and 
> rl_registered list.
> I have seen list corruption in our RDMA testing but the problem is 
> hard to
> reproduce so I have not submitted this patch to upstream.

Any update on testing of this patch?

Thanks,
-Dai

> Thanks,
> -Dai
>
> On 3/5/25 6:40 PM, Li Nan wrote:
> > 在 2025/3/5 22:02, Chuck Lever 写道:
> > > On 3/4/25 9:43 PM, Li Nan wrote:
> > > > We found a following problem in kernel 5.10, and the same problem 
> > > > should
> > > > exist in mainline:
> > > >
> > > > During NFS mount using 'soft' option over RoCE network, we observed 
> > > > kernel
> > > > crash with below trace when network issues occur 
> > > > (congestion/disconnect):
> > > >    nfs: server 10.10.253.211 not responding, timed out
> > > >    BUG: kernel NULL pointer dereference, address: 00000000000000a0
> > > >    RIP: 0010:frwr_unmap_sync+0x77/0x200 [rpcrdma]
> > > >    Call Trace:
> > > >     ? __die_body.cold+0x8/0xd
> > > >     ? no_context+0x155/0x230
> > > >     ? __bad_area_nosemaphore+0x52/0x1a0
> > > >     ? exc_page_fault+0x2dc/0x550
> > > >     ? asm_exc_page_fault+0x1e/0x30
> > > >     ? frwr_unmap_sync+0x77/0x200 [rpcrdma]
> > > >     xprt_release+0x9e/0x1a0 [sunrpc]
> > > >     rpc_release_resources_task+0xe/0x50 [sunrpc]
> > > >     rpc_release_task+0x19/0xa0 [sunrpc]
> > > >     rpc_async_schedule+0x29/0x40 [sunrpc]
> > > >     process_one_work+0x1b2/0x350
> > > >     worker_thread+0x49/0x310
> > > >     ? rescuer_thread+0x380/0x380
> > > >     kthread+0xfb/0x140
> > > >
> > > > Problem analysis:
> > > > The crash happens in frwr_unmap_sync() when accessing 
> > > > req->rl_registered
> > > > list, caused by either NULL pointer or accessing freed MR resources.
> > > > There's a race condition between:
> > > > T1
> > > > __ib_process_cq
> > > >   wc->wr_cqe->done (frwr_wc_localinv)
> > > >    rpcrdma_flush_disconnect
> > > >     rpcrdma_force_disconnect
> > > >      xprt_force_disconnect
> > > >       xprt_autoclose
> > > >        xprt_rdma_close
> > > >         rpcrdma_xprt_disconnect
> > > >          rpcrdma_reqs_reset
> > > >           frwr_reset
> > > >            rpcrdma_mr_pop(&req->rl_registered)
> > > > T2
> > > > rpc_async_schedule
> > > >   rpc_release_task
> > > >    rpc_release_resources_task
> > > >     xprt_release
> > > >      xprt_rdma_free
> > > >       frwr_unmap_sync
> > > >        rpcrdma_mr_pop(&req->rl_registered)
> > > >                     This problem also exists in function 
> > > > rpcrdma_mrs_destroy().
> > >
> > > Dai, is this the same as the system test problem you've been looking 
> > > at?
> >
> > Thank you for looking into it. Is there a patch that needs to be 
> > tested? We
> > are happy to help with the testing.