Re: [PATCH] RDMA/irdma: Avoid free the non-cqp_request scratch

[Leon Romanovsky]

On Fri, Nov 24, 2023 at 10:28:58PM +0800, Shifeng Li wrote:
> On 2023/11/22 1:25, Ismail, Mustafa wrote:
> > > -----Original Message-----
> > > From: Shifeng Li <lishifeng1992@xxxxxxx>
> > > Sent: Monday, November 20, 2023 2:31 AM
> > > To: Ismail, Mustafa <mustafa.ismail@xxxxxxxxx>
> > > Cc: Saleem, Shiraz <shiraz.saleem@xxxxxxxxx>; jgg@xxxxxxxx;
> > > leon@xxxxxxxxxx; linux-rdma@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> > > Ding, Hui <dinghui@xxxxxxxxxxxxxx>; Shifeng Li <lishifeng1992@xxxxxxx>
> > > Subject: [PATCH] RDMA/irdma: Avoid free the non-cqp_request scratch
> > > 
> > > When creating ceq_0 during probing irdma, cqp.sc_cqp will be sent as a
> > > cqp_request to cqp->sc_cqp.sq_ring. If the request is pending when removing
> > > the irdma driver or unplugging its aux device, cqp.sc_cqp will be dereferenced
> > > as wrong struct in irdma_free_pending_cqp_request().
> > > 
> > > crash> bt 3669
> > > PID: 3669   TASK: ffff88aef892c000  CPU: 28  COMMAND: "kworker/28:0"
> > >   #0 [fffffe0000549e38] crash_nmi_callback at ffffffff810e3a34
> > >   #1 [fffffe0000549e40] nmi_handle at ffffffff810788b2
> > >   #2 [fffffe0000549ea0] default_do_nmi at ffffffff8107938f
> > >   #3 [fffffe0000549eb8] do_nmi at ffffffff81079582
> > >   #4 [fffffe0000549ef0] end_repeat_nmi at ffffffff82e016b4
> > >      [exception RIP: native_queued_spin_lock_slowpath+1291]
> > >      RIP: ffffffff8127e72b  RSP: ffff88aa841ef778  RFLAGS: 00000046
> > >      RAX: 0000000000000000  RBX: ffff88b01f849700  RCX: ffffffff8127e47e
> > >      RDX: 0000000000000000  RSI: 0000000000000004  RDI: ffffffff83857ec0
> > >      RBP: ffff88afe3e4efc8   R8: ffffed15fc7c9dfa   R9: ffffed15fc7c9dfa
> > >      R10: 0000000000000001  R11: ffffed15fc7c9df9  R12: 0000000000740000
> > >      R13: ffff88b01f849708  R14: 0000000000000003  R15: ffffed1603f092e1
> > >      ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0000
> > > --- <NMI exception stack> ---
> > >   #5 [ffff88aa841ef778] native_queued_spin_lock_slowpath at ffffffff8127e72b
> > >   #6 [ffff88aa841ef7b0] _raw_spin_lock_irqsave at ffffffff82c22aa4
> > >   #7 [ffff88aa841ef7c8] __wake_up_common_lock at ffffffff81257363
> > >   #8 [ffff88aa841ef888] irdma_free_pending_cqp_request at ffffffffa0ba12cc
> > > [irdma]
> > >   #9 [ffff88aa841ef958] irdma_cleanup_pending_cqp_op at ffffffffa0ba1469
> > > [irdma]
> > >   #10 [ffff88aa841ef9c0] irdma_ctrl_deinit_hw at ffffffffa0b2989f [irdma]
> > >   #11 [ffff88aa841efa28] irdma_remove at ffffffffa0b252df [irdma]
> > >   #12 [ffff88aa841efae8] auxiliary_bus_remove at ffffffff8219afdb
> > >   #13 [ffff88aa841efb00] device_release_driver_internal at ffffffff821882e6
> > >   #14 [ffff88aa841efb38] bus_remove_device at ffffffff82184278
> > >   #15 [ffff88aa841efb88] device_del at ffffffff82179d23
> > >   #16 [ffff88aa841efc48] ice_unplug_aux_dev at ffffffffa0eb1c14 [ice]
> > >   #17 [ffff88aa841efc68] ice_service_task at ffffffffa0d88201 [ice]
> > >   #18 [ffff88aa841efde8] process_one_work at ffffffff811c589a
> > >   #19 [ffff88aa841efe60] worker_thread at ffffffff811c71ff
> > >   #20 [ffff88aa841eff10] kthread at ffffffff811d87a0
> > >   #21 [ffff88aa841eff50] ret_from_fork at ffffffff82e0022f
> > > 
> > > Fixes: 44d9e52977a1 ("RDMA/irdma: Implement device initialization
> > > definitions")
> > > Signed-off-by: Shifeng Li <lishifeng1992@xxxxxxx>
> > > ---
> > >   drivers/infiniband/hw/irdma/utils.c | 2 +-
> > >   1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/infiniband/hw/irdma/utils.c
> > > b/drivers/infiniband/hw/irdma/utils.c
> > > index 445e69e86409..222ec1f761a1 100644
> > > --- a/drivers/infiniband/hw/irdma/utils.c
> > > +++ b/drivers/infiniband/hw/irdma/utils.c
> > > @@ -541,7 +541,7 @@ void irdma_cleanup_pending_cqp_op(struct
> > > irdma_pci_f *rf)
> > >   	for (i = 0; i < pending_work; i++) {
> > >   		cqp_request = (struct irdma_cqp_request *)(unsigned long)
> > >   				      cqp->scratch_array[wqe_idx];
> > > -		if (cqp_request)
> > > +		if (cqp_request && cqp_request != (struct irdma_cqp_request
> > > +*)&cqp->sc_cqp)
> > >   			irdma_free_pending_cqp_request(cqp, cqp_request);
> > >   		wqe_idx = (wqe_idx + 1) % IRDMA_RING_SIZE(cqp-
> > > > sc_cqp.sq_ring);
> > >   	}
> > > --
> > > 2.25.1
> > 
> > Hi Li,
> > 
> > Could you describe how you hit this issue? It seems like the probe would not have completed successfully if the create cceq request was still pending.
> > 
> > A better fix might be to set the scratch to 0 in this case: In irdma_create_ceq(), pass 0 for scratch in call to irdma_sc_cceq_create(&iwceq->sc_ceq, 0), since it doesn't appear to be used in this case. Is it possible to test this fix? Thanks!
> 
> 1. The firmware of net card or the hardware may enter into some kind of error state. And I observed
> that the sq_ring of sc_cqp is full through crash. This means that all the 2048 cqp_requests were pending.
> 
> crash> struct irdma_cqp.sc_cqp ffff88afe3e4eee8
>   sc_cqp = {
>     ...
>     sq_ring = {
>       head = 91,  (The queue head caught up with the queue tail.)
>       tail = 92,
>       size = 2048
>     },
>     ...
>   }
> 
> 2. The issue is not reproducible. But I have tested the solution that pass 0 for scratch in
> call to irdma_sc_cceq_create(&iwceq->sc_ceq, 0) in irdma_create_ceq(). It works well.

Will you submit v2?

Thanks

> 
> Thanks!
> 
>