Re: [PATCH v3] scsi: lpfc: Fix race conditions in ELS retry handling

[Hannes Reinecke]

On 11/10/26 9:16 PM, Kyle Mahlkuch wrote:
> This patch addresses a race condition in the lpfc driver's ELS retry
> event handling that can lead to a use-after-free in the
> lpfc_cancel_retry_delay_tmo(), where the NLP_DELAY_TMO flag is cleared
> before acquiring the lock to check if the retry event is queued. This
> creates a window where the timer can be rescheduled and fire, causing
> both the cancel path and the worker thread to release the same reference,
> resulting in a double-put and use-after-free.
>
> Moved capture ndlp and clear evt_arg1 under hbalock while dequeuing
> LPFC_EVT_ELS_RETRY in lpfc_work_list_done().
>
> Signed-off-by: Kyle Mahlkuch <kmahlkuc@xxxxxxxxxxxxx>
> ---
> Changes since v2:
> - Capture ndlp and clear evt_arg1 under hbalock while dequeuing
>    LPFC_EVT_ELS_RETRY in lpfc_work_list_done(). A requeue by
>    lpfc_els_retry_delay() could cause a NULL dereference in the worker.
>    (AI review)
> - Whitespace and line-wrapping cleanup.
>
> Link: https://lore.kernel.org/lkml/29073ebc-b616-475b- 
> bc05-384da09fbb54@xxxxxxxxxxxxx/T/#u
>
> Changes since v1:
> - Leave test_and_clear_bit op and timer_delete_sync at the beginning of
>    lpfc_cancel_retry_delay_tmo() outside of lock. (Paul Ely)
>
> Link: https://lore.kernel.org/lkml/1c8a764c-fce1-4ce1- 
> b797-47ac328cf3f2@xxxxxxxxxxxxx/T/#u
> ---
>   drivers/scsi/lpfc/lpfc_els.c     | 29 +++++++++++++++++++++++------
>   drivers/scsi/lpfc/lpfc_hbadisc.c | 11 ++++++++++-
>   2 files changed, 33 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c
> index 6f6394a0047c..98d1824aefe1 100644
> --- a/drivers/scsi/lpfc/lpfc_els.c
> +++ b/drivers/scsi/lpfc/lpfc_els.c
> @@ -4477,18 +4477,35 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, 
> uint8_t retry)
>   void
>   lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct 
> lpfc_nodelist *nlp)
>   {
> -    struct lpfc_work_evt *evtp;
> +    struct lpfc_hba *phba = vport->phba;
> +    struct lpfc_work_evt *evtp = &nlp->els_retry_evt;
> +    struct lpfc_nodelist *arg_ndlp = NULL;
> +    unsigned long flags;
>
>       if (!test_and_clear_bit(NLP_DELAY_TMO, &nlp->nlp_flag))
>           return;
>       timer_delete_sync(&nlp->nlp_delayfunc);
>       nlp->nlp_last_elscmd = 0;
> -    if (!list_empty(&nlp->els_retry_evt.evt_listp)) {
> -        list_del_init(&nlp->els_retry_evt.evt_listp);
> -        /* Decrement nlp reference count held for the delayed retry */
> -        evtp = &nlp->els_retry_evt;
> -        lpfc_nlp_put((struct lpfc_nodelist *)evtp->evt_arg1);
> +
> +    /*
> +     * If retry event is queued, remove it under lock and consume
> +     * its payload to prevent double-put. The event holds a reference
> +     * to the nodelist that must be released exactly once.
> +     */
> +    spin_lock_irqsave(&phba->hbalock, flags);
> +    if (!list_empty(&evtp->evt_listp)) {
> +        list_del_init(&evtp->evt_listp);
> +        /* Take local copy and clear to prevent reuse */
> +        arg_ndlp = (struct lpfc_nodelist *)evtp->evt_arg1;
> +        evtp->evt_arg1 = NULL;
> +        smp_wmb(); /* Ensure NULL visible before unlock */
>       }
> +    spin_unlock_irqrestore(&phba->hbalock, flags);
> +
> +    /* Drop the event-held reference outside the lock */
> +    if (arg_ndlp)
> +        lpfc_nlp_put(arg_ndlp);
> +
>       if (test_and_clear_bit(NLP_NPR_2B_DISC, &nlp->nlp_flag)) {
>           if (vport->num_disc_nodes) {
>               if (vport->port_state < LPFC_VPORT_READY) {
> diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/ 
> lpfc_hbadisc.c
> index 4c673dffa671..75457f19d4ea 100644
> --- a/drivers/scsi/lpfc/lpfc_hbadisc.c
> +++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
> @@ -840,12 +840,21 @@ lpfc_work_list_done(struct lpfc_hba *phba)
>       while (!list_empty(&phba->work_list)) {
>           list_remove_head((&phba->work_list), evtp, typeof(*evtp),
>                    evt_listp);
> +        /*
> +         * The ELS retry event can be requeued by lpfc_els_retry_delay()
> +         * as soon as hbalock is dropped. Consume the payload within the
> +         * hbalock to prevent reuse or double-put.
> +         */
> +        if (evtp->evt == LPFC_EVT_ELS_RETRY) {
> +            ndlp = (struct lpfc_nodelist *)(evtp->evt_arg1);
> +            evtp->evt_arg1 = NULL;
> +        }
>           spin_unlock_irq(&phba->hbalock);
>           hba_pci_err = test_bit(HBA_PCI_ERR, &phba->bit_flags);
>           free_evt = 1;
>           switch (evtp->evt) {
>           case LPFC_EVT_ELS_RETRY:
> -            ndlp = (struct lpfc_nodelist *) (evtp->evt_arg1);
> +            /* ndlp was already captured under hbalock */
>               if (!hba_pci_err) {
>                   lpfc_els_retry_delay_handler(ndlp);
>                   free_evt = 0; /* evt is part of ndlp */

Good catch.

Reviewed-by: Hannes Reinecke <hare@xxxxxxxxxx>

Cheers,

Hannes
--
Dr. Hannes Reinecke                  Kernel Storage Architect
hare@xxxxxxx                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich