Re: [PATCH v3] PCI: Lock the pci_cfg_wait queue for the consistency of data

[Bjorn Helgaas]

On Tue, Dec 10, 2019 at 11:15:27AM +0800, Xiang Zheng wrote:
> 7ea7e98fd8d0 ("PCI: Block on access to temporarily unavailable pci
> device") suggests that the "pci_lock" is sufficient, and all the
> callers of pci_wait_cfg() are wrapped with the "pci_lock".
> 
> However, since the commit cdcb33f98244 ("PCI: Avoid possible deadlock on
> pci_lock and p->pi_lock") merged, the accesses to the pci_cfg_wait queue
> are not safe anymore. This would cause kernel panic in a very low chance
> (See more detailed information from the below link). A "pci_lock" is
> insufficient and we need to hold an additional queue lock while read/write
> the wait queue.
> 
> So let's use the add_wait_queue()/remove_wait_queue() instead of
> __add_wait_queue()/__remove_wait_queue(). Also move the wait queue
> functionality around the "schedule()" function to avoid reintroducing
> the deadlock addressed by "cdcb33f98244".

I see that add_wait_queue() acquires the wq_head->lock, while
__add_wait_queue() does not.

But I don't understand why the existing pci_lock is insufficient.  
pci_cfg_wait is only used in pci_wait_cfg() and
pci_cfg_access_unlock().

In pci_wait_cfg(), both __add_wait_queue() and __remove_wait_queue()
are called while holding pci_lock, so that doesn't seem like the
problem.

In pci_cfg_access_unlock(), we have:

  pci_cfg_access_unlock
    wake_up_all(&pci_cfg_wait)
      __wake_up(&pci_cfg_wait, ...)
        __wake_up_common_lock(&pci_cfg_wait, ...)
	  spin_lock(&pci_cfg_wait->lock)
	  __wake_up_common(&pci_cfg_wait, ...)
	    list_for_each_entry_safe_from(...)
	      list_add_tail(...)                <-- problem?
	  spin_unlock(&pci_cfg_wait->lock)

Is the problem that the wake_up_all() modifies the pci_cfg_wait list
without holding pci_lock?

If so, I don't quite see how the patch below fixes it.  Oh, wait,
maybe I do ... by using add_wait_queue(), we protect the list using
the *same* lock used by __wake_up_common_lock.  Is that it?

> Signed-off-by: Xiang Zheng <zhengxiang9@xxxxxxxxxx>
> Cc: Heyi Guo <guoheyi@xxxxxxxxxx>
> Cc: Biaoxiang Ye <yebiaoxiang@xxxxxxxxxx>
> Link: https://lore.kernel.org/linux-pci/79827f2f-9b43-4411-1376-b9063b67aee3@xxxxxxxxxx/
> ---
> 
> v3:
>   Improve the commit subject and message.
> 
> v2:
>   Move the wait queue functionality around the "schedule()".
> 
> ---
>  drivers/pci/access.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/pci/access.c b/drivers/pci/access.c
> index 2fccb5762c76..09342a74e5ea 100644
> --- a/drivers/pci/access.c
> +++ b/drivers/pci/access.c
> @@ -207,14 +207,14 @@ static noinline void pci_wait_cfg(struct pci_dev *dev)
>  {
>  	DECLARE_WAITQUEUE(wait, current);
>  
> -	__add_wait_queue(&pci_cfg_wait, &wait);
>  	do {
>  		set_current_state(TASK_UNINTERRUPTIBLE);
>  		raw_spin_unlock_irq(&pci_lock);
> +		add_wait_queue(&pci_cfg_wait, &wait);
>  		schedule();
> +		remove_wait_queue(&pci_cfg_wait, &wait);
>  		raw_spin_lock_irq(&pci_lock);
>  	} while (dev->block_cfg_access);
> -	__remove_wait_queue(&pci_cfg_wait, &wait);
>  }
>  
>  /* Returns 0 on success, negative values indicate error. */
> -- 
> 2.19.1
> 
>