Re: [PATCH v2] PCI/EDR: Clear PCIe Device Status errors after EDR error recovery

[Sathyanarayanan Kuppuswamy]

Hi Bjorn,

Thanks for the review.

On 3/29/23 3:09 PM, Bjorn Helgaas wrote:
> [+cc Jonathan, author of 068c29a248b6]
> 
> On Wed, Mar 15, 2023 at 04:54:49PM -0700, Kuppuswamy Sathyanarayanan wrote:
>> Commit 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
>> OS owns AER") adds support to clear error status in the Device Status
>> Register(DEVSTA) only if OS owns the AER support. But this change
>> breaks the requirement of the EDR feature which requires OS to cleanup
>> the error registers even if firmware owns the control of AER support.
>>
>> More details about this requirement can be found in PCIe Firmware
>> specification v3.3, Table 4-6 Interpretation of the _OSC Control Field.
>> If the OS supports the Error Disconnect Recover (EDR) feature and
>> firmware sends the EDR event, then during the EDR recovery window, OS
>> is responsible for the device error recovery and holds the ownership of
>> the following error registers.
>>
>> • Device Status Register
>> • Uncorrectable Error Status Register
>> • Correctable Error Status Register
>> • Root Error Status Register
>> • RP PIO Status Register
>>
>> So call pcie_clear_device_status() in edr_handle_event() if the error
>> recovery is successful.
> 
> IIUC, after ac1c8e35a326 ("PCI/DPC: Add Error Disconnect Recover (EDR)
> support") appeared in v5.7-rc1, DEVSTA was always cleared in this path:
> 
>   edr_handle_event
>     pcie_do_recovery
>       pcie_clear_device_status
> 
> After 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if
> OS owns AER") appeared in v5.9-rc1, we only clear DEVSTA if the OS
> owns the AER Capability:
> 
>   edr_handle_event
>     pcie_do_recovery
>       if (pcie_aer_is_native(dev))      # <-- new test
>         pcie_clear_device_status
> 
> So in the case where the OS does *not* own AER, and it receives an EDR
> notification, DEVSTA is not cleared when it should be.  Right?

Correct.

> 
> I assume we should have a Fixes: tag here, since this patch should be
> backported to every kernel that contains 068c29a248b6.  Possibly even
> a stable tag, although it's arguable whether it's "critical" per
> Documentation/process/stable-kernel-rules.rst.

Yes. But this error is only reproducible in the EDR use case. So I am not sure
whether it can be considered a critical fix. 

Fixes: 068c29a248b6 ("PCI/ERR: Clear PCIe Device Status errors only if OS owns AER")


> 
>> Reported-by: Tsaur Erwin <erwin.tsaur@xxxxxxxxx>
> 
> I assume this report was internal, and there's no mailing list post or
> bugzilla issue URL we can include here?

Yes.

> 
>> Signed-off-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@xxxxxxxxxxxxxxx>
>> ---
>>
>> Changes since v1:
>>  * Rebased on top of v6.3-rc1.
>>  * Fixed a typo in pcie_clear_device_status().
>>
>>  drivers/pci/pcie/edr.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/pci/pcie/edr.c b/drivers/pci/pcie/edr.c
>> index a6b9b479b97a..87734e4c3c20 100644
>> --- a/drivers/pci/pcie/edr.c
>> +++ b/drivers/pci/pcie/edr.c
>> @@ -193,6 +193,7 @@ static void edr_handle_event(acpi_handle handle, u32 event, void *data)
>>  	 */
>>  	if (estate == PCI_ERS_RESULT_RECOVERED) {
>>  		pci_dbg(edev, "DPC port successfully recovered\n");
>> +		pcie_clear_device_status(edev);
> 
> It's a little weird to work around a change inside pcie_do_recovery()
> by clearing it here, and that means we clear it twice in the AER
> native case, but I don't see any simpler way to do this, so this seems
> fine as the fix for the current issue.

In AER native case, edr_handle_event() will never be triggered. So it
won't be cleared twice.

Other way is to add a new parameter to pcie_do_recovery(..., edr) and use
it to conditionally call pcie_clear_device_status(). But I think current
way is less complex.

edr_handle_event
  pcie_do_recovery(..., edr=1)
    if (pcie_aer_is_native(dev) || edr)
      pcie_clear_device_status

> 
> Question though: in the AER native case, pcie_do_recovery() calls
> both:
> 
>   pcie_clear_device_status() and
>   pci_aer_clear_nonfatal_status()
> 
> In this patch, you only call pcie_clear_device_status().  Do you care
> about pci_aer_clear_nonfatal_status(), too?

Yes, we care about it. Since we call dpc_process_error() in EDR handler,
it will eventually clear error status via pci_aer_clear_nonfatal_status()
and pci_aer_clear_fatal_status() within dpc_process_error().

> 
> The overall design for clearing status has gotten pretty complicated
> as we've added error handling methods (firmware-first, DPC, EDR), and
> there are so many different places and cases that it's hard to be sure
> we do them all correctly.
> 
> I don't really know how to clean this up, so I'm just attaching my
> notes about the current state:

Good summary! I can see a lot of overlap in clearing
PCI_ERR_UNCOR_STATUS and PCI_EXP_DEVSTA.

> 
>   - AER native handling:
> 
>     handle_error_source
>       if (info->severity == AER_CORRECTABLE)
>         clear PCI_ERR_COR_STATUS              <--
>         if (pcie_aer_is_native(dev))
>           pdrv->err_handler->cor_error_detected()
>           pcie_clear_device_status
>             clear PCI_EXP_DEVSTA              <--
>       else
>         pcie_do_recovery
>           pcie_clear_device_status
>             clear PCI_EXP_DEVSTA              <--
>           pci_aer_clear_nonfatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
> 
>   - Firmware-first handling: status is cleared by firmware before
>     event is reported to OS via HEST
> 
>   - DPC native handling:
> 
>     dpc_handler
>       dpc_process_error
>         if (rp_extensions)
>           dpc_process_rp_pio_error
>             clear PCI_EXP_DPC_RP_PIO_STATUS   <--
>         else if (...)
>           pci_aer_clear_nonfatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
>           pci_aer_clear_fatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
>       pcie_do_recovery
>         if (AER native)
>           pcie_clear_device_status
>             clear PCI_EXP_DEVSTA              <--
>           pci_aer_clear_nonfatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
> 
>   - EDR event handling:
> 
>     edr_handle_event
>       dpc_process_error
>         if (rp_extensions)
>           dpc_process_rp_pio_error
>             clear PCI_EXP_DPC_RP_PIO_STATUS   <--
>         else if (...)
>           pci_aer_clear_nonfatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
>           pci_aer_clear_fatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
>       pci_aer_raw_clear_status
>         clear PCI_ERR_ROOT_STATUS             <--
>         clear PCI_ERR_COR_STATUS              <--
>         clear PCI_ERR_UNCOR_STATUS            <--
>       pcie_do_recovery
>         if (AER native)
>           pcie_clear_device_status
>             clear PCI_EXP_DEVSTA              <--
>           pci_aer_clear_nonfatal_status
>             clear PCI_ERR_UNCOR_STATUS        <--
>       if (PCI_ERS_RESULT_RECOVERED)
>         pcie_clear_device_status
>           clear PCI_EXP_DEVSTA                <--
> 
>>  		acpi_send_edr_status(pdev, edev, EDR_OST_SUCCESS);
>>  	} else {
>>  		pci_dbg(edev, "DPC port recovery failed\n");
>> -- 
>> 2.34.1
>>

-- 
Sathyanarayanan Kuppuswamy
Linux Kernel Developer