Re: [RFC PATCH 0/6] Add support for root port RAS error handling

[Li, Ming]

On 3/16/2024 2:21 AM, Dan Williams wrote:
> Li, Ming wrote:
> [..]
>>> I do expect direct-attach to be a predominant use case, but I want to
>>> make sure that the implementation at least does not make the switch port
>>> error handling case more difficult to implement.
>>
>> Hi Dan,
>>
>> Currently, A rough idea I have is that:
>> If a CXL switch connected to the CXL RP, there should be two cases,
>> 1. no CXL memory device connected to the switch, in this case, I'm not
>> sure whether CXL.cachemem protocol errors is still possibly happened
>> between RP and switch without CXL memory device. If not, maybe we
>> don't need to consider such case?
> 
> Protocol errors can happen between any 2 ports, just like PCI protocol
> errors.

Seems like if we want to handle CXL protocol error for such case on CXL driver side, we need some changes for CXL switch port/dport enumeration, I remembered that switch USP and DSP are enumerated only when a CXL memory device attached.

> 
>> 2. a CXL memory device connected to the switch. I think cxl_pci error
>> handler could also help to handle CXL.cachemem protocol errors
>> happened in switch USP/DSP.
> 
> No, for 2 reasons:
> 
> * The cxl_pci driver is only for general CXL type-3 memory
>   expanders. Even though no CXL.cache devices have upstream drivers they
>   do exist and they would experience protocol errors that the PCI core
>   needs to consider.
> 
> * When a switch is present it is possible to have a protocol error
>   between the switch upstream port and the root port, and not between
>   the switch downstream port and the endpoint.
> 
> The more I think about it I do not think it is appropriate for cxl_pci
> to be involved in clearing root port errors in the VH case, it only
> works the RCH case because of the way the device and the root-port get
> combined.

Thank you for your explanation. 

I think I need some time to consider a appropriate proposal. maybe we can register a CXL error handler into struct pci_driver during CXL port/dport enumeration for root port and switch cases, PCIe AER driver will invoke this registered CXL error handler to handle errors on CXL subsystem side. But I think this solution also meets above issue(current CXL subsystem won't enumerate CXL switch port/dport without CXL mem device).