RE: How to force RC to forward p2p TLPs

[Eric Wehage]

There is no method to force an RC to forward requests p2p. RC's are also not required to support p2p and whether they do or not is implementation specific. There is also currently no PCIe based standard to determine whether an RC supports p2p or in what capacity. Also, even if p2p is supported, it might not be supported in a system that enables Virtualization (the Virtualization would have to setup p2p tables to allow it).

There is currently a PCIe ECR that has been submitted, but is undergoing review, that defines a PCIe capability for bridge devices (Root Ports and Switch Downstream Ports) that would allow them to advertise what level of p2p is supported by the device.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@xxxxxxxxxx] 
Sent: Friday, December 28, 2018 6:29 PM
To: yu <kkndyu@xxxxxxxxx>
Cc: linux-pci@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Logan Gunthorpe <logang@xxxxxxxxxxxx>; Stephen Bates <sbates@xxxxxxxxxxxx>; Jonathan Cameron <jonathan.cameron@xxxxxxxxxx>; Eric Wehage <Eric.Wehage@xxxxxxxxxx>; Alexander Duyck <alexander.h.duyck@xxxxxxxxxxxxxxx>
Subject: Re: How to force RC to forward p2p TLPs

[+cc Logan, Stephen, Jonathan, Eric, Alex]

On Sat, Dec 22, 2018 at 12:50:19PM +0800, yu wrote:
> Hi all,
> 
> We have a PCIE card which has a PEX8732 switch on-board,  and there 
> are two endpoint SOCs like graphic decoder behind the switch, and by 
> default the ACS is enabled in 8732.
> 
> We use the p2p DMA to transfer data between these two endpoint SOCs, 
> and if the host server is not enable ACS in BIOS, the p2p works well, 
> but when ACS is enabled in BIOS, the p2p is always failed. With the 
> help of a protocol analyzer, we can see that the TLP is redirected to 
> RC, and RC just discard it.
> 
> I tried to find how to make RC forward redirected TLP to its original 
> target, but nothing found, it seems this is highly related to the RC 
> vendors.
> 
> In the PCIE 4.0 spec, the section of the RC behavior of the p2p 
> request redirect  said that ''implementation-specific logic within the 
> RC that determines whether the request is directed towards its 
> original target, or blocked as an ACS Violation error. the algorithms 
> and specific controls for making this determination are not 
> architected by this spec''.
> 
> 
> So is there some spec or document to describe how to set the RC? Any 
> suggestion is appreciated.

Not that I'm aware of, but the folks I cc'd would know a lot more about this area.

Bjorn