From: Ding Tianhong
Date: Wed Jul 12 2017 - 05:49:46 EST

On 2017/7/11 8:01, Casey Leedom wrote:
> Hey Alexander,
> Okay, I understand your point regarding the "most likely scenario" being
> TLPs directed upstream to the Root Complex. But I'd still like to make sure
> that we have an agreed upon API/methodology for doing Peer-to-Peer with
> Relaxed Ordering and no Relaxed Ordering to the Root Complex. I don't see
> how the proposed APIs can be used in that fashion.
> Right now the proposed change for cxgb4 is for it to test its own PCIe
> Capability Device Control[Relaxed Ordering Enable] in order to use that
> information to program the Chelsio Hardware to emit/not emit upstream TLPs
> with the Relaxed Ordering Attribute set. But if we're going to have the
> mixed mode situation I describe, the PCIe Capability Device Control[Relaxed
> Ordering Enable] will have to be set which means that we'll be programming
> the Chelsio Hardware to send upstream TLPs with Relaxed Ordering Enable to
> the Root Complex which is what we were trying to avoid in the first place ...
> [[ And, as I noted on Friday evening, the currect cxgb4 Driver hardwires
> the Relaxed Ordering Enable on early dureing device probe, so that
> would minimally need to be addressed even if we decide that we don't
> ever want to support mixed mode Relaxed Ordering. ]]
> We need some method of telling the Chelsio Driver that it should/shouldn't
> use Relaxed Ordering with TLPs directed at the Root Complex. And the same
> is true for a Peer PCIe Device.
> It may be that we should approach this from the completely opposite
> direction and instead of having quirks which identify problematic devices,
> have quirks which identify devices which would benefit from the use of
> Relaxed Ordering (if the sending device supports that). That is, assume the
> using Relaxed Ordering shouldn't be done unless the target device says "I
> love Relaxed Ordering TLPs" ... In such a world, an NVMe or a Graphics
> device might declare love of Relaxed Ordering and the same for a SPARC Root
> Complex (I think that was your example).
> By the way, the sole example of Data Corruption with Relaxed Ordering is
> the AMD A1100 ARM SoC and AMD appears to have given up on that almost as
> soon as it was released. So what we're left with currently is a performance
> problem on modern Intel CPUs ... (And hopefully we'll get a Technical
> Publication on that issue fairly soon.)
> Casey

Hi Casey:

After the long discuss, I think If the PCIe Capability Device Control[Relaxed Ordering
Enable] to be cleared when the platform's RC has some problematic for RO didn't break
anything in your driver, I think you could choose to check the
(!pci_dev_should_disable_relaxed_ordering(root)) in the code to to enable
ROOT_NO_RELAXED_ORDERING for your adapter, and enable the PCIe Capability Device Control
[Relaxed Ordering Enable] bit when you need it, I think we don't have much gap here.
And we could leave the pear-to-pear situation to be fixed later.


> .