Re: Support for configurable PCIe endpoint

[Kishon Vijay Abraham I]

Hi,

On Wednesday 17 August 2016 03:19 PM, Mingkai Hu wrote:
> 
> 
>> -----Original Message-----
>> From: Kishon Vijay Abraham I [mailto:kishon@xxxxxx]
>> Sent: Thursday, August 04, 2016 6:02 PM
>> To: Joao Pinto <Joao.Pinto@xxxxxxxxxxxx>; bhelgaas@xxxxxxxxxx; linux-
>> pci@xxxxxxxxxxxxxxx; arnd@xxxxxxxx; Jingoo Han <jingoohan1@xxxxxxxxx>;
>> Pratyush Anand <pratyush.anand@xxxxxxxxx>
>> Cc: Ley Foon Tan <lftan@xxxxxxxxxx>; Rob Herring <robh@xxxxxxxxxx>;
>> Tanmay Inamdar <tinamdar@xxxxxxx>; Roy Zang <tie-
>> fei.zang@xxxxxxxxxxxxx>; Mingkai Hu <mingkai.hu@xxxxxxxxxxxxx>;
>> Minghuan Lian <minghuan.Lian@xxxxxxxxxxxxx>; Richard Zhu
>> <Richard.Zhu@xxxxxxxxxxxxx>; Lucas Stach <l.stach@xxxxxxxxxxxxxx>;
>> Murali Karicheri <m-karicheri2@xxxxxx>; Thomas Petazzoni
>> <thomas.petazzoni@xxxxxxxxxxxxxxxxxx>; Jason Cooper
>> <jason@xxxxxxxxxxxxxx>; Thierry Reding <thierry.reding@xxxxxxxxx>;
>> Simon Horman <horms@xxxxxxxxxxxx>; Zhou Wang
>> <wangzhou1@xxxxxxxxxxxxx>; Gabriele Paoloni
>> <gabriele.paoloni@xxxxxxxxxx>; Stanimir Varbanov <svarbanov@mm-
>> sol.com>; David Daney <david.daney@xxxxxxxxxx>; linux-
>> kernel@xxxxxxxxxxxxxxx; linux-arm-kernel@xxxxxxxxxxxxxxxxxxx; linux-
>> omap@xxxxxxxxxxxxxxx; Carlos Palminha
>> <CARLOS.PALMINHA@xxxxxxxxxxxx>
>> Subject: Re: Support for configurable PCIe endpoint
>>
>> Hi,
>>
>> On Wednesday 03 August 2016 07:09 PM, Joao Pinto wrote:
>>> Hi Kishon,
>>>
>>> On 8/3/2016 7:03 AM, Kishon Vijay Abraham I wrote:
>>>> Hi,
>>>>
>>>> The PCIe controller present in TI's DRA7 SoC is capable of operating
>>>> either in Root Complex mode or Endpoint mode. (It uses Synopsys
>>>> Designware Core). I'd assume most of the PCIe controllers on other
>>>> platforms that use Designware core should also be capable to operate
>>>> in endpoint mode. But linux kernel right now supports only RC mode.
>>>>
>>>> PCIe endpoint support discussion came up briefly before [1] but it
>>>> was felt the practical use case will find firmware more suitable and
>>>> endpoint support in kernel can be used only for validation or demo.
>>>>
>>>> *) Modify platform driver to support EP mode (in my case pci-dra7xx.c).
>>>>
>>>> *) dt binding specific to EP mode should be created.
>>>>
>>>> Once I complete the implementation and start posting RFC patches, a
>>>> lot of these will become clear. But I want to check if this sounds
>>>> okay to you guys before starting the implementation.
>>>>
>>>> Let me know if you have some other ideas too.
>>>>
>>>> Cheers
>>>> Kishon
>>>>
>>>> [1] -> http://www.spinics.net/lists/linux-pci/msg26026.html
>>>>
>>>
>>> You are rising a topic that we are also addressing in Synopsys.
>>>
>>> For the PCIe RC hardware validation we are currently using the
>>> standard pcie-designware and pcie-designware-plat drivers.
>>>
>>> For the Endpoint we have to use an internal software package. Its main
>>> purpose is to initialize the IP registers, eDMA channels and make data
>>> transfer to prove that the everything is working properly. This is
>>> done in 2 levels, a custom driver built and loaded and an application
>>> that makes some ioctl to the driver executing some interesting
>>> functions to check the Endpoint status and make some data exchange.
>>
>> hmm.. the platform I have doesn't have a DMA in PCIe IP
>> (http://www.ti.com/lit/ug/spruhz6g/spruhz6g.pdf). So in your testing does
>> the EP access RC memory? i.e the driver in the RC allocates memory from it's
>> DDR and gives it's DDR address to the EP. The EP then transfers data to this
>> address. (This is a typical use case with ethernet PCIe cards). IIUC that's not
>> simple with configurable EPs. I'd like to know more about your testing though.
>>
> 
> Hi Kishon,
> 
> This is a typical user case for EP to use DMA transfer data to/from RC memory.
> In our case, we implement ring (like BD ring) or register in EP to communicate
> The address allocated in RC memory, then EP can transfer data to/from RC memory.

Initially I had some confusion w.r.t this because the address allocated in RC
memory can also be an address in EP system. For example let's assume we connect
two similar systems one configured as RC and the other configured as EP. The
PCI driver in the RC allocates memory in it's DDR (say 0x80000000) and programs
this address in the EP. Since it's a similar system, 0x80000000 will also be an
address in the EPs DDR. This will result in EP transferring data to it's own
DDR (at 0x80000000) instead of the same address in RC.

But later realized instead of directly using the DDR address given by RC, this
address should only be used to program the outbound window. That way the target
of the outbound window can be an address given by the RC and source should be
an address from the address space in the EP's system.

Do you also use the RC memory address to program the outbound window?

Thanks
Kishon