Re: [PATCH V7 0/3] Generate device tree node for pci devices

[Frank Rowand]

On 3/21/23 03:44, Christian Gmeiner wrote:
> Hi all
> 
> Am Do., 9. März 2023 um 09:52 Uhr schrieb Clément Léger
> <clement.leger@xxxxxxxxxxx>:
>>
>> Le Wed, 8 Mar 2023 01:31:52 -0600,
>> Frank Rowand <frowand.list@xxxxxxxxx> a écrit :
>>
>>> On 3/6/23 18:52, Rob Herring wrote:
>>>> On Mon, Mar 6, 2023 at 3:24 PM Frank Rowand <frowand.list@xxxxxxxxx> wrote:
>>>>>
>>>
>>> < snip >
>>>
>>> Hi Rob,
>>>
>>> I am in no position to comment intelligently on your comments until I
>>> understand the SoC on PCI card model I am asking to be described in
>>> this subthread.
>>
>> Hi Frank,
>>
>> Rather than answering all of the assumptions that were made in the upper
>> thread (that are probably doing a bit too much of inference), I will
>> re-explain that from scratch.
>>
>> Our usecase involves the lan966x SoCs. These SoCs are mainly targeting
>> networking application and offers multiple SFP and RGMII interfaces.
>> This Soc can be used in two exclusive modes (at least for the intended
>> usage):
>>
>> SoC mode:
>>    The device runs Linux by itself, on ARM64 cores included in the
>>    SoC. This use-case of the lan966x is currently almost upstreamed,
>>    using a traditional Device Tree representation of the lan996x HW
>>    blocks [1] A number of drivers for the different IPs of the SoC have
>>    already been merged in upstream Linux (see
>>    arch/arm/boot/dts/lan966x.dtsi)
>>
>> PCI mode:
>>   The lan966x SoC is configured as a PCIe endpoint (PCI card),
>>   connected to a separate platform that acts as the PCIe root complex.
>>   In this case, all the IO memories that are exposed by the devices
>>   embedded on this SoC are exposed through PCI BARs 0 & 1 and the ARM64
>>   cores of the SoC are not used. Since this is a PCIe card, it can be
>>   plugged on any platform, of any architecture supporting PCIe.
>>
>> This work only focus on the *PCI mode* usage. In this mode, we have the
>> following prerequisites:
>> - Should work on all architectures (x86, ARM64, etc)
>> - Should be self-contained in the driver
>> - Should be able to reuse all existing platform drivers
>>
>> In PCI mode, the card runs a firmware (not that it matters at all by
>> the way) which configure the card in PCI mode at boot time. In this
>> mode, it exposes a single PCI physical function associated with
>> vendor/product 0x1055/0x9660. This is not a multi-function PCI device !
>> This means that all the IO memories (peripheral memories, device
>> memories, registers, whatever you call them) are accessible using
>> standard readl()/writel() on the BARs that have been remapped. For
>> instance (not accurate), in the BAR 0, we will have this kind of memory
>> map:
>>
>>            BAR0
>>    0x0 ┌───────────┐
>>        │           │
>>        ├───────────┤
>>        │   Clock   │
>>        │ controller│
>>        ├───────────┤
>>        │           │
>>        ├───────────┤
>>        │   I2C     │
>>        │ controller│
>>        ├───────────┤
>>        │           │
>>        ├───────────┤
>>        │   MDIO    │
>>        │ Controller│
>>        ├───────────┤
>>        │           │
>>        ├───────────┤
>>        │  Switch   │
>>        │ Controller│
>>        ├───────────┤
>>        │           │
>>        │   ...     │
>>
>>
>> It also exposes either a single interrupt via the legacy interrupt
>> (which can then be demuxed by reading the SoC internal interrupt
>> controller registers), or multiple interrupts using MSI interrupts.
>>
>> As stated before, all these peripherals are already supported in SoC
>> mode and thus, there are aleready existing platform drivers for each of
>> them. For more information about the devices that are exposed please
>> see link [1] which is the device-tree overlay used to describe the
>> lan9662 card.
>>
>> In order to use the ethernet switch, we must configure everything that
>> lies around this ethernet controller, here are a few amongst all of
>> them:
>> - MDIO bus
>> - I2C controller for SFP modules access
>> - Clock controller
>> - Ethernet controller
>> - Syscon
>>
>> Since all the platform drivers already exist for these devices, we
>> want to reuse them. Multiple solutions were thought of (fwnode, mfd,
>> ACPI, device-tree) and eventually ruled out for some of them and efforts
>> were made to try to tackle that (using fwnode [2], device-tree [3])
>>
>> One way to do so is to use a device-tree overlay description that is
>> loaded dynamically on the PCI device OF node. This can be done using the
>> various device-tree series series that have been proposed (included
>> this one). On systems that do not provide a device-tree of_root, create
>> an empty of_root node (see [4]). Then during PCI enumeration, create
>> device-tree node matching the PCI tree that was enumerated (See [5]).
>> This is needed since the PCI card can be plugged on whatever port the
>> user wants and thus it can not be statically described using a fixed
>> "target-path" property in the overlay.
>>
>> Finally, to glue everything together, we add a PCI driver for the
>> VID/PID of the PCI card (See [6]). This driver is responsible of adding
>> the "ranges" property in the device-tree PCI node to remap the child
>> nodes "reg" property to the PCI memory map. This is needed because the
>> PCI memory addresses differ between platform, enumeration order and so
>> on.Finally, the driver will load the device-tree overlay (See [1]) to
>> the PCI device-tree node. Eventually, a call to
>> of_platform_default_populate() will probe the nodes and platform
>> drivers.
>>
>> I hope this will help you understanding what is going on here. In the
>> meantime, I'm also trying to obtain public documentation about the
>> lan966x SoC.
>>
>> [1]
>> https://github.com/clementleger/linux/blob/bf9b4ef803d86c4ae59a4ca195a4152b0d5c3cea/drivers/mfd/lan966x_pci.dts
>> [2]
>> https://lore.kernel.org/netdev/YhPSkz8+BIcdb72R@xxxxxxxxxxxxxxxxxx/T/
>> [3]
>> https://lore.kernel.org/lkml/20220427094502.456111-1-clement.leger@xxxxxxxxxxx/
>> [4]
>> https://lore.kernel.org/lkml/20230223213418.891942-1-frowand.list@xxxxxxxxx/
>> [5]
>> https://lore.kernel.org/lkml/1674183732-5157-1-git-send-email-lizhi.hou@xxxxxxx/
>> [6]
>> https://github.com/clementleger/linux/blob/bf9b4ef803d86c4ae59a4ca195a4152b0d5c3cea/drivers/mfd/lan966x_pci_of.c
>>
>> --
>> Clément Léger,
>> Embedded Linux and Kernel engineer at Bootlin
>> https://bootlin.com
> 
> What is missing to move on with this patch set?

I need to evaluate what Clément Léger wrote in the email you replied to.  I had overlooked this
reply to my questions.  From a quick scan, it looks like he _probably_ provided the context I
was looking for to understand the architecture of the proposal.  But it is late Sunday night,
so I won't get to this tonight.

-Frank

>