RE: [PATCH net-next v19 01/13] rtase: Add pci table supported in this module

[Larry Chiu]

> On Wed, May 22, 2024 at 04:43:11AM +0000, Larry Chiu wrote:
> >
> > > On Tue, May 21, 2024 at 06:20:04AM +0000, Larry Chiu wrote:
> > > >
> > > > >> + *  Below is a simplified block diagram of the chip and its relevant
> > > interfaces.
> > > > >> + *
> > > > >> + *               *************************
> > > > >> + *               *                       *
> > > > >> + *               *  CPU network device   *
> > > > >> + *               *                       *
> > > > >> + *               *   +-------------+     *
> > > > >> + *               *   |  PCIE Host  |     *
> > > > >> + *               ***********++************
> > > > >> + *                          ||
> > > > >> + *                         PCIE
> > > > >> + *                          ||
> > > > >> + *      ********************++**********************
> > > > >> + *      *            | PCIE Endpoint |             *
> > > > >> + *      *            +---------------+             *
> > > > >> + *      *                | GMAC |                  *
> > > > >> + *      *                +--++--+  Realtek         *
> > > > >> + *      *                   ||     RTL90xx Series  *
> > > > >> + *      *                   ||                     *
> > > > >> + *      *     +-------------++----------------+    *
> > > > >> + *      *     |           | MAC |             |    *
> > > > >> + *      *     |           +-----+             |    *
> > > > >> + *      *     |                               |    *
> > > > >> + *      *     |     Ethernet Switch Core      |    *
> > > > >> + *      *     |                               |    *
> > > > >> + *      *     |   +-----+           +-----+   |    *
> > > > >> + *      *     |   | MAC |...........| MAC |   |    *
> > > > >> + *      *     +---+-----+-----------+-----+---+    *
> > > > >> + *      *         | PHY |...........| PHY |        *
> > > > >> + *      *         +--++-+           +--++-+        *
> > > > >> + *      *************||****************||***********
> > > > >> + *
> > > > >> + *  The block of the Realtek RTL90xx series is our entire chip
> > > > >> + architecture,
> > > > >> + *  the GMAC is connected to the switch core, and there is no PHY
> in
> > > between.
> > > > >
> > > > >Given this architecture, this driver cannot be used unless there is a
> switch
> > > > >driver as well. This driver is nearly ready to be merged. So what are
> your
> > > > >plans for the switch driver? Do you have a first version you can post?
> That
> > > > >will reassure us you do plan to release a switch driver, and not use a
> SDK in
> > > > >userspace.
> > > > >
> > > > >        Andrew
> > > >
> > > > Hi Andrew,
> > > > This GMAC is configured after the switch is boot-up and does not
> require a
> > > > switch driver to work.
> > >
> > > But if you cannot configure the switch, it is pointless passing the switch
> > > packets. The Linux architecture is that Linux needs to be able to control
> the
> > > switch somehow. There needs to be a driver with the switchdev API on
> its
> > > upper side which connects it to the Linux network stack. Ideally the
> lower
> > > side of this driver can directly write switch registers. Alternatively it can
> make
> > > some sort of RPC to firmware which configures the switch.
> > >
> > > Before committing this MAC driver, we will want to be convinced there is
> a
> > > switchdev driver for the switch.
> > >
> > >         Andrew
> >
> >
> > I know what you mean.
> > But actually this GMAC works like a NIC connected to an Ethernet Switch
> not a
> > management port, its packets communicating with other ports.
> 
> Linux has two different models for switches.
> 
> The first is switchdev. Linux has a netdev per port of the switch, and
> use you those netdev's to manage the switch, just as if they are
> individual NICs.
> 
> The second is very, very old, since the beginning of Ethernet
> switches. The cable comes out of the machine and plugs into the
> switch. Linux has no idea there is a switch there, the switch is just
> part of the magic if networking. This also means Linux cannot manage
> the switch, it is a different box, a different administration domain.
> 
> The second model does not really work here. The switch is not in
> another box at the end of a cable. It is integrated into the SoC!
> 
> > The PCIe Endpoint is a multi-function device, the other function is used to
> > control the switch register, we are still working on where to put this driver
> in
> > Linux. We thought it should be separated into different device drivers, or
> you
> > think we should register two pcie functions in this driver.
> 
> Look at the architecture of other switch drivers. There are two broad
> categories.
> 
> 1) Pure switchdev drivers, e.g. mellanox, sparx5, prestera. There is
> one driver which provides both the netdev interfaces per port, and
> implements the switchdev API for managing the switch.
> 
> 2) DSA + switchdev, e.g. mv88e6xxx, rtl8365, starfigher2, etc. These
> use a conventional NIC to provide the conduit to pass packets to the
> switch. These packets have additional headers, added by a tag driver,
> indicating which port a packet should go out. And there is a switch
> driver, which makes use of the DSA framework to manage the switch. DSA
> provides the netdev per port.
> 
> This is actually something i ask you about with version 1 of the
> patches. I've forget what your answer was, and we concentrated on
> getting your code up to mainline quality. Now it is time to go back to
> that question.
> 
> How do you control where a packet passed over this GMAC NIC goes
> within the switch? Is there an additional header? Are their fields in
> the DMA descriptor?
> 
> If your hardware is DSA like, you can write another driver which binds
> to a different PCI function. If however you use DMA descriptors, you
> need a pure switchdev driver, one driver which binds to multiple PCI
> functions.
> 
>         Andrew

Thank you very much for your clear reply.

As I mentioned, it works like a NIC connected to an Ethernet Switch, not a
Management port.
The packets from this GMAC are routed according to switch rules such as
ACL, L2, .... and it does not control packet forwarding through any special
header or descriptor. In this case, we have our switch tool which is used 
for provisioning these rules in advance. Once the switch boots up, the 
rules will be configured into the switch after the initialization. With this 
driver and the provisioning by our switch tool, it can make switch forward 
the frame as what you want. So it's not a DSA like device.

In another case, we do have other function which is used for controlling 
the switch registers instead of sending packets from the switch ports.
At the meanwhile, we are investigating how to implement the function to
Integrate into switchdev.