RFC: Advice on adding support for Qualcomm IPQ9574 SoC Ethernet

[Kiran Kumar C . S . K]

Hello netdev,

We are planning to publish driver patches for adding Ethernet support
for Qualcomm's IPQ9574 SoC, and looking for some advice on the approach
to follow. There are two new drivers (described below) split across four
patch series, totaling to 40 patches. These two drivers depend on a
couple of clock controller drivers which are currently in review with
the community.

Support is currently being added only for IPQ9574 SoC. However the
drivers are written for the Qualcomm PPE (packet process engine)
architecture, and are easily extendable for additional IPQ SoC (Ex:
IPQ5332) that belong to the same network architecture family.

Given the number of patches for IPQ9574, we were wondering whether it is
preferred to publish the four series together, since having all the code
available could help clarify the inter-workings of the code. Or whether
it is preferred to publish the patches sequentially, depending on the
review progress?

As part of this email, we also wanted to give a brief overview of the
various hardware blocks involved, the various driver patch series, and
list the dependencies between them. Hopefully this will help the review
process (this info will be also added to driver doc). The rest of the
email addresses this.

Thank you for your time!

With Regards
Kiran

=========================================================================
Section Layout
==============
1. IPQ Ethernet hardware overview
   1.1 PPE: Internal blocks overview
   1.2 Clock controllers for network function
2. List of Patch series and dependencies


1. IPQ Ethernet hardware overview
=================================
The PPE (packet process engine) is a hardware block in IPQ SoC which
provides Ethernet and L2/L3 networking offload functions. These L2/L3
functions help offload network processing functions from the CPU.
Specifically w.r.to Ethernet functionality, it broadly comprises of
three components: Ethernet DMA, Switch core and GMACs/xGMACs.

On IPQ9574 SoC, the PPE includes 6 ethernet ports (1G/10G MACs) which
can connect to external PHY. For packet transfer between host CPU and
these ethernet ports, the PPE interfaces with the host CPU using
Ethernet-DMA (EDMA) on a special CPU connected port.

It also includes a switch core which can transfer packets between the
switch GMAC ports and CPU, or transfer packets between the switch ports.

The below diagram for IPQ9574 depicts the hardware blocks within and
outside PPE, which work together to support Ethernet functionality.

Fig.1.1 PPE and hardware block connectivity diagram for IPQ9574
---------------------------------------------------------------
         +---------+
         |  48MHZ  |
         +----+----+
              |(clock)
              v
         +----+----+
  +------| CMN PLL |
  |      +----+----+
  |           |(clock)
  |           v
  |      +----+----+           +----+----+  clock   +----+----+
  |  +---|  NSSCC  |           |   GCC   |--------->|   MDIO  |
  |  |   +----+----+           +----+----+          +----+----+
  |  |        |(clock & reset)      |(clock & reset)
  |  |        v                     v
  |  |   +-----------------------------+----------+----------+---------+
  |  |   |       +-----+               |EDMA FIFO |          | EIP FIFO|
  |  |   |       | SCH |               +----------+          +---------+
  |  |   |       +-----+                      |               |        |
  |  |   |  +------+   +------+            +-------------------+       |
  |  |   |  |  BM  |   |  QM  |            | L2/L3 Switch Core |       |
  |  |   |  +------+   +------+            +-------------------+       |
  |  |   |                                   |                         |
  |  |   | +-------+ +-------+ +-------+ +-------+ +-------+ +-------+ |
  |  |   | |  MAC0 | |  MAC1 | |  MAC2 | |  MAC3 | | XGMAC4| |XGMAC5 | |
  |  |   | +---+---+ +---+---+ +---+---+ +---+---+ +---+---+ +---+---+ |
  |  |   |     |         |         |         |         |         |     |
  |  |   +-----+---------+---------+---------+---------+---------+-----+
  |  |         |         |         |         |         |         |
  |  |     +---+---------+---------+---------+---+ +---+---+ +---+---+
  +--+---->|             PCS0                    | |  PCS1 | | PCS2  |
  | clock  +---+---------+---------+---------+---+ +---+---+ +---+---+
  |            |         |         |         |         |         |
  |        +---+---------+---------+---------+---+ +---+---+ +---+---+
  | clock  +----------------+                    | |       | |       |
  +------->|Clock Controller|   4-port Eth PHY   | | PHY4  | | PHY5  |
           +----------------+--------------------+ +-------+ +-------+


1.1 PPE: Internal blocks overview
=================================

The Switch core
---------------
It has maximum 8 ports, comprising 6 GMAC ports and two DMA interfaces
(for Ethernet DMA and EIP security processor) on the IPQ9574.

GMAC/xGMAC
----------
There are 6 GMAC and 6 XGMAC in IPQ9574. Depending on the board ethernet
configuration, either GMAC or XGMAC is selected by the PPE driver to
interface with the PCS. The PPE driver initializes and manages these
GMACs, and registers one netdevice per GMAC.

EDMA (Ethernet DMA)
-------------------
This is a common ethernet DMA block inside PPE, which is used to
transmit and receive packets between Ethernet ports in the PPE switch,
and the ARM CPU cores. The PPE driver includes the ethernet DMA driver,
and registers one netdevice per PPE port.

PCS
---
The PCS blocks are outside the PPE, and provides the connection between
PPE's GMAC/XGMAC and the external ethernet PHY. There are 3 PCS
instances supported by IPQ9574. The PCS provides the PCS/xPCS function
to support modes such as SGMII/2500BASE-X/QSGMII/USXGMII/10G-BASE-R modes.

SCH (Scheduler)
---------------
The PPE driver initializes this block to enable traffic scheduling for
switch ports at egress.

QM (Queue Manager)
------------------
This block manages the various egress queues of the PPE switch.

The queues inside the switch core in PPE are mapped to the switch
ports. The PPE driver initializes this block to enable the switch
port to queue mappings as per the SoC's port configuration for
IPQ9574.

BM (Buffer Manager)
-------------------
This block manages the buffer thresholds for PPE port flow control.
The buffer availability for a port at run time will influence the
behavior of flow control on that port.

1.2 Clock controllers for network function
==========================================
Common PLL block
----------------
This block takes in a fixed reference clock as input and provide
clocks to other hardware blocks and peripherals such as NSS clock
controller, and to external PHYs as output.

NSS clock controller
--------------------
The NSS clock controller supplies clock to ethernet hardware blocks in
the IPQ such as PPE and PCS blocks, as shown in the diagram. It takes
clock input from Common PLL.

2. List of patch series and dependencies
========================================

Clock drivers (currently in review)
===================================
1) CMN PLL driver patch series:
	Currently in review with community.
	https://lore.kernel.org/linux-arm-msm/20240827-qcom_ipq_cmnpll-v3-0-8e009cece8b2@xxxxxxxxxxx/


2) NSS clock controller (NSSCC) driver patch series
	Currently in review with community.
	https://lore.kernel.org/linux-arm-msm/20240626143302.810632-1-quic_devipriy@xxxxxxxxxxx/


Networking drivers (to be posted for review next week)
======================================================

The following patch series are planned to be pushed for the PPE and PCS
drivers, to support ethernet function. These patch series are listed
below in dependency order.

3) PCS driver patch series:
        Driver for the PCS block in IPQ9574. New IPQ PCS driver will
        be enabled in drivers/net/pcs/
	Dependent on NSS CC patch series (2).

4) PPE base driver patch series
        Base PPE driver support for IPQ9574.
	Configures scheduler, BM, QM, MAC during initialization for
		IPQ9574.
	Dependent on NSS CC patch series (2).

	We plan to update the below patch series which was paused
        earlier, with the new code which addresses the earlier
        shortcomings.

	https://lore.kernel.org/netdev/20240110114033.32575-1-quic_luoj@xxxxxxxxxxx/

5) PPE MAC support patch series
        PPE driver patches to configure the various functions of
        ethernet MAC in PPE hardware and to enable phylink support for
        each ethernet port.

        Dependent on NSS CC (2) and PPE base (4) patch series

6) Ethernet DMA driver patch series (part of PPE driver)
        Enables DMA driver support for PPE EDMA, netdevice
        registration and operation for each of the PPE ports.

        Dependendent on NSS CC (2), PPE base (4), PPE MAC (5) patch
        series.

Other notable dependencies
===============================
MDIO driver:
        Already merged in net-next via the below patch from Christian.

https://lore.kernel.org/linux-arm-msm/170686862773.17682.10435156329986246682.git-patchwork-notify@xxxxxxxxxx/