Re: [RFC PATCH net-next 01/28] docs: networking: ti: add driver doc for CPSW Proxy Client

[Siddharth Vadapalli]

On Tue, Jun 04, 2024 at 04:20:16PM +0200, Andrew Lunn wrote:
> On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote:

[...]
> 
> If you do want to add a third model, where Linux has some insight into
> the switch, you need to coordinate with other vendors in the
> automotive world, and come up with a model which everybody can
> use. What i don't want is a TI model, followed by a Realtek model,
> followed by a vendor XYZ model. So if you need more than what the
> first model above provides, you will need to get a consortium of
> vendors together to design a new model a few vendors agree on.

I believe that a third model is required given the System Architecture
and the use-case that it must cater to. I agree completely that having a
vendor specific implementation should always be the last step when it is
just not possible to generalize any portion of the implementation. I will
describe the existing Architecture on the TI SoC and will also attempt to
generalize the implementation below. I hope that you could review it and
guide me towards the generic, vendor-agnostic implementation which will
also address the use-case that this series corresponds to. I am willing
to work on the generic implementation since I assume that this series
does keep it generic enough that it could be extended to be vendor
independent. So there might be minor changes required when switching to
the generic model. On the other hand, based on the description that I
provide below, if you think that the existing models can also be slightly
modified to accomodate the use-case, I will surely take that into
consideration and work on the corresponding implementation.

System Architecture and Implementation Details
==============================================

The CPSW Ethernet Switch has a single Host Port (CPU facing port) through
which it can receive data from the Host(s) and transmit data to the
Host(s). The exchange of data occurs via TX/RX DMA Channels (Hardware
Queues). These Hardware Queues are a limited resource (8 TX Channels and
up to 64 RX Flows). If the Operating System on any of the cores is the
sole user of CPSW then all of these Hardware Queues can be claimed by that
OS. However, when CPSW has to be shared across the Operating Systems on
various cores with the aim of enabling Ethernet Functionality for the
Applications running on different cores, it is necessary to share these
Hardware Queues in a manner that prevents conflicts. On the control path
which corresponds to the configuration of CPSW to get it up and running,
since there is no Integrated Processor within CPSW that can be programmed
with a startup configuration, either the Operating System or Firmware
running on one of the cores has to take the responsibility of setting it.
One option in this case happens to be the Ethernet Switch Firmware (EthFw)
which is loaded by the Bootloader on a remote core at the same time that
Linux and other Operating Systems begin booting. EthFw quickly powers on
and configures CPSW getting the Forwarding Path functional. Once Linux and
other Operating Systems on various cores are ready, they can communicate
with EthFw to obtain details of the Hardware Queues allocated to them to
exchange data with CPSW. With the knowledge of the Hardware Queues that
have been allocated, Linux can use the DMA APIs to setup these queues
to exchange data with CPSW.

Setting up the Hardware Queues alone isn't sufficient to exchange data
with the external network. Consider the following example:
The ethX interface in userspace which has been created to transmit/receive
data to/from CPSW has the user-assigned MAC Address of "M". The ping
command is run with the destination IP of "D". This results in an ARP
request sent from ethX which is transmitted out of all MAC Ports of CPSW
since it is a Broadcast request. Assuming that "D" is a valid
destination IP, the ARP reply is received on one of the MAC Ports which
is now a Unicast reply with the destination MAC Address of "M". The ALE
(Address Lookup Engine) in CPSW has learnt that the MAC Address "M"
corresponds to the Host Port when the ARP request was sent out. So the
Unicast reply isn't dropped. The challenge however is determining which
RX DMA Channel (Flow) to send the Unicast reply on. In the case of a
single Operating System owning all Hardware Queues, sending it on any of
the RX DMA Channels would have worked. In the current case where the RX
DMA Channels map to different Hosts (Operating Systems and Applications),
the mapping between the MAC Address "M" and the RX DMA Channel has to be
setup to ensure that the correct Host receives the ARP reply. This
necessitates a method to inform the MAC Address "M" associated with the
interface ethX to EthFw so that EthFw can setup the MAC Address "M" to
RX DMA Channel map accordingly.

At this point, Linux can exchange data with the external network via CPSW,
but no device on the external network can initiate the communication by
itself unless it already has the ARP entry for the IP Address of ethX.
That's because CPSW doesn't support packet replication implying that any
Broadcast/Multicast packets received on the MAC Ports can only be sent
on one of the RX DMA Channels. So the Broadcast/Multicast packets can
only be received by one Host. Consider the following example:
A PC on the network tries to ping the IP Address of ethX. In both of the
following cases:
1. Linux hasn't yet exchanged data with the PC via ethX.
2. The MAC Address of ethX has changed.
the PC sends an ARP request to one of the MAC Ports on CPSW to figure
out the MAC Address of ethX. Since the ARP request is a Broadcast
request, it is not possible for CPSW to determine the correct Host,
since the Broadcast MAC isn't unique to any Host. So CPSW is forced
to send the Broadcast request to a preconfigured RX DMA Channel which
in this case happens to be the one mapped to EthFw. Thus, if EthFw
is aware of the IP Address of ethX, it can generate and send the ARP
reply containing the MAC Address "M" of ethX that it was informed of.
With this, the PC can initiate communication with Linux as well.

Similarly, in the case of Multicast packets, if Linux wishes to receive
certain Multicast packets, it needs to inform the same to EthFw which
shall then replicate the Multicast packets it received from CPSW and
transmit them via alternate means (Shared Memory for example) to Linux.

The following is a summary of the steps so far required to enable
Ethernet Functionality for applications running on Linux:
1. Determine and setup the Hardware Queues allocated to Linux
2. Inform the MAC Address of ethX to EthFw
3. Inform the IP Address of ethX to EthFw
4. Inform any of the Multicast Addresses associated with ethX to EthFw

All data between Linux (Or any Operating System) and EthFw is exchanged
via the Hardware Mailboxes with the help of the RPMsg framework. Since
all the resource allocation information comes from EthFw, the
vendor-specific implementation in the Linux Client is limited to the DMA
APIs used to setup the Hardware Queues and to transmit/receive data with
the Ethernet Switch. Therefore, it might be possible to move most of the
vendor specific implementation to the Switch Configuration Firmware
(similar to EthFw), to make the Linux Client implementation as generic
and vendor agnostic as possible. I believe that this series more or less
does the same, just using custom terminology which can be made generic.

I can update this series to a generic implementation along with proper
documentation and naming convention to enable any vendor to reuse the
same without having to modify the implementation. The RPMsg ABIs can
be given generic names with extensive documentation and also designed
to be extensible enough to cater to functional enhancements over time.

Kindly let me know your thoughts on this.

Regards,
Siddharth.