Re: Synopsis XLGMII MAC, IEEE 802.3 and XLGMII interfaces

[Serge Semin]

Hi Russel

On Wed, Aug 23, 2023 at 11:55:34AM +0100, Russell King (Oracle) wrote:
> Hi,
> 
> Okay, I think it's time to get to the bottom of this, and below are my
> ramblings so far.

Thanks for the raised topic. Especially the last part which was very
actual in my recent work.

Regarding DW XLGMAC. Alas it's the only DW MAC IP-core for which I
currently don't have a databook. So I'll respond to you based on my
experience working with the DW GMAC/XGMAC/XPCS hardware and with the
STMMAC+XPCS drivers.

> 
> According to IEEE 802.3 80.1.3, XLGMII is the 40 Gb/s MII and CGMII
> is the 100 Gb/s MII. 81.1 goes on to state:
> 
> The XLGMII/CGMII has the following characteristics:
> 
>   a)    The XLGMII supports a speed of 40 Gb/s.
>   b)    The CGMII supports a speed of 100 Gb/s.
>   c)    Data and delimiters are synchronous to a clock reference.
>   d)    It provides independent 64-bit wide transmit and receive data paths.
>   e)    It supports full duplex operation only.
> 
> That seems very clear.

Right.

> 
> According to
> https://www.synopsys.com/dw/ipdir.php?ds=dwc_ether_enterprise_mac
> XLGMII operates at 25 Gb/s, 40 Gb/s, 50 Gb/s and 100 Gb/s. This
> appears to "disagree" with IEEE 802.3.
> 
> It appears, at least to me, that CGMII and XLGMII are physically
> similar interfaces, the only difference seems to be the speed of
> operation, so it would be entirely possible to have a MAC that
> can operate at both speeds - or indeed at other speeds such as
> Synopsys' "XLGMII" MAC.

DW XLGMAC IP-core supports three various sub-interfaces (see at the
bottom of the page in the link you cited): "Interfaces to the PHY layer
through a configurable interface supporting XLGMII/CGMII, XGMII or
GMII interface". Based on what is done in the DW XGMAC I assume that
it's either the same physical port which signals are somehow
multiplexed for working with different protocols or just separate
physical interfaces which can't work simultaneously but can be
connected to a PCS/PMA in a SoC/platform-dependent way. At least
that's the way the XGMII and GMII interfaces are implemented on the DW
XGMAC IP-core: either they are multiplexed on the same wires or the
GMII port is separately synthesized within the same MAC.

The XLGMII and CGMII interfaces are indeed similar the only difference
is speed which most likely is regulated by means of the reference
clock (TX_CLK/RX_CLK) rate which by IEEE 802.3 C81 shall be just
"one-sixty-fourth of the MAC transmit/receive data rate". So it's
625MHz for XLGMII and 1.5625GHz for CGMII. From that perspective
nothing prevents a vendor to implement an XLGMII/CGMII interface
working with any speed. For instance the 50/25Gbps speeds can be
reached just be halfing the base 100Gbps value.

XGMII and GMII are a bit different story. Physically they represent
completely different interfaces. But as I noted earlier DW XGMAC can
be synthesized with having these interfaces being either multiplexed
in a single port or provided as separate ports. But in both cases only
a single interface can be activated at a time.

The most modern DW XGMAC IP-core document I have is of v2.11a
(released at 2015). It states that the controller can expose two
interfaces: XGMII working with 10Gbps and GMII working with 2.5Gbps
and 1Gbps (that's what the STMMAC driver had supported before commit
5b0d7d7da64b ("net: stmmac: Add the missing speeds that XGMAC
supports")). Moreover in order for all those GMII speeds to work
correctly first XGMAC_TX_CONFIG must have the respective field updated
_and_ the clk_tx_156pt25_i reference clock must be driven with
312.5-MHz clock for 2.5G mode and with 125-MHz clock for 1G mode
(312.5 / 2.5 = 125).

Note my HW databook doesn't say anything about the XGMII 5.0/2.5Gbps and
MII 100/10Mbps modes. But seeing Jose added them in 2019 to the STMMAC
driver most likely these modes were introduced later perhaps in v3.x
or in the latest v2.x IP-cores. I bet these speeds are configured in
the same way as the GMII modes described above: set the respective
mode in the XMGAC_TX_CONFIG register (added by Jose) and drive the
reference clock with specific rate (it's platform-specific setting).

Considering all the above it's most likely DW XLGMAC works in a
similar way for the various speeds: XLGMII/CGMII - 100/50/40/25Gbps.
At least based on the available in the STMMAC
(drivers/net/ethernet/stmicro/stmmac) and Synopsys DW XLGMAC
(drivers/net/ethernet/synosps) code, the only thing that needs to be
done is to set the respective mode in the MAC Tx Control register. The
reference clock update is likely a platform-specific thing. Seeing
there is no any glue-driver available in kernel supporting DW XLGMAC
alas we can't proof the later statement.

Finally consolidating all of the above we can now get a notion about
the way the DW XGLMAC working in all the modes: XLGMII/CGMII
100/50/40/25Gbps, XGMII 10/5/2.5/1Gbps, GMII/MII 1000/100Mbps.

> 
> Now, the problem is... phylink interprets PHY_INTERFACE_MODE_XLGMII
> to mean the IEEE 802.3 defined 40 Gb/s mode - in that it _only_
> supports 40 Gb/s over that interface mode. Clearly, that's the right
> thing to be doing, because this is what IEEE defines and vendor
> stuff doesn't apply for generic code.
> 
> phylib hasn't defined PHY_INTERFACE_MODE_CGMII yet, so phylink doesn't
> provide that (maybe it should.) However, when it does, phylink will
> then allow 100 Gb/s over CGMII.
> 
> Are there other standards that define 25 and 50 Gb/s over an XLGMII-
> like link?

I failed to find any. So my bet these are the Synopsys-specific speed
modes.

> 
> Given the way things are at present, it means that the switch statement
> in stmmac_mac_link_up() for PHY_INTERFACE_MODE_XLGMII, only one case
> will ever be used - SPEED_40000. I'm guessing this isn't a problem as
> no one has reported any problems.

No one has reported any problems because there seems to be no actual
user of that STMMAC code in the kernel. There is no DT-bindings
defined, there is no DW XLGMAC specific compatible string defined,
there is no any xlgmac/xgmac glue-driver added. So IMO one of the
next situation might be relevant in this case:
1. Jose tested out the submitted XLGMAC patch with having a generic
"snps,dwgmac" or "snps,dwxgmac"-compatible node defined, made sure
it's working for the 40Gbps speed and that was it.
2. Alternatively are you sure the speed is validated before being
passed to the stmmac_mac_link_up() in the phylink_resolve() method
after the Auto-negotiation is finished?

Regarding the generic questions you raised before. IMO:
1. PHY_INTERFACE_MODE_CGMII should be defined with at least 100Gbps
speed allowed.
2.1 Either "xlgmii" and "cgmii" interfaces need to be extended with
wider speed settings but being marked with an in-situ comment that
these are vendor-specific speeds,
2.2 or some new interfaces could be defined like
"xlgmii-ext"/"cgmii-ext" or even "xxvgmii"/"lgmii" with the extended
set of speeds.

As for me option 2.1 seems much better but we need to decide to what
interface the vendor-specific speeds are supposed to be added. I can't
say for sure what interface Synopsys DW XLGMAC defines as working with
25/50Gbps speeds especially seeing XLGMII and CGMII are physically
compatible with only reference clock different. So IMO CGMII looks a
bit more suitable for them since those speeds are the direct quotient
of 100Gbps - base CGMII interface speed.

> 
> Then... there's the inclusion of 10G, 2.5G and 1G in the XLGMII switch.
> The above link to Synopsys website suggests that XGMII is used for 10G
> and GMII for 1G.

As I explained above XGMII interface can work with the 10/5/2.5Gbps speeds
and GMII - with the 2.5/1Gbps speeds. I have doubts these speeds need
to be added to the XLGMII/CGMII interface modes.

> 
> Given that this is just the MAC, and we would normally have a PCS after
> it (which may be XPCS), the interface used between the MAC and PCS isn't
> all that relevant, since PHY_INTERFACE_MODE_xxx primerily defines the
> interface to any PHY that is connected, and as a secondary function the
> interface to fiber (since [MAC - PCS - Serdes] - PHY - media and
> [MAC - PCS - Serdes] - media are common, it has made sense to use the
> PHY interface mode to define the protocol used on that part of the
> link that a PHY _could_ be connected to.)
> 
> So, I'm not convinced that the fact that this MAC core uses XLGMII,
> XGMII and GMII has any relevance to the PHY_INTERFACE_MODE_ passed
> by phylink into functions such as stmmac_mac_link_up(), _unless_ a
> PHY is connected directly to the MAC.
> 
> If XPCS is connected, that interface mode would be whatever XPCS is
> using to talk to the "outside world" not what the connection is
> between the MAC and XPCS.

That's what I thought too and that's exactly the issues I met a few
months ago while I was working on adding the XGMII interface support
to the XPCS driver. After a lot of digging into the problem I came to
the next semantics implemented in the STMMAC and XPCS drivers:

1. XLGMII - phy-mode only activates C73 AN,
            PHY-less setup is only supported.
   Basically the mode is utilized to activate the C73
   Auto-negotiation. So after it's finished the respective speed
   would be set in DW XLGMAC registers. Seeing PHY can't be attached
   in this setup the interface mode isn't that relevant except it
   just makes the PHY-link core and the XPCS driver creating a proper
   supported link-modes field and activate respective speed in the
   MAC.

   IMO seeing this PHY-mode doesn't imply external PHY attached and always
   activates Clause 73 auto-negotiation, this MAC-PCS interface mode is useful
   for the "Backplane" setups when you have XLGMAC/XPCS/PMA directly attached
   to a network fabrics and you actually don't know what physical link
   mode is there. At least that's the way I decided to implement for
   the "xgmii" PHY-mode support in the DW XPCS driver: it doesn't imply
   any external PHY detected and initialized, it just activates the
   C73 auto-negotiation and re-configures XPCS from 10GBase-R, to
   10GBase-KX4, to 1000Base-X until the link gets up or fails. Though in
   that case DW XPCS needs to have some CSR updated to get different
   modes activated. There is no any similar setups (for instance in
   order to switch between Base-KR, Base-KR2 and Base-KR4 looking for
   a supported physical link) performed in the "xlgmii" mode
   implementation in the driver.

Here are some other my observations I came up to on my way of the
STMMAC/XPCS drivers alteration:

2. 1000Base-X, SGMII - phy-mode indicates PCS/PMA<=>PHY interface,
                       PHY and PHY-less setups are supported.
   No problem with that. There are both AN and actual PCS setups in
   the driver.

3. 2500Base-X - phy-mode indicates PCS/PMA<=>PHY interface,
                PHY and PHY-less setups are supported.
   No series problem with this too except in fact this mode doesn't
   imply auto-negotiation but has Autoneg flag set. Most likely it's
   needed to prevent the pcs_validate() method to clear out the external
   PHY link-mode Autoneg flag.

4. USXGMII - phy-mode indicates PCS/PMA<=>PHY interface,
             PHY-less setup is only supported.
   DW XGMAC actually doesn't support USXGMII interface, instead
   it has a multi-protocol port(s) XGMII/GMII/MII as I noted before.
   So the only thing that is required for the MAC is to activate the
   respective interface and speed by means of the MAC Tx Control
   register. The rest is supposed to be done by the DW XPCS/PMA/PHY
   IP-cores chain: https://www.synopsys.com/dw/ipdir.php?ds=dwc_multi_protocol_10g_phy
   It seems weird why this mode doesn't permit an external PHY
   attached but only relies on the auto-negotiation.

5. 10GBase-KR - phy-mode only activates C73 AN,
                PHY-less setup is only supported.
   This mode is very weird. First it isn't supported by the STMMAC
   driver. Second no actual PCS setup is performed except the
   auto-negotiation ones. So the XPCS driver relies on the default
   device state. Third both PHY-mode and the link-modes match. It's
   kind of strange seeing no PHY is implied by the STMMAC driver
   in this case. I failed to find any NIC driver in kernel which
   would have XPCS attached and have the PHY_INTERFACE_MODE_10GKR mode
   supported. So at the first glance this mode support seems redundant.

6. Another questionable part in the XPCS driver is the way the
   pcs_validate() method is implemented. In case of the platform with
   an external PHY available it filters out all the link-modes the PHY
   can advertise including the actual physical link modes. It doesn't
   look correct especially in case of 2. and 3. IMO this filtering should
   have been activated only on the PHY-less systems only or even dropped
   seeing none of the PCS driver do that.

I might have missed something but I failed to find a well
justification at the very least to the entries 5 and 6.

> If we want to know what that is, then
> stmmac needs to be asking XPCS for that information (and maybe
> phylink needs to get that from its PCS on behalf of the MAC driver.)

My thoughts about this is in the entry 1 above.

-Serge(y)

> 
> I think there's a lot that needs to be thought about here.
> 
> Part 2 of this will be USXGMII... which I'll do as a separate email.
> 
> Thanks.
> 
> -- 
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!