Re: [PATCH] net: dsa: qca8k: Disable mgmt Ethernet for qca8327

[Michał Kępień]

Christian, Andrew,

Here is what I managed to determine for the QCA8327 switch on the board
I have access to.

> One bug that was there was that when PBUS was used to access single port
> PHY register (instead of direct MDIO) the link up/down was broken. My
> theory was that the Switch chip had some latch logic that was only
> triggered with MDIO. Using PBUS didn't trigger such thing.
> 
> Could be that the QCA 8327 switch also have some kind of HW bug where specific
> register needs to go with MDIO or some refresh/latch logic are not
> correctly triggered.

My problem sounds similar.  The exact symptom I am observing is that at
a random point in time after switch setup, PHY register reads performed
via mgmt Ethernet frames start returning all zeros.  The mgmt Ethernet
transactions themselves are completed correctly, they just return all
zeros in the data part.  When the issue is triggered, _all_ subsequent
PHY register reads performed via mgmt Ethernet frames for the affected
PHY return all zeros, i.e. it never recovers.

The problem is triggered on distinct PHYs one by one, in a seemingly
random order; I did not identify any patterns.  Given enough time, all
PHYs eventually start returning all zeros for all of their register
reads.

At one point, I tweaked qca8k_internal_mdio_read() so that when a PHY
register read fails in this specific way, it is immediately retried over
MDIO.  Those fallback reads over MDIO also returned all zeros.

No timeouts are occurring for mgmt Ethernet frames.  I instrumented all
wait_for_completion_timeout() call sites in qca8k-8xxx.c and have yet to
see any of them log a timeout (or any other error, for that matter).  In
other words, all requests sent via mgmt Ethernet frames are responded to
by the switch in a timely manner, so I have no reason to believe that
the retry logic is involved here in any way.

All of this reproducibly happens on a device that does not even have any
Ethernet cables connected to it.

Manually probing registers 2 and 3 on the external MDIO bus reveals no
other devices except the expected five switch PHYs.

I used tcpdump to capture the traffic going through the DSA conduit
device, from the moment the switch is configured until the failure
occurs.  I found nothing suspicious: the last PHY register read before
the failed one happens about a second earlier and the sequence of mgmt
Ethernet frames sent and received for the last _successful_ read is a
1:1 match against the first _failing_ read, sans the sequence numbers.
No other traffic is interleaved, which is expected given that both the
external MDIO bus mutex and the mgmt Ethernet mutex are held throughout
each distinct PHY management transaction.

Given the above, I am out of ideas for something to latch on to for
further investigations.  It seems that over time, using mgmt Ethernet
for PHY access on this board makes the hardware transition into some
broken state that it cannot recover from.  As I am able to reliably
trigger this problem, I am open to ideas for further experiments that
could possibly shed some light on this.

> An idea might be to limit the mgmt to vlan and fdb and see if the problem
> is still there. (after all those are the path where mgmt would benefit due
> to the multiple register access required)

Indeed, it seems that the simplest workaround for this glitch is to move
the PHY nodes in the DTS file from the ethernet-switch node to its
parent, the mdio node, and use phy-handle for ethernet-ports to
configure qca8k in "external MDIO" mode.  That way, the external MDIO
bus is used for PHY management while mgmt Ethernet is still used for MIB
access and switch configuration, which seems to be stable; with PHY
management moved to MDIO, I ran "ethtool -S <iface>" in a loop for a few
hours and everything still seemingly worked fine.  This seems to be a
cleaner workaround than tweaking the driver code, so I don't plan to
submit any further patches in this spirit.

Thank you for your guidance.  If you have any ideas for further
experiments in this area, I am all ears.

-- 
Best regards,
Michał Kępień