RE: [PATCH net-next v2 1/1] Documentation: networking: add Twisted Pair Ethernet diagnostics at OSI Layer 1

[Divya.Koppera]

Hi @Oleksij Rempel,

One more comment including previous ones(Re-sending as I faced mail delivery issue to some mailing addresses).

> -----Original Message-----
> From: Oleksij Rempel <o.rempel@xxxxxxxxxxxxxx>
> Sent: Thursday, October 3, 2024 11:36 AM
> To: Andrew Lunn <andrew@xxxxxxx>; Heiner Kallweit
> <hkallweit1@xxxxxxxxx>; David S. Miller <davem@xxxxxxxxxxxxx>; Eric
> Dumazet <edumazet@xxxxxxxxxx>; Jakub Kicinski <kuba@xxxxxxxxxx>; Paolo
> Abeni <pabeni@xxxxxxxxxx>; Rob Herring <robh@xxxxxxxxxx>; Krzysztof
> Kozlowski <krzk+dt@xxxxxxxxxx>; Florian Fainelli <f.fainelli@xxxxxxxxx>;
> Maxime Chevallier <maxime.chevallier@xxxxxxxxxxx>; Kory Maincent
> <kory.maincent@xxxxxxxxxxx>; Lukasz Majewski <lukma@xxxxxxx>;
> Jonathan Corbet <corbet@xxxxxxx>
> Cc: Oleksij Rempel <o.rempel@xxxxxxxxxxxxxx>; kernel@xxxxxxxxxxxxxx;
> linux-kernel@xxxxxxxxxxxxxxx; netdev@xxxxxxxxxxxxxxx; Russell King
> <linux@xxxxxxxxxxxxxxx>
> Subject: [PATCH net-next v2 1/1] Documentation: networking: add Twisted
> Pair Ethernet diagnostics at OSI Layer 1
> 
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the
> content is safe
> 
> This patch introduces a diagnostic guide for troubleshooting Twisted
> Pair  Ethernet variants at OSI Layer 1. It provides detailed steps for
> detecting  and resolving common link issues, such as incorrect wiring,
> cable damage,  and power delivery problems. The guide also includes
> interface verification  steps and PHY-specific diagnostics.
> 
> Signed-off-by: Oleksij Rempel <o.rempel@xxxxxxxxxxxxxx>
> ---
> changes v2:
> - add link to the networking/index.rst
> ---
>  Documentation/networking/diagnostic/index.rst |   17 +
>  .../twisted_pair_layer1_diagnostics.rst       | 1756 +++++++++++++++++
>  Documentation/networking/index.rst            |    1 +
>  3 files changed, 1774 insertions(+)
>  create mode 100644 Documentation/networking/diagnostic/index.rst
>  create mode 100644
> Documentation/networking/diagnostic/twisted_pair_layer1_diagnostics.rst
> 
> diff --git a/Documentation/networking/diagnostic/index.rst
> b/Documentation/networking/diagnostic/index.rst
> new file mode 100644
> index 0000000000000..86488aa46b484
> --- /dev/null
> +++ b/Documentation/networking/diagnostic/index.rst
> @@ -0,0 +1,17 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +======================
> +Networking Diagnostics
> +======================
> +
> +.. toctree::
> +   :maxdepth: 2
> +
> +   twisted_pair_layer1_diagnostics.rst
> +
> +.. only::  subproject and html
> +
> +   Indices
> +   =======
> +
> +   * :ref:`genindex`
> diff --git
> a/Documentation/networking/diagnostic/twisted_pair_layer1_diagnostics.rst
> b/Documentation/networking/diagnostic/twisted_pair_layer1_diagnostics.rst
> new file mode 100644
> index 0000000000000..ec962400d8907
> --- /dev/null
> +++
> b/Documentation/networking/diagnostic/twisted_pair_layer1_diagnostics.rst
> @@ -0,0 +1,1756 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Diagnostic Concept for Investigating Twisted Pair Ethernet Variants at OSI
> Layer 1
> +===============================================================
> ===================
> +
> +Introduction
> +------------
> +
> +This documentation is designed for two primary audiences:
> +
> +1. **Users and System Administrators**: For those dealing with real-world
> +   Ethernet issues, this guide provides a practical, step-by-step
> +   troubleshooting flow to help identify and resolve common problems in
> Twisted
> +   Pair Ethernet at OSI Layer 1. If you're facing unstable links, speed drops,
> +   or mysterious network issues, jump right into the step-by-step guide and
> +   follow it through to find your solution.
> +
> +2. **Kernel Developers**: For developers working with network drivers and
> PHY
> +   support, this documentation outlines the diagnostic process and highlights
> +   areas where the Linux kernel’s diagnostic interfaces could be extended or
> +   improved. By understanding the diagnostic flow, developers can better
> +   prioritize future enhancements.
> +
> +Structure
> +------------
> +
> +The document is divided into two parts:
> +
> +- **Part 1: Step-by-Step Troubleshooting Guide**: This is for those who
> need
> +  immediate help with their networking issues. We get it - no one wants to
> sift
> +  through endless technical descriptions when the link is down. This section
> +  provides concise, actionable steps to get your network up and running as
> +  quickly as possible.
> +
> +- **Part 2: In-Depth Descriptions and Technical Insights**: For those who
> want
> +  to dig deeper, this section offers detailed descriptions of hardware
> +  specifications, typical problems, and diagnostic techniques. It provides
> +  context for the issues addressed in the troubleshooting guide and serves as
> a
> +  reference for developers and network engineers who need to understand
> the
> +  reasoning behind each step.
> +
> +By the end of this documentation, whether you’re a frustrated admin or a
> +curious developer, we hope you’ll have the tools and understanding
> necessary to
> +diagnose and resolve twisted pair Ethernet issues at the physical layer. And if
> +you discover new or unresolved problems, feel free to extend this
> documentation
> +with your findings!
> +
> +Step-by-Step Diagnostic Guide from Linux (General Ethernet)
> +-----------------------------------------------------------
> +
> +This diagnostic guide covers common Ethernet troubleshooting scenarios,
> +focusing on **link stability and detection** across different Ethernet
> +environments, including **Single-Pair Ethernet (SPE)** and **Multi-Pair
> +Ethernet (MPE)**, as well as power delivery technologies like **PoDL**
> (Power
> +over Data Line) and **PoE** (Clause 33 PSE).
> +
> +The guide is designed to help users diagnose physical layer (Layer 1) issues
> on
> +systems running **Linux kernel version 6.11 or newer**, utilizing **ethtool
> +version 6.10 or later** and **iproute2 version 6.4.0 or later**.
> +
> +In this guide, we assume that users may have **limited or no access to the
> link
> +partner** and will focus on diagnosing issues locally.
> +
> +Diagnostic Scenarios
> +~~~~~~~~~~~~~~~~~~~~
> +
> +- **Link is up and stable, but no data transfer**: If the link is stable but
> +  there are issues with data transmission, refer to the **OSI Layer 2
> +  Troubleshooting Guide**.
> +
> +- **Link is unstable**: Link resets, speed drops, or other fluctuations
> +  indicate potential issues at the hardware or physical layer.
> +
> +- **No link detected**: The interface is up, but no link is established.
> +
> +Verify Interface Status
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Begin by verifying the status of the Ethernet interface to check if it is
> +administratively up. Unlike `ethtool`, which provides information on the link
> +and PHY status, it does not show the **administrative state** of the
> interface.
> +To check this, you should use the `ip` command, which describes the
> interface
> +state within the angle brackets `"<>"` in its output.
> +
> +For example, in the output `<NO-CARRIER,BROADCAST,MULTICAST,UP>`, the
> important
> +keywords are:
> +
> +- **UP**: The interface is in the administrative "UP" state.
> +- **NO-CARRIER**: The interface is administratively up, but no physical link
> is
> +  detected.
> +
> +If the output shows `<BROADCAST,MULTICAST>`, this indicates the interface
> is in
> +the administrative "DOWN" state.
> +
> +- **Command:** `ip link show dev <interface>`
> +
> +- **Expected Output:**
> +
> +  .. code-block:: bash
> +
> +     4: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 ...
> +        link/ether 88:14:2b:00:96:f2 brd ff:ff:ff:ff:ff:ff
> +
> +- **Interpreting the Output:**
> +
> +  - **Administrative UP State**:
> +
> +    - If the output contains **"UP"**, the interface is administratively up,
> +      and the system is trying to establish a physical link.
> +
> +    - If you also see **"NO-CARRIER"**, it means the physical link has not
> been
> +      detected, indicating potential Layer 1 issues like a cable fault,
> +      misconfiguration, or no connection at the link partner. In this case,
> +      proceed to the **Inspect Link Status and PHY Configuration** section.
> +
> +  - **Administrative DOWN State**:
> +
> +    - If the output lacks **"UP"** and shows only states like
> +      **"<BROADCAST,MULTICAST>"**, it means the interface is
> administratively
> +      down. In this case, bring the interface up using the following command:
> +
> +      .. code-block:: bash
> +
> +         ip link set dev <interface> up
> +
> +- **Next Steps**:
> +
> +  - If the interface is **administratively up** but shows **NO-CARRIER**,
> +    proceed to the **Inspect Link Status and PHY Configuration** section to
> +    troubleshoot potential physical layer issues.
> +
> +  - If the interface was **administratively down** and you have brought it
> up,
> +    ensure to **repeat this verification step** to confirm the new state of the
> +    interface before proceeding
> +
> +  - **If the interface is up and the link is detected**:
> +
> +    - If the output shows **"UP"** and there is **no `NO-CARRIER`**, the
> +      interface is administratively up, and the physical link has been
> +      successfully established. If everything is working as expected, the Layer
> +      1 diagnostics are complete, and no further action is needed.
> +
> +    - If the interface is up and the link is detected but **no data is being
> +      transferred**, the issue is likely beyond Layer 1, and you should proceed
> +      with diagnosing the higher layers of the OSI model. This may involve
> +      checking Layer 2 configurations (such as VLANs or MAC address issues),
> +      Layer 3 settings (like IP addresses, routing, or ARP), or Layer 4 and
> +      above (firewalls, services, etc.).
> +
> +    - If the **link is unstable** or **frequently resetting or dropping**, this
> +      may indicate a physical layer issue such as a faulty cable, interference,
> +      or power delivery problems. In this case, proceed with the next step in
> +      this guide.
> +
> +Inspect Link Status and PHY Configuration
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Use `ethtool -I` to check the link status, PHY configuration, supported link
> +modes, and additional statistics such as the **Link Down Events** counter.
> This
> +step is essential for diagnosing Layer 1 problems such as speed mismatches,
> +duplex issues, and link instability.
> +
> +For both **Single-Pair Ethernet (SPE)** and **Multi-Pair Ethernet (MPE)**
> +devices, you will use this step to gather key details about the link. **SPE**
> +links generally support a single speed and mode without autonegotiation
> (with
> +the exception of **10BaseT1L**), while **MPE** devices typically support
> +multiple link modes and autonegotiation.
> +
> +- **Command:** `ethtool -I <interface>`
> +
> +- **Example Output for SPE Interface (Non-autonegotiation)**:
> +
> +  .. code-block:: bash
> +
> +     Settings for spe4:
> +         Supported ports: [ TP ]
> +         Supported link modes:   100baseT1/Full
> +         Supported pause frame use: No
> +         Supports auto-negotiation: No
> +         Supported FEC modes: Not reported
> +         Advertised link modes: Not applicable
> +         Advertised pause frame use: No
> +         Advertised auto-negotiation: No
> +         Advertised FEC modes: Not reported
> +         Speed: 100Mb/s
> +         Duplex: Full
> +         Auto-negotiation: off
> +         master-slave cfg: forced slave
> +         master-slave status: slave
> +         Port: Twisted Pair
> +         PHYAD: 6
> +         Transceiver: external
> +         MDI-X: Unknown
> +         Supports Wake-on: d
> +         Wake-on: d
> +         Link detected: yes
> +         SQI: 7/7
> +         Link Down Events: 2
> +
> +- **Example Output for MPE Interface (Autonegotiation)**:
> +
> +  .. code-block:: bash
> +
> +     Settings for eth1:
> +         Supported ports: [ TP    MII ]
> +         Supported link modes:   10baseT/Half 10baseT/Full
> +                                 100baseT/Half 100baseT/Full
> +         Supported pause frame use: Symmetric Receive-only
> +         Supports auto-negotiation: Yes
> +         Supported FEC modes: Not reported
> +         Advertised link modes:  10baseT/Half 10baseT/Full
> +                                 100baseT/Half 100baseT/Full
> +         Advertised pause frame use: Symmetric Receive-only
> +         Advertised auto-negotiation: Yes
> +         Advertised FEC modes: Not reported
> +         Link partner advertised link modes:  10baseT/Half 10baseT/Full
> +                                              100baseT/Half 100baseT/Full
> +         Link partner advertised pause frame use: Symmetric Receive-only
> +         Link partner advertised auto-negotiation: Yes
> +         Link partner advertised FEC modes: Not reported
> +         Speed: 100Mb/s
> +         Duplex: Full
> +         Auto-negotiation: on
> +         Port: Twisted Pair
> +         PHYAD: 10
> +         Transceiver: internal
> +         MDI-X: Unknown
> +         Supports Wake-on: pg
> +         Wake-on: p
> +         Link detected: yes
> +         Link Down Events: 1
> +
> +- **Interpreting the ethtool output**:
> +
> +  - **Supported ports**: Specifies the physical connection type, such as
> +    **Twisted Pair (TP)**.
> +
> +  - **Supported link modes**:
> +
> +    - For **SPE**: This typically indicates one supported mode.
> +    - For **MPE**: Multiple link modes are supported, such as
> **10baseT/Half,
> +      10baseT/Full, 100baseT/Half, 100baseT/Full**.
> +
> +  - **Supported pause frame use**: Not used for layer 1 diagnostic
> +
> +  - **Supports auto-negotiation**:
> +
> +    - For most **SPE** links (e.g., **100baseT1**), autonegotiation is **not
> +      supported**.

Just for your reference,  we are submitting patches for lan887x(100/1000BaseT1) and soon we will add support for auto-negotiation as well.

> +
> +    - For **10BaseT1L** and **MPE** links, autonegotiation is typically
> +      **Yes**, allowing dynamic negotiation of speed and duplex settings.
> +
> +  - **Supported FEC modes**: Forward Error Correction (FEC). Currently not
> +    used on this guide.
> +
> +  - **Advertised link modes**:
> +
> +    - For **SPE** (except **10BaseT1L**), this field will be **Not
> +      applicable**, as no link modes can be advertised without
> autonegotiation.
> +
> +    - For **MPE** and **10BaseT1L** links, this will list the link modes that
> +      the interface is currently advertising to the link partner.
> +
> +  - **Advertised pause frame use**: Not used for layer 1 diagnostic
> +
> +  - **Advertised auto-negotiation**:
> +
> +    - For **SPE** links (except **10BaseT1L**), this will be **No**.
> +

May be to generalize statement for T1 phys, I would suggest it should be referred as "Yes" in case of auto-negotiation is enabled, "No" if auto-negotiation is disabled.

We are submitting patches for lan887x(100/1000BaseT1) and soon we will add support for auto-negotiation as well.

> +    - For **MPE** and **10BaseT1L** links, this will be **Yes** if
> +      autonegotiation is enabled.
> +
........
> +  - **Auto-negotiation**: Indicates whether auto-negotiation is enabled on
> the
> +    **local interface**. This shows that the interface is set to negotiate
> +    speed and duplex settings with the link partner. However, even if
> +    **auto-negotiation** is enabled locally and the link is established, the
> +    link partner might not be using auto-negotiation. In such cases, many PHYs
> +    are capable of detecting a **forced mode** on the link partner and
> +    adjusting to the correct speed and duplex.
> +
> +    If the link partner is in **forced mode**, the **"Link partner
> +    advertised"** fields will not be present in the `ethtool` output, as the
> +    partner isn't advertising any link modes or capabilities. Additionally, the
> +    **"Link partner advertised"** fields may also be missing if the **PHY
> +    driver** does not support reporting this information, or if the **MAC
> +    driver** is not utilizing the Linux **PHYlib** framework to retrieve and
> +    report the PHY status.
> +
> +  - **Master-slave configuration**: Indicates the current configuration of the
> +    **master-slave role** for the interface. This is relevant for certain
> +    Ethernet standards, such as **Single-Pair Ethernet (SPE)** and high-speed
> +    Ethernet configurations like **1000Base-T** and above, where one device
> +    must act as the **master** and the other as the **slave** for proper link
> +    establishment.
> +
> +    In **auto-negotiation** mode, the master-slave role is typically
> negotiated
> +    automatically. However, there are options to specify **preferred-master**
> +    or **preferred-slave** roles. For example, switches often prefer the
> master
> +    role to reduce the time domain crossing delays.
> +
> +    In **forced mode**, it is essential to manually configure the master-slave
> +    roles correctly on both link partners. If both sides are forced to the same
> +    role (e.g., both forced to master), the link will fail to establish.
> +
> +    A combination of **auto-negotiation** with **forced roles** can lead to
> +    unexpected behavior. If one side forces a role while the other side uses
> +    auto-negotiation, it can result in mismatches, especially if both sides
> +    force overlapping roles (preferring overlapping roles is usually not a
> +    problem). This configuration should be avoided to ensure reliable link
> +    establishment.
> +
> +  - **Master-slave status**: Displays the current **master-slave role** of the
> +    interface, indicating whether the interface is operating as the **master**
> +    or the **slave**. This field is particularly relevant in **auto-negotiation
> +    mode**, where the master-slave role is determined dynamically during
> the
> +    negotiation process.
> +

You may want to consider following case.

If both sides of interface are in same role then there will be resolution error status as below

        master-slave cfg: forced master
        master-slave status: resolution error
                            
> +    In **auto-negotiation**, the role is chosen based on the configuration
> +    preferences of both link partners (e.g., **preferred-master** or
> +    **preferred-slave**). The **master-slave status** field shows the
> outcome
> +    of this negotiation.
> +
> +    In **forced mode**, the master-slave configuration is manually set, so the
> +    **status** and **configuration** will always be the same, making this
> field
> +    less relevant in that case.
> +
...
> +

/Divya