Re: i.MX28 based system losing eth0 on boot

From: Brian Lilly
Date: Wed May 07 2014 - 15:52:26 EST

Next message: Thomas Gleixner: "Re: [PATCH V2] clocksource: register cpu notifier to remove timer from dying CPU"
Previous message: Elliott, Robert (Server Storage): "RE: [PATCH 5/5] x86, nmi: Add better NMI stats to /proc/interrupts and show handlers"
In reply to: Florian Fainelli: "Re: i.MX28 based system losing eth0 on boot"
Next in thread: fugang.duan@xxxxxxxxxxxxx: "RE: i.MX28 based system losing eth0 on boot"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Florian:

Thank you for your help.

After doubling the timeout length it worked.

I managed to get my hands on a imx28evk board and compared our
component load versus theirs, to find they have a 1.5k pull-up on
ENET_MDIO to +3.3v which wasn't present on our board. Adding a 1.5k
pull-up resistor on ENET_MDIO solves the problem, and boots as
expected without patching anything.

Sorry for the trouble on this.

Apparently our EE had some question as to whether or not the pull-up
was necessary, and put it in the schematic, and the footprint on the
board, but marked it as a DNP, which of course left it off the board
and out of the BOM.

<facepalm>

On Wed, May 7, 2014 at 12:34 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
> 2014-05-07 12:16 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>> Also, in 3.14, commenting out both "return -ETIMEDOUT" instances in
>> fec_main.c results in a working interface.
>> Please let me know if you have any questions.
>
> At this point, you could probably instrument the interrupt handler and
> see if you get FEC_MDIO interrupt causes at all?
>
>>
>> Thank you.
>>
>> Brian Lilly
>> Crystalfontz America, Incorporated
>> 12412 East Saltese Road
>> Spokane Valley, WA 99216
>> brian@xxxxxxxxxxxxxxxx http://www.crystalfontz.com
>> Twitter: @Crystalfontz
>> US toll-free (888) 206-9720 voice (509) 892-1200
>>
>>
>> On Tue, May 6, 2014 at 8:07 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>>> 2014-05-06 15:27 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>>> It would appear that I don't have that commit. I could move to 3.14
>>>> to see if it makes a difference, but the last couple of responses have
>>>> been on 3.12.18 -- or perhaps I'm missing something else.
>>>
>>> I did miss that you were also seeing the problem in 3.12. At that
>>> point, I believe that the driver was working around a potential PHY
>>> bug that is not covered by the SMSC PHY driver, or that the MDIO
>>> timeout is simply not long enough, or that your MDIO interrupts fire
>>> much longer than what the timeout allows, or that these interrupts are
>>> not reliable.
>>>
>>> You could probably try to ignore the timeout and see if you get
>>> sensible data out of the MDIO bus regardless.
>>>
>>>> Please let me know if you have any questions.
>>>>
>>>> Thank you.
>>>>
>>>> Brian Lilly
>>>> Crystalfontz America, Incorporated
>>>> 12412 East Saltese Road
>>>> Spokane Valley, WA 99216
>>>> brian@xxxxxxxxxxxxxxxx http://www.crystalfontz.com
>>>> Twitter: @Crystalfontz
>>>> US toll-free (888) 206-9720 voice (509) 892-1200
>>>>
>>>>
>>>> On Tue, May 6, 2014 at 3:06 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>>>>> 2014-05-06 14:40 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>>>>> The PHY on board is the SMSC LAN8720
>>>>>>
>>>>>> With the generic PHY driver selected: http://pastebin.com/A4MH4Ptw
>>>>>>
>>>>>> [ 28.828761] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>> [ 28.840626] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>> [ 30.827536] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>> [ 30.833739] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>>> [ 32.986999] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>>> [ 37.316421] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>> [ 38.345047] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>> [ 39.506210] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>> [ 40.374961] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>
>>>>>> With the SMSC PHY driver selected: http://pastebin.com/DhdDyrMv
>>>>>>
>>>>>> [ 28.778974] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>> [ 28.791742] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>> [ 30.773078] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>> [ 30.779286] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>>> [ 32.934692] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>>> [ 37.242162] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>> [ 38.270611] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>> [ 39.415256] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>> [ 40.300454] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>
>>>>> Thanks for trying this, at least this is consistent no matter which
>>>>> PHY driver we are using. Just to rule out a potential PHY power-down
>>>>> issue, could you try to revert the following commit
>>>>> be9dad1f9f26604fb71c0d53ccb39a8f1d425807 ("net: phy: suspend phydev
>>>>> when going to HALTED") and see if that works better for you?
>>>>>
>>>>> Thanks!
>>>>>
>>>>>>
>>>>>> On Tue, May 6, 2014 at 12:24 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>>>>>>> 2014-05-06 12:12 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>>>>>>> It is happening during boot up:
>>>>>>>>
>>>>>>>> <snip, kernel 3.12 >
>>>>>>>>
>>>>>>>> Configuring network interfaces... [ 35.117114] fec 800f0000.ethernet
>>>>>>>> eth0: Freescale FEC PHY driver [SMSC LAN8710/LAN8720]
>>>>>>>
>>>>>>> Note that the SMSC PHY driver is picked up here, and that specific
>>>>>>> driver implements a different phy_read_status() callback due to how
>>>>>>> the PHY operates. The PHY driver also overrides the config_init()
>>>>>>> callback to perform some PHY-specific initialization. See below for
>>>>>>> more.
>>>>>>>
>>>>>>>> (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>>> [ 35.129967] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>>>> udhcpc (v1.21.1) started
>>>>>>>>
>>>>>>>> Sending discover...
>>>>>>>>
>>>>>>>> [ 37.113901] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>>> [ 37.120134] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>>>>> Sending discover...
>>>>>>>>
>>>>>>>> Sending select for 10.10.10.217...
>>>>>>>> Lease of 10.10.10.217 obtained, lease time 86400
>>>>>>>> /etc/udhcpc.d/50default: Adding DNS 10.10.10.13
>>>>>>>> [ 39.319957] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>>>>> done.
>>>>>>>> Starting rpcbind daemon...done.
>>>>>>>> net.ipv4.conf.default.rp_filter = 1
>>>>>>>> net.ipv4.conf.all.rp_filter = 1
>>>>>>>> Mon Apr 14 22:40:00 UTC 2014
>>>>>>>> INIT: Entering runlevel: 5
>>>>>>>> Starting Xserver
>>>>>>>> Starting system message bus: dbus.
>>>>>>>> Starting Connection Manager
>>>>>>>> Starting wpa_supplicant
>>>>>>>> Successfully initialized wpa_supplicant
>>>>>>>> Starting Dropbear SSH server
>>>>>>>> [ 44.754915] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>>
>>>>>>> The correct PHY driver is selected here...
>>>>>>>
>>>>>>>> [ 45.781364] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>>> [ 46.826170] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>>>> [ 47.811385] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>>
>>>>>>> But we are still seeing MDIO read timeouts, which is not great.
>>>>>>>
>>>>>>>>
>>>>>>>> With a different kernel (3.14):
>>>>>>>>
>>>>>>>> [ 28.989897] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>>> [ 30.991210] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>>> [ 37.369372] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>>
>>>>>>> Here, the Generic PHY driver has been selected, which will use the
>>>>>>> MII_BMSR register contents to determine the Link status and
>>>>>>> parameters. You might want to make sure that your board selects the
>>>>>>> appropriate PHY driver, such that we are not chasing two issues here.
>>>>>>>
>>>>>>>> [ 38.398346] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>>> [ 39.438412] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>>> [ 39.468419] fec 800f0000.ethernet eth0: MDIO write timeout
>>>>>>>> [ 40.498848] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>>
>>>>>>> It would also be helpful to print the register that were accessed,
>>>>>>> such that you could correlate this with the exact steps in the PHY
>>>>>>> library state machine. Please also retry the experiment with the SMSC
>>>>>>> PHY driver enabled, as it does some PHY specific initialization that
>>>>>>> seems to be relevant. Then we are hopefully left with only the MDIO
>>>>>>> timeout issue and not the PHY mis-configuration + MDIO timeout.
>>>>>>>
>>>>>>>>
>>>>>>>> Afterward I have to ifdown eth0, ifup eth0 and then it functions
>>>>>>>> normally, without reverting the commit.
>>>>>>>>
>>>>>>>> root@cfa100xx:~# ifdown eth0
>>>>>>>> [ 1154.679658] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>>> root@cfa100xx:~# ifup eth0
>>>>>>>> udhcpc (v1.21.1) started
>>>>>>>> Sending discover...
>>>>>>>> [ 1156.679547] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>>> Sending discover...
>>>>>>>> Sending select for 10.10.10.217...
>>>>>>>> Lease of 10.10.10.217 obtained, lease time 86400
>>>>>>>> ip: RTNETLINK answers: File exists
>>>>>>>>
>>>>>>>> --
>>>>>>>> Brian
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, May 6, 2014 at 11:11 AM, Uwe Kleine-KÃnig
>>>>>>>> <u.kleine-koenig@xxxxxxxxxxxxxx> wrote:
>>>>>>>>> Hello Brian,
>>>>>>>>>
>>>>>>>>> On Tue, May 06, 2014 at 09:44:34AM -0700, Brian Lilly wrote:
>>>>>>>>>> With commit a264b981f2c76e281ef27e7232774bf6c54ec865 we're having eth0
>>>>>>>>>> come up, then brought right back down with an MDIO rx timeout moments
>>>>>>>>>> after. Adding back in the removed code keeps the interface alive and
>>>>>>>>>> it's working afterward without trouble. I've tested the re-inserted
>>>>>>>>>> code in 3.12, 3.14 without issue on our boards.
>>>>>>>>> So you can reliably trigger that problem? You're just doing
>>>>>>>>>
>>>>>>>>> ifconfig eth0 1.2.3.4 up
>>>>>>>>>
>>>>>>>>> (or equivalent) and the interface goes down without further
>>>>>>>>> interference with the above mentioned commit? The exact error you're
>>>>>>>>> seeing is
>>>>>>>>>
>>>>>>>>> MDIO read timeout
>>>>>>>>>
>>>>>>>>> (with some prefix saying something about fec and eth0 I think)?
>>>>>>>>>
>>>>>>>>> This error is also present with a264b981f2 reverted, just doesn't affect
>>>>>>>>> eth0 being functional? Does the timeout always happen, or only on
>>>>>>>>> specific addresses?
>>>>>>>>>
>>>>>>>>> This is not a proper fix, but does it help to increment FEC_MII_TIMEOUT?
>>>>>>>>>
>>>>>>>>>> Is there something else that can be done to prevent the MDIO timeouts?
>>>>>>>>>> We are using basically the same schematic for networking as the
>>>>>>>>>> imx28evk.
>>>>>>>>> Hard to say, but assuming it works just fine on the imx28evk for you,
>>>>>>>>> too, there seems to be some hardware difference that makes your machine
>>>>>>>>> fail. (That doesn't mean it's not fixable in software.)
>>>>>>>>>
>>>>>>>>> I don't know if a mdio read error is intended to make the device go
>>>>>>>>> down, maybe one the the netdev guys can answer that.
>>>>>>>>> Assuming that it's not intended, instrument the code, find out how that
>>>>>>>>> timeout makes your device go down and find the wrong branch. I'd start
>>>>>>>>> with adding stackdumps when the mdio timeout happens and when
>>>>>>>>> fec_enet_start_xmit is called with fep->link == 0.
>>>>>>>>>
>>>>>>>>> Best regards
>>>>>>>>> Uwe
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Pengutronix e.K. | Uwe Kleine-KÃnig |
>>>>>>>>> Industrial Linux Solutions | http://www.pengutronix.de/ |
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Florian
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Florian
>>>
>>>
>>>
>>> --
>>> Florian
>
>
>
> --
> Florian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Thomas Gleixner: "Re: [PATCH V2] clocksource: register cpu notifier to remove timer from dying CPU"
Previous message: Elliott, Robert (Server Storage): "RE: [PATCH 5/5] x86, nmi: Add better NMI stats to /proc/interrupts and show handlers"
In reply to: Florian Fainelli: "Re: i.MX28 based system losing eth0 on boot"
Next in thread: fugang.duan@xxxxxxxxxxxxx: "RE: i.MX28 based system losing eth0 on boot"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]