Re: i.MX28 based system losing eth0 on boot

From: Florian Fainelli
Date: Wed May 07 2014 - 15:35:19 EST

Next message: Kevin Hilman: "Re: IPMI misbehaving on ARM defconfigs, was [Re: stable boot: 18 pass, 4 fail (v3.10.39)]"
Previous message: Ingo Molnar: "Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event"
In reply to: Brian Lilly: "Re: i.MX28 based system losing eth0 on boot"
Next in thread: Brian Lilly: "Re: i.MX28 based system losing eth0 on boot"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

2014-05-07 12:16 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
> Also, in 3.14, commenting out both "return -ETIMEDOUT" instances in
> fec_main.c results in a working interface.
> Please let me know if you have any questions.

At this point, you could probably instrument the interrupt handler and
see if you get FEC_MDIO interrupt causes at all?

>
> Thank you.
>
> Brian Lilly
> Crystalfontz America, Incorporated
> 12412 East Saltese Road
> Spokane Valley, WA 99216
> brian@xxxxxxxxxxxxxxxx http://www.crystalfontz.com
> Twitter: @Crystalfontz
> US toll-free (888) 206-9720 voice (509) 892-1200
>
>
> On Tue, May 6, 2014 at 8:07 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>> 2014-05-06 15:27 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>> It would appear that I don't have that commit. I could move to 3.14
>>> to see if it makes a difference, but the last couple of responses have
>>> been on 3.12.18 -- or perhaps I'm missing something else.
>>
>> I did miss that you were also seeing the problem in 3.12. At that
>> point, I believe that the driver was working around a potential PHY
>> bug that is not covered by the SMSC PHY driver, or that the MDIO
>> timeout is simply not long enough, or that your MDIO interrupts fire
>> much longer than what the timeout allows, or that these interrupts are
>> not reliable.
>>
>> You could probably try to ignore the timeout and see if you get
>> sensible data out of the MDIO bus regardless.
>>
>>> Please let me know if you have any questions.
>>>
>>> Thank you.
>>>
>>> Brian Lilly
>>> Crystalfontz America, Incorporated
>>> 12412 East Saltese Road
>>> Spokane Valley, WA 99216
>>> brian@xxxxxxxxxxxxxxxx http://www.crystalfontz.com
>>> Twitter: @Crystalfontz
>>> US toll-free (888) 206-9720 voice (509) 892-1200
>>>
>>>
>>> On Tue, May 6, 2014 at 3:06 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>>>> 2014-05-06 14:40 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>>>> The PHY on board is the SMSC LAN8720
>>>>>
>>>>> With the generic PHY driver selected: http://pastebin.com/A4MH4Ptw
>>>>>
>>>>> [ 28.828761] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>> [ 28.840626] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>> [ 30.827536] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>> [ 30.833739] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>> [ 32.986999] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>> [ 37.316421] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>> [ 38.345047] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>> [ 39.506210] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>> [ 40.374961] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>
>>>>> With the SMSC PHY driver selected: http://pastebin.com/DhdDyrMv
>>>>>
>>>>> [ 28.778974] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>> [ 28.791742] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>> [ 30.773078] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>> [ 30.779286] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>> [ 32.934692] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>> [ 37.242162] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>> [ 38.270611] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>> [ 39.415256] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>> [ 40.300454] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>
>>>> Thanks for trying this, at least this is consistent no matter which
>>>> PHY driver we are using. Just to rule out a potential PHY power-down
>>>> issue, could you try to revert the following commit
>>>> be9dad1f9f26604fb71c0d53ccb39a8f1d425807 ("net: phy: suspend phydev
>>>> when going to HALTED") and see if that works better for you?
>>>>
>>>> Thanks!
>>>>
>>>>>
>>>>> On Tue, May 6, 2014 at 12:24 PM, Florian Fainelli <f.fainelli@xxxxxxxxx> wrote:
>>>>>> 2014-05-06 12:12 GMT-07:00 Brian Lilly <brian@xxxxxxxxxxxxxxxx>:
>>>>>>> It is happening during boot up:
>>>>>>>
>>>>>>> <snip, kernel 3.12 >
>>>>>>>
>>>>>>> Configuring network interfaces... [ 35.117114] fec 800f0000.ethernet
>>>>>>> eth0: Freescale FEC PHY driver [SMSC LAN8710/LAN8720]
>>>>>>
>>>>>> Note that the SMSC PHY driver is picked up here, and that specific
>>>>>> driver implements a different phy_read_status() callback due to how
>>>>>> the PHY operates. The PHY driver also overrides the config_init()
>>>>>> callback to perform some PHY-specific initialization. See below for
>>>>>> more.
>>>>>>
>>>>>>> (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>> [ 35.129967] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>>> udhcpc (v1.21.1) started
>>>>>>>
>>>>>>> Sending discover...
>>>>>>>
>>>>>>> [ 37.113901] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>> [ 37.120134] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>>>>>>> Sending discover...
>>>>>>>
>>>>>>> Sending select for 10.10.10.217...
>>>>>>> Lease of 10.10.10.217 obtained, lease time 86400
>>>>>>> /etc/udhcpc.d/50default: Adding DNS 10.10.10.13
>>>>>>> [ 39.319957] IPv6: ADDRCONF(NETDEV_UP): usb0: link is not ready
>>>>>>> done.
>>>>>>> Starting rpcbind daemon...done.
>>>>>>> net.ipv4.conf.default.rp_filter = 1
>>>>>>> net.ipv4.conf.all.rp_filter = 1
>>>>>>> Mon Apr 14 22:40:00 UTC 2014
>>>>>>> INIT: Entering runlevel: 5
>>>>>>> Starting Xserver
>>>>>>> Starting system message bus: dbus.
>>>>>>> Starting Connection Manager
>>>>>>> Starting wpa_supplicant
>>>>>>> Successfully initialized wpa_supplicant
>>>>>>> Starting Dropbear SSH server
>>>>>>> [ 44.754915] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>> [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>
>>>>>> The correct PHY driver is selected here...
>>>>>>
>>>>>>> [ 45.781364] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>> [ 46.826170] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>>>>>>> [ 47.811385] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>
>>>>>> But we are still seeing MDIO read timeouts, which is not great.
>>>>>>
>>>>>>>
>>>>>>> With a different kernel (3.14):
>>>>>>>
>>>>>>> [ 28.989897] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>> [ 30.991210] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>> [ 37.369372] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>
>>>>>> Here, the Generic PHY driver has been selected, which will use the
>>>>>> MII_BMSR register contents to determine the Link status and
>>>>>> parameters. You might want to make sure that your board selects the
>>>>>> appropriate PHY driver, such that we are not chasing two issues here.
>>>>>>
>>>>>>> [ 38.398346] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>> [ 39.438412] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>> [ 39.468419] fec 800f0000.ethernet eth0: MDIO write timeout
>>>>>>> [ 40.498848] fec 800f0000.ethernet eth0: MDIO read timeout
>>>>>>
>>>>>> It would also be helpful to print the register that were accessed,
>>>>>> such that you could correlate this with the exact steps in the PHY
>>>>>> library state machine. Please also retry the experiment with the SMSC
>>>>>> PHY driver enabled, as it does some PHY specific initialization that
>>>>>> seems to be relevant. Then we are hopefully left with only the MDIO
>>>>>> timeout issue and not the PHY mis-configuration + MDIO timeout.
>>>>>>
>>>>>>>
>>>>>>> Afterward I have to ifdown eth0, ifup eth0 and then it functions
>>>>>>> normally, without reverting the commit.
>>>>>>>
>>>>>>> root@cfa100xx:~# ifdown eth0
>>>>>>> [ 1154.679658] fec 800f0000.ethernet eth0: Freescale FEC PHY driver
>>>>>>> [Generic PHY] (mii_bus:phy_addr=800f0000.etherne:00, irq=-1)
>>>>>>> root@cfa100xx:~# ifup eth0
>>>>>>> udhcpc (v1.21.1) started
>>>>>>> Sending discover...
>>>>>>> [ 1156.679547] libphy: 800f0000.etherne:00 - Link is Up - 100/Full
>>>>>>> Sending discover...
>>>>>>> Sending select for 10.10.10.217...
>>>>>>> Lease of 10.10.10.217 obtained, lease time 86400
>>>>>>> ip: RTNETLINK answers: File exists
>>>>>>>
>>>>>>> --
>>>>>>> Brian
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 6, 2014 at 11:11 AM, Uwe Kleine-KÃnig
>>>>>>> <u.kleine-koenig@xxxxxxxxxxxxxx> wrote:
>>>>>>>> Hello Brian,
>>>>>>>>
>>>>>>>> On Tue, May 06, 2014 at 09:44:34AM -0700, Brian Lilly wrote:
>>>>>>>>> With commit a264b981f2c76e281ef27e7232774bf6c54ec865 we're having eth0
>>>>>>>>> come up, then brought right back down with an MDIO rx timeout moments
>>>>>>>>> after. Adding back in the removed code keeps the interface alive and
>>>>>>>>> it's working afterward without trouble. I've tested the re-inserted
>>>>>>>>> code in 3.12, 3.14 without issue on our boards.
>>>>>>>> So you can reliably trigger that problem? You're just doing
>>>>>>>>
>>>>>>>> ifconfig eth0 1.2.3.4 up
>>>>>>>>
>>>>>>>> (or equivalent) and the interface goes down without further
>>>>>>>> interference with the above mentioned commit? The exact error you're
>>>>>>>> seeing is
>>>>>>>>
>>>>>>>> MDIO read timeout
>>>>>>>>
>>>>>>>> (with some prefix saying something about fec and eth0 I think)?
>>>>>>>>
>>>>>>>> This error is also present with a264b981f2 reverted, just doesn't affect
>>>>>>>> eth0 being functional? Does the timeout always happen, or only on
>>>>>>>> specific addresses?
>>>>>>>>
>>>>>>>> This is not a proper fix, but does it help to increment FEC_MII_TIMEOUT?
>>>>>>>>
>>>>>>>>> Is there something else that can be done to prevent the MDIO timeouts?
>>>>>>>>> We are using basically the same schematic for networking as the
>>>>>>>>> imx28evk.
>>>>>>>> Hard to say, but assuming it works just fine on the imx28evk for you,
>>>>>>>> too, there seems to be some hardware difference that makes your machine
>>>>>>>> fail. (That doesn't mean it's not fixable in software.)
>>>>>>>>
>>>>>>>> I don't know if a mdio read error is intended to make the device go
>>>>>>>> down, maybe one the the netdev guys can answer that.
>>>>>>>> Assuming that it's not intended, instrument the code, find out how that
>>>>>>>> timeout makes your device go down and find the wrong branch. I'd start
>>>>>>>> with adding stackdumps when the mdio timeout happens and when
>>>>>>>> fec_enet_start_xmit is called with fep->link == 0.
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>> Uwe
>>>>>>>>
>>>>>>>> --
>>>>>>>> Pengutronix e.K. | Uwe Kleine-KÃnig |
>>>>>>>> Industrial Linux Solutions | http://www.pengutronix.de/ |
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Florian
>>>>
>>>>
>>>>
>>>> --
>>>> Florian
>>
>>
>>
>> --
>> Florian

--
Florian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Kevin Hilman: "Re: IPMI misbehaving on ARM defconfigs, was [Re: stable boot: 18 pass, 4 fail (v3.10.39)]"
Previous message: Ingo Molnar: "Re: [PATCH v1 03/11] perf: Allow for multiple ring buffers per event"
In reply to: Brian Lilly: "Re: i.MX28 based system losing eth0 on boot"
Next in thread: Brian Lilly: "Re: i.MX28 based system losing eth0 on boot"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]