Re: Realtek NIC uses over 1 Watt with no traffic

[Heiner Kallweit]

On 20.11.2018 21:31, Paul Menzel wrote:
> Dear Heiner,
> 
> 
> Thank you for your reply.
> 
> Am 20.11.18 um 21:14 schrieb Heiner Kallweit:
>> On 20.11.2018 15:45, Andrew Lunn wrote:
>>> On Tue, Nov 20, 2018 at 09:40:25AM +0100, Paul Menzel wrote:
> 
>>>> Using Ubuntu 18.10, Linux 4.18.0-11-generic, PowerTOP 2.9 shows, the NIC
>>>> uses 1.77 Watts. A network cable is plugged in, but there is no real traffic
>>>> according to `iftop`. Only an email program is running.
>>>>
>>>> ÂÂÂÂ $ lspci -nn -s 3:00.1
>>>> ÂÂÂÂ 03:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev
>>>> 12)
>>>>
>>>> Is that a measurement error, or does the NIC really need that much power?
> 
>>> This sounds like Energy Efficient Ethernet, EEE, is not enabled.
>>>
>>> What does ethtool --show-eee ethX say?
> 
> ÂÂÂ $ sudo ethtool --show-eee enp3s0f1
> ÂÂÂ Cannot get EEE settings: Operation not supported
> 
>> The r8169 driver doesn't support the get_eee ethtool_ops callback.
>> For certain chip versions EEE gets enabled in the PHY init, for others
>> not and some don't seem to support EEE at all.
>>
>> Apart from EEE one important factor affecting power consumption is ASPM.
>> This was recently enabled for certain chip versions.
>>
>> Information that would help:
>>
>> whether Wake-on-LAN is enabled ("Wake-on:" line from ethtool output)
> 
> ```
> $ sudo ethtool enp3s0f1
> Settings for enp3s0f1:
> ÂÂÂÂSupported ports: [ TP AUI BNC MII FIBRE ]
> ÂÂÂÂSupported link modes:ÂÂ 10baseT/Half 10baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 100baseT/Half 100baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1000baseT/Full
> ÂÂÂÂSupported pause frame use: Symmetric Receive-only
> ÂÂÂÂSupports auto-negotiation: Yes
> ÂÂÂÂSupported FEC modes: Not reported
> ÂÂÂÂAdvertised link modes:Â 10baseT/Half 10baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 100baseT/Half 100baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1000baseT/Full
> ÂÂÂÂAdvertised pause frame use: Symmetric Receive-only
> ÂÂÂÂAdvertised auto-negotiation: Yes
> ÂÂÂÂAdvertised FEC modes: Not reported
> ÂÂÂÂLink partner advertised link modes:Â 10baseT/Half 10baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 100baseT/Half 100baseT/Full
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1000baseT/Full
> ÂÂÂÂLink partner advertised pause frame use: Symmetric
> ÂÂÂÂLink partner advertised auto-negotiation: Yes
> ÂÂÂÂLink partner advertised FEC modes: Not reported
> ÂÂÂÂSpeed: 1000Mb/s
> ÂÂÂÂDuplex: Full
> ÂÂÂÂPort: MII
> ÂÂÂÂPHYAD: 0
> ÂÂÂÂTransceiver: internal
> ÂÂÂÂAuto-negotiation: on
> ÂÂÂÂSupports Wake-on: pumbg
> ÂÂÂÂWake-on: g
> ÂÂÂÂCurrent message level: 0x00000033 (51)
> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ drv probe ifdown ifup
> ÂÂÂÂLink detected: yes
> ```
> 
> So, itâs enabled (g Wake on MagicPacket(tm)).
> 
> Running `sudo ethtool -s enp3s0f1 wol d;` doesnât change anything though.
> 
>> lspci -vv output for the Realtek NIC
> 
> Here is the output (quoted, so that Thunderbird does not wrap the line).
> 
>> $ sudo lspci -vv -s 3:00.1
>> 03:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
>> ÂÂÂÂSubsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
>> ÂÂÂÂControl: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
>> ÂÂÂÂStatus: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>> ÂÂÂÂLatency: 0, Cache Line Size: 64 bytes
>> ÂÂÂÂInterrupt: pin A routed to IRQ 19
>> ÂÂÂÂRegion 0: I/O ports at e000 [size=256]
>> ÂÂÂÂRegion 2: Memory at df114000 (64-bit, non-prefetchable) [size=4K]
>> ÂÂÂÂRegion 4: Memory at df110000 (64-bit, non-prefetchable) [size=16K]
>> ÂÂÂÂCapabilities: [40] Power Management version 3
>> ÂÂÂÂÂÂÂ Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
>> ÂÂÂÂÂÂÂ Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
>> ÂÂÂÂCapabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
>> ÂÂÂÂÂÂÂ Address: 0000000000000000Â Data: 0000
>> ÂÂÂÂCapabilities: [70] Express (v2) Endpoint, MSI 01
>> ÂÂÂÂÂÂÂ DevCap:ÂÂÂ MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>> ÂÂÂÂÂÂÂÂÂÂÂ ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
>> ÂÂÂÂÂÂÂ DevCtl:ÂÂÂ Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
>> ÂÂÂÂÂÂÂÂÂÂÂ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
>> ÂÂÂÂÂÂÂÂÂÂÂ MaxPayload 128 bytes, MaxReadReq 4096 bytes
>> ÂÂÂÂÂÂÂ DevSta:ÂÂÂ CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
>> ÂÂÂÂÂÂÂ LnkCap:ÂÂÂ Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
>> ÂÂÂÂÂÂÂÂÂÂÂ ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
>> ÂÂÂÂÂÂÂ LnkCtl:ÂÂÂ ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+

L0s is missing here, no idea why.

>> ÂÂÂÂÂÂÂÂÂÂÂ ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
>> ÂÂÂÂÂÂÂ LnkSta:ÂÂÂ Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>> ÂÂÂÂÂÂÂ DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
>> ÂÂÂÂÂÂÂ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
>> ÂÂÂÂÂÂÂ LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
>> ÂÂÂÂÂÂÂÂÂÂÂÂ EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>> ÂÂÂÂCapabilities: [b0] MSI-X: Enable+ Count=4 Masked-
>> ÂÂÂÂÂÂÂ Vector table: BAR=4 offset=00000000
>> ÂÂÂÂÂÂÂ PBA: BAR=4 offset=00000800
>> ÂÂÂÂCapabilities: [d0] Vital Product Data
>> pcilib: sysfs_read_vpd: read failed: Input/output error
>> ÂÂÂÂÂÂÂ Not readable
>> ÂÂÂÂCapabilities: [100 v2] Advanced Error Reporting
>> ÂÂÂÂÂÂÂ UESta:ÂÂÂ DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> ÂÂÂÂÂÂÂ UEMsk:ÂÂÂ DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>> ÂÂÂÂÂÂÂ UESvrt:ÂÂÂ DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>> ÂÂÂÂÂÂÂ CESta:ÂÂÂ RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ NonFatalErr+
>> ÂÂÂÂÂÂÂ CEMsk:ÂÂÂ RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>> ÂÂÂÂÂÂÂ AERCap:ÂÂÂ First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>> ÂÂÂÂCapabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00
>> ÂÂÂÂCapabilities: [170 v1] Latency Tolerance Reporting
>> ÂÂÂÂÂÂÂ Max snoop latency: 3145728ns
>> ÂÂÂÂÂÂÂ Max no snoop latency: 3145728ns
>> ÂÂÂÂCapabilities: [178 v1] L1 PM Substates
>> ÂÂÂÂÂÂÂ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂ PortCommonModeRestoreTime=150us PortTPowerOnTime=150us
>> ÂÂÂÂÂÂÂ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ T_CommonMode=0us LTR1.2_Threshold=0ns
>> ÂÂÂÂÂÂÂ L1SubCtl2: T_PwrOn=10us
>> ÂÂÂÂKernel driver in use: r8169
>> ÂÂÂÂKernel modules: r8169
> 
> Some Active State Power Management levels seem to be enabled.
> 
>> Info from powertop about package C states. With ASPM my system reaches
>> 50% PC7 + 50% PC10.
> 
> That seems to be the case on my TUXEDO Book BU1406 too.
> 
>> ÂÂÂÂÂÂÂÂÂ PaketÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ KernÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ CPU 0ÂÂÂÂÂÂ CPU 2
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | C0 aktivÂÂÂ 1,7%ÂÂÂÂÂÂÂ 1,1%
>>  | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>  | | C1E 0,2% 0,8 ms 0,1% 0,2 ms
>> C2 (pc2)ÂÂÂ 5,2%ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>> C3 (pc3) 82,1% | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,1% 0,2 ms

Relevant are the package states and your system reaches pc3 only. The "Tunables" section
in powertop may provide hints how to save more power.

>> C6 (pc6) 0,0% | C6 (cc6) 1,3% | C6 0,8% 0,5 ms 1,4% 0,6 ms
>> C7 (pc7) 0,0% | C7 (cc7) 90,8% | C7s 0,0% 1,6 ms 0,0% 0,0 ms
>> C8 (pc8)ÂÂÂ 0,0%ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | C8ÂÂÂÂÂÂÂÂÂ 6,0%ÂÂÂ 1,8 ms 10,1%ÂÂÂ 2,0 ms
>> C9 (pc9) 0,0% | | C9 0,2% 2,8 ms 0,2% 2,9 ms
>> C10 (pc10)Â 0,0%ÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | C10ÂÂÂÂÂÂÂ 88,7%ÂÂ 12,7 ms 84,4%ÂÂ 14,9 ms
>>
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ KernÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂ CPU 1ÂÂÂÂÂÂ CPU 3
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | C0 aktivÂÂÂ 1,0%ÂÂÂÂÂÂÂ 0,8%
>>  | | POLL 0,0% 0,0 ms 0,0% 0,0 ms
>>  | | C1E 0,1% 0,3 ms 0,1% 0,3 ms
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>  | C3 (cc3) 0,0% | C3 0,0% 0,2 ms 0,0% 0,2 ms
>>  | C6 (cc6) 1,1% | C6 0,9% 0,6 ms 0,8% 0,5 ms
>>  | C7 (cc7) 92,2% | C7s 0,0% 1,7 ms 0,0% 0,0 ms
>>  | | C8 6,2% 1,7 ms 5,4% 1,7 ms
>>  | | C9 0,3% 1,7 ms 0,1% 1,9 ms
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | C10ÂÂÂÂÂÂÂ 88,8%ÂÂ 12,1 ms 90,7%ÂÂ 14,8 ms
>>
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂ GPUÂÂÂÂ |
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ |
>>  | Powered On 2,2% |
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | RC6ÂÂÂÂÂÂÂ 97,8%ÂÂÂ |
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | RC6pÂÂÂÂÂÂÂ 0,0%ÂÂÂ |
>> ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ | RC6ppÂÂÂÂÂÂ 0,0%ÂÂÂ |
> 
>> dmesg output filtered for "r8169". Primarily relevant is the line with
>> the chip name and XID.
> 
> Please find them below.
> 
>> $ sudo dmesg | grep r8169
>> [ 5.318442] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 418
>> [ÂÂÂ 5.318470] r8169 0000:03:00.1: enabling device (0000 -> 0003)
>> [ÂÂÂ 5.340324] libphy: r8169: probed
>> [ÂÂÂ 5.340630] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 136

Good to know. For this chip version rtl8168g_2_hw_phy_config() is used to configure the PHY,
but this function just loads the firmware. So we don't know whether EEE is enabled.

What you could do to test further is limiting the speed to 100MBit or 10MBit via ethtool.
If this reduces power consumption significantly it's a hint that indeed the PHY seems
to be the one to be blamed.

>> [ÂÂÂ 5.340632] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>> [ÂÂÂ 5.340673] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 9217 usecs
>> [ÂÂÂ 5.799967] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>> [ÂÂ 10.036968] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>> [ 676.940934] calling rtl8169_pci_driver_init+0x0/0x1000 [r8169] @ 22235
>> [Â 676.952411] libphy: r8169: probed
>> [Â 676.952701] r8169 0000:03:00.1 eth0: RTL8411, 80:fa:5b:3b:dd:f0, XID 5c800800, IRQ 139
>> [Â 676.952702] r8169 0000:03:00.1 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
>> [Â 676.952736] initcall rtl8169_pci_driver_init+0x0/0x1000 [r8169] returned 0 after 11518 usecs
>> [Â 676.954420] r8169 0000:03:00.1 enp3s0f1: renamed from eth0
>> [Â 676.975161] Generic PHY r8169-301:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-301:00, irq=IGNORE)
>> [Â 680.518923] r8169 0000:03:00.1 enp3s0f1: Link is Up - 1Gbps/Full - flow control rx/tx
>> [ 1751.285899] r8169 0000:03:00.1: invalid short VPD tag 00 at offset 1
> 
> 
> Kind regards,
> 
> Paul
> 

Heiner