RE: Problems with ixgbe driver
From: Holger Kiehl
Date: Mon Jun 17 2013 - 05:11:52 EST
first, thank you for the quick help!
On Fri, 14 Jun 2013, Tantilov, Emil S wrote:
No, there is absolute no indication in dmesg or /var/log/messages. But here
From: netdev-owner@xxxxxxxxxxxxxxx [mailto:netdev-owner@xxxxxxxxxxxxxxx] On
Behalf Of Holger Kiehl
Sent: Friday, June 14, 2013 4:50 AM
Cc: linux-kernel; netdev@xxxxxxxxxxxxxxx
Subject: Problems with ixgbe driver
I have dual port 10Gb Intel network card on a 2 socket (Xeon X5690) with
a total of 12 cores. Hyperthreading is enabled so there are 24 cores.
The problem I have is that when other systems send large amount of data
the network with the intel ixgbe driver gets very slow. Ping times go up
from 0.2ms to appr. 60ms. Some FTP connections stall for more then 2
minutes. What is strange is that heatbeat is configured on the system
with a serial connection to another node and kernel always reports
If the network slows down so much there should be some indication in dmesg. Like Tx hangs perhaps.
Can you provide the output of dmesg and ethtool -S from the offending interface after the issue occurs?
the ethtool output when ping times go up:
root@helena:~# ethtool -S eth6
This was with the 3.15.1 driver and setting the combinde queue to 24 via
ethtool, as you suggested below.
Okay, thank you! I did not know this.
ttyS0: 4 input overrun(s)
when lot of data is send and the ping time goes up.
On the network there are three vlan's configured. The network is bonded
(active-backup) together with another HP NC523SFP 10Gb 2-port Server
Adapter. When I switch the network to this card the problem goes away.
Also the ttyS0 input overruns disappear. Note also both network cards
are connected to the same switch.
The system uses Scientific Linux 6.4 with kernel.org kernel. I noticed
this behavior with kernel 3.9.5 and 3.9.6-rc1. Before I did not notice
it because traffic always went over the HP NC523SFP qlcnic card.
In search for a solution to the problem I found a newer ixgbe driver
3.15.1 (3.9.6-rc1. has 3.11.33-k) and tried that. But it has the same
problem. However when I load the module as follows:
modprobe ixgbe RSS=8,8
the problem goes away. The kernel.org ixgbe driver does not offer this
option. Why? It seems that both drivers have problems on systems with
If you are using newer kernel and ethtool version you can use `ethtool -L ethX combined Y` to control the number of queues per interface.
Yes, I think so too. But what could it be? Please, just ask what other
24 cpu's. But I cannot believe that I am the only one who noticed this,
since ixgbe is widely used.
We run traffic with multiple queues all the time and I don't think what you are reporting is a generic issue. Most likely it's something related to your setup/system.
information I could provide. As I already mentioned earlier the ixgbe card
is bonded with a qlogic nic and I have two (not three) vlan configured over
over this bond. Maybe the following is useful (eth6 is the ixgbe driver):
root@helena:~# ethtool -k eth6
Features for eth6:
tx-checksum-ip-generic: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-scatter-gather-fraglist: off [fixed]
tx-tcp-ecn-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
fcoe-mtu: off [fixed]
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
Sorry, but I could not find out how I can open a new bug. I could just view
It would really be nice if one could set the RSS=8,8 option for kernel.org
ixgbe driver too. Or if someone could tell me where I can force the driver
to Receive Side Scaling to 8 even if it means editing the source code.
Below I have added some additional information. Please CC me since I
am not subscribed to any of these lists. And please do not hesitate
to ask if more information is needed.
I would suggest that you open up a bug at e1000.sf.net - describe your configuration and attach the relevant info (dmesg, ethtool -S, lspci etc). This would make it easier for us to follow.
existing bugs. Please give me a hint what I need to do.
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/