Regression: Bisected, IRQ and MSI allocations screwed without sparseirq

From: Jesper Dangaard Brouer
Date: Thu Nov 20 2008 - 14:49:18 EST

Next message: Joerg Roedel: "[PATCH] x86: always define DECLARE_PCI_UNMAP* macros"
Previous message: Russell King - ARM Linux: "Re: O_DIRECT patch for processors with VIPT cache for mainline kernel (specifically arm in our case)"
In reply to: David Miller: "Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf+ regression IRQs)"
Next in thread: Thomas Gleixner: "Re: Regression: Bisected, IRQ and MSI allocations screwed withoutsparse irq"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hi Thomas Gleixner,

I have bisected a regression to your commit
3235e936c0cc3589309280b6f59e5096779adae3,
"x86: remove sparse irq from Kconfig".

Its actually not necessary your fault, as your commit simply removes
the config option HAVE_SPARSE_IRQ. This revels the bug / regression
I'm exposted to.

Guess I should bisect again to find the exact faulty commit, but I'm
rather sick of bisecting at the moment, and though you might have a
better idea whats going wrong. I would rather spend my time
performance tuning the multiqueue routing code...

[The regression]:

During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
get really good performance (900-1200kpps) compared to 2.6.28 (davem
git net-2.6).

The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs. Due the reduced number of IRQs the NIU driver cannot get
enough IRQs to the interfaces, and starts to use "IO-APIC" based
IRQs.

On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
the usb driver. That my performance problem on 2.6.28.

[Other related bugs]:
Is that unloading the "niu" driver will give a kernel BUG during
deallocation og MSI interrupts. (See dmesg output below if interested)

(I have attached full bisect history)

Cheers,
Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

On Wed, 19 Nov 2008, David Miller wrote:

From: Jesper Dangaard Brouer <hawk@xxxxxxx>
Date: Wed, 19 Nov 2008 23:58:12 +0100 (CET)

Well that was not the real cause of the performance loss. Because
on kernel 2.6.27 I get really good performance (900-1200kpps)
compared to 2.6.28 (git net-2.6).

The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs.

Due the reduced number of IRQs the NIU driver cannot get enough IRQs
to the interfaces, and starts to use "IO-APIC" based IRQs.

This is almost certainly related to the driver unload bug.

I know you ran into unbuildable/unbootable kernels during a bisect,
but you really need to track down this regression.

------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-)

Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0
EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
[<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
[<c021ef41>] ? pci_disable_msix+0x31/0x40
[<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
[<c0219c29>] ? pci_device_remove+0x19/0x40
[<c026b0d4>] ? __device_release_driver+0x54/0x80
[<c026b197>] ? driver_detach+0x97/0xa0
[<c026a475>] ? bus_remove_driver+0x75/0xa0
[<c026b609>] ? driver_unregister+0x39/0x40
[<c0219e51>] ? pci_unregister_driver+0x21/0x80
[<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
[<c0145d74>] ? sys_delete_module+0x114/0x1d0
[<c016810a>] ? remove_vma+0x3a/0x50
[<c0168c29>] ? do_munmap+0x189/0x1e0
[<c0103229>] ? sysenter_do_call+0x12/0x21
[<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]--- ~~ -*-text-*-

-------------------------------------------------------
Bisecting IRQ change:
What change reduced the IRQs
-------------------------------------------------------
Jesper Dangaard Brouer (jdb@xxxxxxx)
-------------------------------------------------------
$LastChangedRevision: 786 $
$Date: 2008-11-20 20:44:51 +0100 (Thu, 20 Nov 2008) $
-------------------------------------------------------

git clone
~~~~~~~~~

+---------
cd /var/kernels/git/davem
git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect-irqs
+---------

Description / Reason to find
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

During my testing of the Sun Neptune based NICs.

On kernel 2.6.27 I get really good performance (900-1200kpps)
compared to 2.6.28 (git net-2.6).

The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs.

Due the reduced number of IRQs the NIU driver cannot get enough IRQs
to the interfaces, and starts to use "IO-APIC" based IRQs.

On kernel 2.6.28:

My eth2 is using 10 IRQs all "PCI-MSI-edge".

BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared
with the usb driver...

Think thats must be my performance problem on 2.6.28.

Known: Good and bad
~~~~~~~~~~~~~~~~~~~

GOOD:
git bisect good v2.6.27

BAD:
git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43

[92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch
'tracing-v28-for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

HiSTORY:
~~~~~~~~

+--------
cd /var/kernels/git/davem/net-2.6-bisect-irqs/
git bisect start
git bisect good v2.6.27
+--------

+--------------
git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
Bisecting: 3220 revisions left to test after this
[af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
+--------------

CONFIG_LOCALVERSION="-bisect"

+-------------
cp ../net-2.6-bisect/.config .
script make_oldconfig_01
make oldconfig
exit
#Script done, file is make_oldconfig_01
+-------------

+----------------
time make -j6 bzImage modules
#
#real 9m22.739s
#user 16m56.776s
#sys 1m4.672s
+----------------

Booted kernel: GOOD: irqs and (niu rmmod good)

+----------------
git bisect good
Bisecting: 1614 revisions left to test after this
[36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
+----------------

Compiling:

+----------------
time make -j6 bzImage modules
+----------------

Booted kernel: GOOD: irqs and (niu rmmod good)

+-----------
git bisect good
Bisecting: 807 revisions left to test after this
[1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
+-----------

Compiling...

+----------------
time make -j6 bzImage modules
#real 10m1.561s
#user 17m23.293s
#sys 1m5.744s
+----------------

Installing...

Booted kernel:

+----
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #3 SMP PREEMPT Thu Nov 20 12:33:02 CET 2008 i686 GNU/Linux
+----

Results: GOOD: irqs and (niu rmmod good)

+------
git bisect good
Bisecting: 403 revisions left to test after this
[1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
+------

Compiling...

+----------------
time make -j6 bzImage modules
#real 10m9.371s
#user 17m21.781s
#sys 1m6.052s
+----------------

Installing...

Booting ...

+-------
dcu-router-ng:~# uname -a
Linux dcu-router-ng 2.6.27-bisect #4 SMP PREEMPT Thu Nov 20 12:50:39 CET 2008 i686 GNU/Linux
+-------

Results: GOOD: irqs and (niu rmmod good)

+-------
git-bisect good
Bisecting: 223 revisions left to test after this
[dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
+-------

+----------------
time make -j6 bzImage modules
+----------------

Booting ...

+--------
Linux dcu-router-ng 2.6.27-bisect #5 SMP PREEMPT Thu Nov 20 13:58:34 CET 2008 i686 GNU/Linux
+--------

Results: BAD: irqs and (niu rmmod also BAD)

+-------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 125 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 1 2 2 IO-APIC-edge serial
8: 0 2 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 103 108 108 112 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6, eth0
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, eth3
20: 0 0 0 0 PCI-MSI-edge eth2
21: 0 0 0 0 PCI-MSI-edge eth2
22: 24 23 23 23 IO-APIC-fasteoi uhci_hcd:usb5, eth2
23: 0 0 0 0 PCI-MSI-edge eth2
24: 0 0 0 0 PCI-MSI-edge eth2
25: 0 0 0 0 PCI-MSI-edge eth2
26: 0 0 0 0 PCI-MSI-edge eth2
27: 0 0 0 0 PCI-MSI-edge eth2
28: 0 0 0 0 PCI-MSI-edge eth2
29: 0 0 0 0 PCI-MSI-edge eth2
30: 0 0 0 0 PCI-MSI-edge eth2
31: 0 0 0 0 PCI-MSI-edge eth2
32: 0 0 0 0 PCI-MSI-edge eth2
34: 271 268 268 264 PCI-MSI-edge cciss0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3301 2970 2594 2389 Local timer interrupts
RES: 28 560 6 13 Rescheduling interrupts
CAL: 50 104 99 62 Function call interrupts
TLB: 241 224 287 279 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+-------

OUTPUT "rmmod niu" (gives segfault) and "dmesg"

+-------
------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:632!
invalid opcode: 0000 [#1] PREEMPT SMP
Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-)

Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5
EIP: 0060:[<c021ecac>] EFLAGS: 00010286 CPU: 2
EIP is at msi_free_irqs+0xdc/0xe0
EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500
ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000)
Stack:
f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000
f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858
f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4
Call Trace:
[<c021ecb8>] ? msix_free_all_irqs+0x8/0x10
[<c021ef41>] ? pci_disable_msix+0x31/0x40
[<f809d3f8>] ? niu_pci_remove_one+0x88/0x8a [niu]
[<c0219c29>] ? pci_device_remove+0x19/0x40
[<c026b0d4>] ? __device_release_driver+0x54/0x80
[<c026b197>] ? driver_detach+0x97/0xa0
[<c026a475>] ? bus_remove_driver+0x75/0xa0
[<c026b609>] ? driver_unregister+0x39/0x40
[<c0219e51>] ? pci_unregister_driver+0x21/0x80
[<f809a0ad>] ? niu_exit+0xd/0x10 [niu]
[<c0145d74>] ? sys_delete_module+0x114/0x1d0
[<c016810a>] ? remove_vma+0x3a/0x50
[<c0168c29>] ? do_munmap+0x189/0x1e0
[<c0103229>] ? sysenter_do_call+0x12/0x21
[<c0330000>] ? quirk_disable_msi+0x30/0x50
Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55
EIP: [<c021ecac>] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8
---[ end trace f72de2e283920207 ]---
+-------

+------
git-bisect bad
Bisecting: 89 revisions left to test after this
[db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
+------

Related config change? (make oldconfig)

+------
script make_oldconfig_02
make oldconfig
Script done, file is make_oldconfig_02
+------

+------
Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) ? ?Y

This enables support for sparse irq, esp for msi/msi-x. the irq
number will be bus/dev/fn + 12bit. You may need if you have lots of
cards supports msi-x installed.

If you don't know what to do here, say Y.
+------

Compiling...

+----------------
time make -j6 bzImage modules
#
#real 9m29.556s
#user 17m10.396s
#sys 1m5.056s
+----------------

Booting ...

+-------
Linux dcu-router-ng 2.6.27-bisect #6 SMP PREEMPT Thu Nov 20 14:25:40 CET 2008 i686 GNU/Linux
+-------

The output from /proc/interrupts changed, very weird!
BUT eth3 does use a "PCI-MSI-edge" interrupt.

Guess this is a GOOD state even though it looks weird.

Unloading NIU driver also GOOD.

+---------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0x0: 124 1 0 0 IO-APIC-edge timer
0x1: 1 0 0 1 IO-APIC-edge i8042
0x3: 2 1 2 2 IO-APIC-edge serial
0x8: 1 0 0 1 IO-APIC-edge rtc
0x9: 0 0 0 0 IO-APIC-fasteoi acpi
0xc: 0 1 2 1 IO-APIC-edge i8042
0x10: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
0x11: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
0x12: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
0x6000fe: 288 289 290 290 PCI-MSI-edge cciss0
0x16: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
0xb00100: 0 0 0 0 PCI-MSI-edge eth2
0xb000ff: 0 0 0 0 PCI-MSI-edge eth2
0xb000fe: 0 0 0 0 PCI-MSI-edge eth2
0xb000fd: 0 0 0 0 PCI-MSI-edge eth2
0xb000fc: 0 0 0 0 PCI-MSI-edge eth2
0xb000fb: 0 0 0 0 PCI-MSI-edge eth2
0xb000fa: 0 0 0 0 PCI-MSI-edge eth2
0xb000f9: 0 0 0 0 PCI-MSI-edge eth2
0xb000f8: 0 0 0 0 PCI-MSI-edge eth2
0xb000f7: 0 0 0 0 PCI-MSI-edge eth2
0xb000f6: 0 0 0 0 PCI-MSI-edge eth2
0xb000f5: 0 0 0 0 PCI-MSI-edge eth2
0xb000f4: 0 0 0 0 PCI-MSI-edge eth2
0xb01100: 0 0 0 0 PCI-MSI-edge eth3
0xb010ff: 0 0 0 0 PCI-MSI-edge eth3
0xb010fe: 0 0 0 0 PCI-MSI-edge eth3
0xb010fd: 0 0 0 0 PCI-MSI-edge eth3
0xb010fc: 0 0 0 0 PCI-MSI-edge eth3
0xb010fb: 0 0 0 0 PCI-MSI-edge eth3
0xb010fa: 0 0 0 0 PCI-MSI-edge eth3
0xb010f9: 0 0 0 0 PCI-MSI-edge eth3
0xb010f8: 0 0 0 0 PCI-MSI-edge eth3
0xb010f7: 0 0 0 0 PCI-MSI-edge eth3
0xb010f6: 0 0 0 0 PCI-MSI-edge eth3
0x13: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
0x300100: 210 210 210 208 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3630 3265 3103 2711 Local timer interrupts
RES: 34 226 12 417 Rescheduling interrupts
CAL: 89 55 90 78 Function call interrupts
TLB: 253 205 311 267 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+---------

Guess it a GOOD situation...

+------
git-bisect good
Bisecting: 44 revisions left to test after this
[ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
+------

Compiling ...

+--------
time make -j6 bzImage modules
#real 9m28.062s
#user 17m7.492s
#sys 1m4.248s
+--------

Installing ...

Booting ...

+------
Linux dcu-router-ng 2.6.27-bisect #7 SMP PREEMPT Thu Nov 20 14:52:45 CET 2008 i686 GNU/Linux
+------

Still looks GOOD (/proc/interrupts still looks weird).

And rmmod NIU driver GOOD.

+------
git-bisect good
Bisecting: 22 revisions left to test after this
[922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
+------

Compiling ...

+--------
time make -j6 bzImage modules
#real 0m34.622s
#user 0m41.139s
#sys 0m5.812s
+--------

Install ...

Booting ...

+-----
Linux dcu-router-ng 2.6.27-bisect #8 SMP PREEMPT Thu Nov 20 15:04:11 CET 2008 i686 GNU/Linux
+-----

Looks GOOD, and /proc/interrupts changed again! Now the interrupts
are not i HEX anymore, but in decimal, but still strange/large
numbers for MSI.

Unloading NIU driver GOOD.

+------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 2 1 2 IO-APIC-edge serial
8: 0 0 1 1 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 828 821 828 823 PCI-MSI-edge cciss0
22: 23 24 24 22 IO-APIC-fasteoi uhci_hcd:usb5
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
3145984: 9993 9994 9987 9993 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 10075 11448 8001 8787 Local timer interrupts
RES: 297 17 349 26 Rescheduling interrupts
CAL: 173 189 95 173 Function call interrupts
TLB: 299 259 330 345 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+------

+-------
git-bisect good
Bisecting: 11 revisions left to test after this
[a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
+-------

Compiling

+--------
time make -j6 bzImage modules
#real 9m30.767s
#user 17m10.808s
#sys 1m6.388s
+--------

Installing ...

Booting ...

+-----
Linux dcu-router-ng 2.6.27-bisect #9 SMP PREEMPT Thu Nov 20 15:28:17 CET 2008 i686 GNU/Linux
+-----

BAD kernel version, max IRQ is 34. And eth3 got assigned a
IO-APIC-fasteoi shared with uhci_hcd:usb2.

Also BAD unloading of NIU driver.

BUG is some where in between:

git log 922402f15a85f7a064926eb1db68cc52bc4d4a91..a1aca5de08a0cb840a90fb3f729a5940f8d21185 | grep ^commit | wc -l

11 commits

+-------
git-bisect bad
Bisecting: 5 revisions left to test after this
[3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
+-------

Compiling...

+--------
time make -j6 bzImage modules
+--------

Install ...

Booting

+------
Linux dcu-router-ng 2.6.27-bisect #10 SMP PREEMPT Thu Nov 20 15:56:10 CET 2008 i686 GNU/Linux
+------

BAD kernel.

BAD rmmod NIU driver.

+---------
git bisect bad
Bisecting: 2 revisions left to test after this
[4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
+---------

Compiling...

+------
time make -j6 bzImage modules
#real 7m23.814s
#user 12m15.718s
#sys 0m42.183s
+------

Config change prompting:

+-----
Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) Y
+-----

Installing ...

Booting ...

+-------
Linux dcu-router-ng 2.6.27-bisect #11 SMP PREEMPT Thu Nov 20 16:19:29 CET 2008 i686 GNU/Linux
+-------

GOOD!!!

+--------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 2 2 2 2 IO-APIC-edge serial
8: 1 0 0 1 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 285 288 293 287 PCI-MSI-edge cciss0
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
22: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
3145984: 244 242 238 241 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3715 3104 2853 2542 Local timer interrupts
RES: 88 52 280 258 Rescheduling interrupts
CAL: 76 75 93 59 Function call interrupts
TLB: 245 241 312 283 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+---------

+---------
git-bisect good
Bisecting: 1 revisions left to test after this
[7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
+----------

Compiling ...

+------
time make -j6 bzImage modules
+------

Install...

Boot ...

+-------
Linux dcu-router-ng 2.6.27-bisect #12 SMP PREEMPT Thu Nov 20 16:33:11 CET 2008 i686 GNU/Linux
+-------

+--------
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 124 0 0 0 IO-APIC-edge timer
1: 0 0 1 1 IO-APIC-edge i8042
3: 1 2 2 2 IO-APIC-edge serial
8: 2 0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 IO-APIC-fasteoi acpi
12: 1 2 1 0 IO-APIC-edge i8042
16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6
17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2
18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3
6291710: 268 269 268 267 PCI-MSI-edge cciss0
11534592: 0 0 0 0 PCI-MSI-edge eth2
11534591: 0 0 0 0 PCI-MSI-edge eth2
11534590: 0 0 0 0 PCI-MSI-edge eth2
11534589: 0 0 0 0 PCI-MSI-edge eth2
11534588: 0 0 0 0 PCI-MSI-edge eth2
11534587: 0 0 0 0 PCI-MSI-edge eth2
11534586: 0 0 0 0 PCI-MSI-edge eth2
11534585: 0 0 0 0 PCI-MSI-edge eth2
11534584: 0 0 0 0 PCI-MSI-edge eth2
11534583: 0 0 0 0 PCI-MSI-edge eth2
11534582: 0 0 0 0 PCI-MSI-edge eth2
11534581: 0 0 0 0 PCI-MSI-edge eth2
11534580: 0 0 0 0 PCI-MSI-edge eth2
11538688: 0 0 0 0 PCI-MSI-edge eth3
11538687: 0 0 0 0 PCI-MSI-edge eth3
11538686: 0 0 0 0 PCI-MSI-edge eth3
11538685: 0 0 0 0 PCI-MSI-edge eth3
11538684: 0 0 0 0 PCI-MSI-edge eth3
11538683: 0 0 0 0 PCI-MSI-edge eth3
11538682: 0 0 0 0 PCI-MSI-edge eth3
11538681: 0 0 0 0 PCI-MSI-edge eth3
11538680: 0 0 0 0 PCI-MSI-edge eth3
11538679: 0 0 0 0 PCI-MSI-edge eth3
11538678: 0 0 0 0 PCI-MSI-edge eth3
19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4
22: 25 23 24 25 IO-APIC-fasteoi uhci_hcd:usb5
3145984: 175 174 176 178 PCI-MSI-edge eth0
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 3508 2902 2765 2489 Local timer interrupts
RES: 238 35 461 6 Rescheduling interrupts
CAL: 61 90 59 81 Function call interrupts
TLB: 257 220 299 300 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
SPU: 0 0 0 0 Spurious interrupts
ERR: 0
MIS: 0
+--------

GOOD.

+----------
git-bisect good
3235e936c0cc3589309280b6f59e5096779adae3 is first bad commit
commit 3235e936c0cc3589309280b6f59e5096779adae3
Author: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
Date: Wed Oct 15 13:16:00 2008 +0200

x86: remove sparse irq from Kconfig

This code is not ready yet.

Signed-off-by: Thomas Gleixner <tglx@xxxxxxxxxxxxx>

:040000 040000 6043e32465556e828de0fbb6aa497b277239af01 2dd75ba207990d83a3a4c7b7b16abccfe2d5e10d M arch
+--------

Found bad commit:
3235e936c0cc3589309280b6f59e5096779adae3

Git bisect LOG
~~~~~~~~~~~~~~

+-------
git-bisect log
git-bisect start
# good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27
git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5
# bad: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43
# good: [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2
git-bisect good af5c2bd16ac2e5688c3bf46ea1f95112d696d294
# good: [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input
git-bisect good 36ac1d2f323f8bf8bc10c25b88f617657720e241
# good: [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen()
git-bisect good 1aece34833721d64eb33fc15cd923c727296d3d3
# good: [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse
git-bisect good 1d9a8a47d659f053abeca9ece45651b4d94780c8
# bad: [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files
git-bisect bad dd3a1db900f2a215a7d7dd71b836e149a6cf5fed
# good: [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function
git-bisect good db4b5525caafd846ec20f95afbc6403c792e22cf
# good: [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI
git-bisect good ba374c9baef910fbc5373541d98c50f15e82c3f8
# good: [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4
git-bisect good 922402f15a85f7a064926eb1db68cc52bc4d4a91
# bad: [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal
git-bisect bad a1aca5de08a0cb840a90fb3f729a5940f8d21185
# bad: [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig
git-bisect bad 3235e936c0cc3589309280b6f59e5096779adae3
# good: [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out
git-bisect good 4c66a73f0796dacc2ff0d4af75794ec843ceb3d1
# good: [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n
git-bisect good 7ef0c30dbf96a8d9a234e90c248eb19df3c031be
+-------

Email
~~~~~

To:
Thomas Gleixner <tglx@xxxxxxxxxxxxx>
David Miller <davem@xxxxxxxxxxxxx>,
Jesper Dangaard Brouer <jdb@xxxxxxx>,
netdev <netdev@xxxxxxxxxxxxxxx>,
linux-kernel@xxxxxxxxxxxxxxx,
Robert Olsson <Robert.Olsson@xxxxxxxxxxx>

Subj.:
Regression: Bisected, IRQ and MSI allocations screwed without sparse irq

Hi Thomas Gleixner,

I have bisected a regression to your commit
3235e936c0cc3589309280b6f59e5096779adae3, "x86: remove sparse irq
from Kconfig".

Its actually not necessary your fault, as your commit simply removes
the config option HAVE_SPARSE_IRQ. This revels the bug / regression
I'm exposted to.

Guess I should bisect again to find the exact faulty commit, but I'm
rather sick of bisecting at the moment, and though you might have a
better idea whats going wrong. I would rather spend my time
performance tuning the multiqueue routing code...

[The regression]:

During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I
get really good performance (900-1200kpps) compared to 2.6.28 (davem
git net-2.6).

The cause of this problem (tracked down together with Robert Olsson)
is that on 2.6.28 I have a lot less IRQs available. It seems max 34
IRQs. Due the reduced number of IRQs the NIU driver cannot get
enough IRQs to the interfaces, and starts to use "IO-APIC" based
IRQs.

On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT
my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with
the usb driver. That my performance problem on 2.6.28.

[Other related bugs]:
Is that unloading the "niu" driver will give a kernel BUG.

Next message: Joerg Roedel: "[PATCH] x86: always define DECLARE_PCI_UNMAP* macros"
Previous message: Russell King - ARM Linux: "Re: O_DIRECT patch for processors with VIPT cache for mainline kernel (specifically arm in our case)"
In reply to: David Miller: "Re: NIU driver: Sun x8 Express Quad Gigabit Ethernet Adapter (perf+ regression IRQs)"
Next in thread: Thomas Gleixner: "Re: Regression: Bisected, IRQ and MSI allocations screwed withoutsparse irq"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]