Kirkwood PCI Express and bridges
From: Chris Packham
Date: Fri Jun 21 2019 - 00:03:50 EST
Hi All,
I'm in the process of updating the kernel version used on our products
from 4.4 -> 5.1.
We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
Switch ASIC. The Switch ASIC presents as multiple PCI devices.
The hardware setup looks like this
__________
[ Kirkwood ] --- [ IDT 5T5 ] ---+--- | |
+--- | Switch |
+--- | |
+--- |__________|
On the 4.4 based kernel things are fine
[root@awplus flash]# lspci -t
-[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
+-03.0-[04]----00.0
+-04.0-[05]----00.0
\-05.0-[06]----00.0
But on the 5.1 based kernel things get a little weird
[root@awplus flash]# lspci -t
-[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
+-01.0
+-02.0-[02-06]--
+-03.0-[02-06]--
+-04.0-[02-06]--
+-05.0-[02-06]--
+-06.0-[02-06]--
+-07.0-[02-06]--
+-08.0-[02-06]--
+-09.0-[02-06]--
+-0a.0-[02-06]--
+-0b.0-[02-06]--
+-0c.0-[02-06]--
+-0d.0-[02-06]--
+-0e.0-[02-06]--
+-0f.0-[02-06]--
+-10.0-[02-06]--
+-11.0-[02-06]--
+-12.0-[02-06]--
+-13.0-[02-06]--
+-14.0-[02-06]--
+-15.0-[02-06]--
+-16.0-[02-06]--
+-17.0-[02-06]--
+-18.0-[02-06]--
+-19.0-[02-06]--
+-1a.0-[02-06]--
+-1b.0-[02-06]--
+-1c.0-[02-06]--
+-1d.0-[02-06]--
+-1e.0-[02-06]--
\-1f.0-[02-06]--+-02.0-[03]----00.0
+-03.0-[04]----00.0
+-04.0-[05]----00.0
\-05.0-[06]----00.0
I'll start bisecting to see where things started going wrong. I just
wondered if this rings any bells for anyone.
The startup output also seems to be quite unhappy
Detected board: alliedtelesis,SBx81GC40
Booting into Linux kernel ...
** 143 printk messages dropped **
pci 0000:01:19.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1a.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1b.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1b.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1c.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1d.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1e.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1e.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1f.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1f.0: PME# supported from D0 D3hot D3cold
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:07.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:09.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0b.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0d.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0e.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0f.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:10.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:11.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:12.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:13.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:14.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:15.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:16.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:17.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:18.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:19.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1b.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1d.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1e.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1f.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:02.0: [111d:803c] type 01 class 0x060400
pci 0000:02:02.0: enabling Extended Tags
pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
pci 0000:02:03.0: [111d:803c] type 01 class 0x060400
pci 0000:02:03.0: enabling Extended Tags
pci 0000:02:03.0: PME# supported from D0 D3hot D3cold
pci 0000:02:04.0: [111d:803c] type 01 class 0x060400
pci 0000:02:04.0: enabling Extended Tags
pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
pci 0000:02:05.0: [111d:803c] type 01 class 0x060400
pci 0000:02:05.0: enabling Extended Tags
pci 0000:02:05.0: PME# supported from D0 D3hot D3cold
PCI: bus2: Fast back to back transfers disabled
pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:03:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:03:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:03:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus3: Fast back to back transfers disabled
pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
pci 0000:04:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:04:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus4: Fast back to back transfers disabled
pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
pci 0000:05:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:05:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus5: Fast back to back transfers disabled
pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
pci 0000:06:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:06:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:06:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus6: Fast back to back transfers disabled
pci_bus 0000:06: busn_res: [bus 06-ff] end is updated to 06
pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 06
pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 06
pci 0000:00:01.0: BAR 8: no space for [mem size 0x1c000000]
pci 0000:00:01.0: BAR 8: failed to assign [mem size 0x1c000000]
pci 0000:00:01.0: BAR 6: assigned [mem 0xe0000000-0xe00007ff pref]
pci 0000:01:01.0: BAR 2: no space for [mem size 0x08000000]
pci 0000:01:01.0: BAR 2: failed to assign [mem size 0x08000000]
pci 0000:01:00.0: BAR 8: no space for [mem size 0x10000000]
pci 0000:01:00.0: BAR 8: failed to assign [mem size 0x10000000]
pci 0000:01:00.0: BAR 9: no space for [mem size 0x00400000 64bit pref]
pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit pref]
pci 0000:01:01.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:01:01.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:02.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:03.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:03.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:04.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:04.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:05.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:05.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:02.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:03:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:03:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:03:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: PCI bridge to [bus 03]
pci 0000:04:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:04:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: PCI bridge to [bus 04]
pci 0000:05:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:05:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:05:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:05:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: PCI bridge to [bus 05]
pci 0000:06:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:06:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:06:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:06:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: PCI bridge to [bus 06]
pci 0000:01:00.0: PCI bridge to [bus 02-06]
pci 0000:00:01.0: PCI bridge to [bus 01-06]