Re: Kirkwood PCI Express and bridges

[Chris Packham]

On 24/06/19 4:08 PM, Chris Packham wrote:
> Hi Thomas,
> 
> On 21/06/19 6:17 PM, Thomas Petazzoni wrote:
>> Hello Chris,
>>
>> On Fri, 21 Jun 2019 04:03:27 +0000
>> Chris Packham <Chris.Packham@xxxxxxxxxxxxxxxxxxx> wrote:
>>
>>> I'm in the process of updating the kernel version used on our products
>>> from 4.4 -> 5.1.
>>>
>>> We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
>>> Switch ASIC. The Switch ASIC presents as multiple PCI devices.
>>>
>>> The hardware setup looks like this
>>>                                         __________
>>> [ Kirkwood ] --- [ IDT 5T5 ] ---+---  |          |
>>>                                  +---  |  Switch  |
>>>                                  +---  |          |
>>>                                  +---  |__________|
>>>
>>> On the 4.4 based kernel things are fine
>>>
>>> [root@awplus flash]# lspci -t
>>> -[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
>>>                                               +-03.0-[04]----00.0
>>>                                               +-04.0-[05]----00.0
>>>                                               \-05.0-[06]----00.0
>>>
>>> But on the 5.1 based kernel things get a little weird
>>>
>>> [root@awplus flash]# lspci -t
>>> -[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
>>>                               +-01.0
>>>                               +-02.0-[02-06]--
>>>                               +-03.0-[02-06]--
>>>                               +-04.0-[02-06]--
>>>                               +-05.0-[02-06]--
>>>                               +-06.0-[02-06]--
>>>                               +-07.0-[02-06]--
>>>                               +-08.0-[02-06]--
>>>                               +-09.0-[02-06]--
>>>                               +-0a.0-[02-06]--
>>>                               +-0b.0-[02-06]--
>>>                               +-0c.0-[02-06]--
>>>                               +-0d.0-[02-06]--
>>>                               +-0e.0-[02-06]--
>>>                               +-0f.0-[02-06]--
>>>                               +-10.0-[02-06]--
>>>                               +-11.0-[02-06]--
>>>                               +-12.0-[02-06]--
>>>                               +-13.0-[02-06]--
>>>                               +-14.0-[02-06]--
>>>                               +-15.0-[02-06]--
>>>                               +-16.0-[02-06]--
>>>                               +-17.0-[02-06]--
>>>                               +-18.0-[02-06]--
>>>                               +-19.0-[02-06]--
>>>                               +-1a.0-[02-06]--
>>>                               +-1b.0-[02-06]--
>>>                               +-1c.0-[02-06]--
>>>                               +-1d.0-[02-06]--
>>>                               +-1e.0-[02-06]--
>>>                               \-1f.0-[02-06]--+-02.0-[03]----00.0
>>>                                               +-03.0-[04]----00.0
>>>                                               +-04.0-[05]----00.0
>>>                                               \-05.0-[06]----00.0
>>>
>>>
>>> I'll start bisecting to see where things started going wrong. I just
>>> wondered if this rings any bells for anyone.
>>
>> I am almost sure that the culprit is
>> 1f08673eef1236f7d02d93fcf596bb8531ef0d12 ("PCI: mvebu: Convert to PCI
>> emulated bridge config space").
> 
> The problem seems to pre-date this commit. I've gone back as far as 4.18
> and the problem still exists (in fact there are more duplicate devices).
> I'll keep going back (unfortunately due to out platform being out of
> tree it's not a simple bisect).
> 
>> I still think it makes sense to share the bridge emulation code between
>> the mvebu and aardvark drivers, but this sharing has required making
>> the code very different, with lots of subtle differences in behavior in
>> how registers are emulated.
> 
> Agreed. Bugs love to hide in duplicated code.
> 
> I will admit to being ignorant about the need for an emulated bridge. I
> know it has something to do with the type of transaction used for the
> downstream devices. I also know that these systems won't work without an
> emulated bridge.
> 
>> Unfortunately, I don't have access to one of these complicated PCI
>> setup with a HW switch on the way, so I couldn't test this kind of
>> setups.
>>
>> Do you mind helping with figuring out what the issues are ? That would
>> be really nice.
> 
> No problem. As I said I'll keep going to find a point where behaviour
> turns bad for me. I suspect we might find other problems along the way.
> 

Some progress. Our defconfig had CONFIG_CMDLINE="pci=pcie_scan_all" in 
it. This dated back to before we were using a devicetree with our 
kirkwood platforms. At some point this started having an effect on the 
emulated bridge.