Re: [GIT PULL] PCI fixes for v4.10

[Rafael J. Wysocki]

On Sunday, February 12, 2017 08:05:02 PM Lukas Wunner wrote:
> On Fri, Feb 10, 2017 at 06:39:16PM -0800, Yinghai Lu wrote:
> > On Thu, Feb 9, 2017 at 12:11 PM, Bjorn Helgaas <helgaas@xxxxxxxxxx> wrote:
> > > On Thu, Feb 09, 2017 at 09:09:50AM -0600, Bjorn Helgaas wrote:
> > > > On Thu, Feb 09, 2017 at 05:06:48AM +0100, Lukas Wunner wrote:
> > > > > https://patchwork.kernel.org/patch/9557113/
> > > > > https://patchwork.kernel.org/patch/9562007/
> > >
> > > I apologize: I had quirks on the brain, but neither of the patches
> > > above is device-specific.  So neither is claiming broken hardware.
> > >
> > > However, 9557113 claims we get unwanted PME interrupts if the slot is
> > > occupied when we suspend to D3hot.  This is what I want to explore
> > > further, because that hardware behavior doesn't really make sense to
> > > me.
> > >
> > > 9562007 apparently fixes something, but at this point it's a debugging
> > > patch (no changelog or signed-off-by) so not a candidate for tossing
> > > into v4.10 at this late date.
> > 
> > Agreed. It should need more test coverage.  Found more problems.
> > 
> > Actually we don't need 9557113 as even with that, we still saw link up
> > when power off slots with some cards.
> > 
> > please check updated version of 9562007, that fix power on/off link up
> > problem.
> 
> Thank you for debugging this further.  The patch I've submitted today
> reinstates runtime PM for hotplug ports but constrains it to those on
> a Thunderbolt daisy chain.  The patch allows enabling the feature on
> other hardware by booting with pcie_port_pm=force.
> 
> A few things to keep in mind:
> 
> * On Thunderbolt hotplug ports, interrupts are sent even if the port
>   is in D3hot, which as Bjorn has pointed out contradicts the PCI PM
>   spec r1.2, table 5-4.  This may be caused by liberal interpretation
>   of the spec by Intel when designing the Thunderbolt controllers,
>   or perhaps Thunderbolt controllers simply do not possess a "real",
>   fully-fledged PCIe switch.  I let the hotplug ports go to D3hot,
>   expecting them to continue delivering interrupts but YMMV.
> 
> * You've reported that the hotplug port must be in D0 to enable and
>   disable power on the slot.  I think this is not required by the spec.
>   Thunderbolt hotplug ports do not support power control.  My suspicion
>   is that the ports on your machine must remain in D0 as long as the
>   slot is occupied, i.e. they must not runtime suspend to D3hot.  Can
>   this happen?  Yes.  I release the runtime PM ref once a slot has been
>   enabled or disabled.  The device remains runtime active as long as it
>   has active children.  If all children runtime suspend, the port will
>   go to D3hot, which might cause trouble if this implies that slot power
>   is turned off.  To test this you need a card whose Linux driver supports
>   runtime PM (e.g. Nvidia GPU, boot with nouveau.runpm=1).
> 
> * If the hotplug slot has runtime suspended to D3hot and there are ports
>   above it that also runtime suspend to D3hot, its config space is no
>   longer accessible and in-band interrupts won't come through.  A side-band
>   signaling method such as PME WAKE# is required to deliver interrupts from
>   this state.

It actually can use in-band PME messages too, at least if my interpretation of
this part of the spec is correct (or reflects the interpretation of the people
who design the chips in question to be more precise).

Thanks,
Rafael