Re: [PATCH v7 1/5] PCI: qcom: Add system suspend and resume support

From: Krishna Chaitanya Chundru
Date: Mon Oct 03 2022 - 08:11:10 EST


On 9/30/2022 12:23 AM, Bjorn Helgaas wrote:
On Mon, Sep 26, 2022 at 09:00:11PM +0530, Krishna Chaitanya Chundru wrote:
On 9/23/2022 7:56 PM, Bjorn Helgaas wrote:
On Fri, Sep 23, 2022 at 07:29:31AM +0530, Krishna Chaitanya Chundru wrote:
On 9/23/2022 12:12 AM, Bjorn Helgaas wrote:
On Thu, Sep 22, 2022 at 09:09:28PM +0530, Krishna Chaitanya Chundru wrote:
On 9/21/2022 10:26 PM, Bjorn Helgaas wrote:
On Wed, Sep 21, 2022 at 03:23:35PM +0530, Krishna Chaitanya Chundru wrote:
On 9/20/2022 11:46 PM, Bjorn Helgaas wrote:
On Tue, Sep 20, 2022 at 03:52:23PM +0530, Krishna chaitanya chundru wrote:
In qcom platform PCIe resources( clocks, phy etc..) can
released when the link is in L1ss to reduce the power
consumption. So if the link is in L1ss, release the PCIe
resources. And when the system resumes, enable the PCIe
resources if they released in the suspend path.
What's the connection with L1.x? Links enter L1.x based on
activity and timing. That doesn't seem like a reliable
indicator to turn PHYs off and disable clocks.
This is a Qcom PHY-specific feature (retaining the link state in
L1.x with clocks turned off). It is possible only with the link
being in l1.x. PHY can't retain the link state in L0 with the
clocks turned off and we need to re-train the link if it's in L2
or L3. So we can support this feature only with L1.x. That is
the reason we are taking l1.x as the trigger to turn off clocks
(in only suspend path).
This doesn't address my question. L1.x is an ASPM feature, which
means hardware may enter or leave L1.x autonomously at any time
without software intervention. Therefore, I don't think reading the
current state is a reliable way to decide anything.
After the link enters the L1.x it will come out only if there is
some activity on the link. AS system is suspended and NVMe driver
is also suspended( queues will  freeze in suspend) who else can
initiate any data.
I don't think we can assume that nothing will happen to cause exit
from L1.x. For instance, PCIe Messages for INTx signaling, LTR, OBFF,
PTM, etc., may be sent even though we think the device is idle and
there should be no link activity.
I don't think after the link enters into L1.x there will some
activity on the link as you mentioned, except for PCIe messages like
INTx/MSI/MSIX. These messages also will not come because the client
drivers like NVMe will keep their device in the lowest power mode.

The link will come out of L1.x only when there is config or memory
access or some messages to trigger the interrupts from the devices.
We are already making sure this access will not be there in S3. If
the link is in L0 or L0s what you said is expected but not in L1.x
Forgive me for being skeptical, but we just spent a few months
untangling the fact that some switches send PTM request messages even
when they're in a non-D0 state. We expected that devices in D3hot
would not send such messages because "why would they?" But it turns
out the spec allows that, and they actually *do*.

I don't think it's robust interoperable design for a PCI controller
driver like qcom to assume anything about PCI devices unless it's
required by the spec.
From pci spec 4, in sec 5.5
"Ports that support L1 PM Substates must not require a reference clock while
in L1 PM Substates
other than L1.0".
If there is no reference clk we can say there is no activity on the link.
If anything needs to be sent (such as LTR, or some messages ), the link
needs to be back in L0 before it
sends the packet to the link partner.

To exit from L1.x clkreq pin should be asserted.

In suspend after turning off clocks and phy we can enable to trigger an
interrupt whenever the clk req pin asserts.
In that interrupt handler, we can enable the pcie resources back.
From the point of view of the endpoint driver, ASPM should be
invisible -- no software intervention required. I think you're
suggesting that the PCIe controller driver could help exit L1.x by
handling a clk req interrupt and enabling clock and PHY then.

But doesn't L1.x exit also have to happen within the time the endpoint
can tolerate? E.g., I think L1.2 exit has to happen within the LTR
time advertised by the endpoint (PCIe r6.0, sec 5.5.5). How can we
guarantee that if software is involved?
It is true that it is difficult to guarantee those delays. On our internal
boards, we are able to achieve this but that is not with linux kernel.

With NVMe attach we have connected the protocol analyzer and tried to see if
there are any transactions over the link. We found there are no transactions
on the link once the link enters L1.x till we resume the system. As the NVMe
is a passive system it is not initiating any transactions.

This whole requirement came from the NVMe driver, it requires keeping the link
active state when the system is suspended.

There are only two things we can in do in PCIe suspend as we have to turn off
PCIe clocks to allow the system to the lowest possible power state.
1) Keep the device in D3 cold and turn off all the clocks and phy etc.( It is not
an ideal one as this decreases the NVMe lifetime because link-down and link-up
is treated as a power cycle by a few NVMe devices).
2) This is the one we are proposing where we turn off the clocks, phy once the
link enters L1ss.

Can you please suggest us any other possible solutions to meet NVMe requirement
(That is to keep the link active during suspend) and the Qcom platform
requirement (that is to turn off all the clocks to allow a lower possible
power state)? Qcom PCIe controller is compatible with v3.1 specification only.


Thanks & Regards,
Krishna Chaitanya.