Re: [PATCH v6 0/5] PCI: qcom: Add system suspend & resume support

[Bjorn Helgaas]

On Mon, Sep 12, 2022 at 09:40:30PM +0530, Krishna Chaitanya Chundru wrote:
> On 9/10/2022 1:21 AM, Bjorn Helgaas wrote:
> > On Fri, Sep 09, 2022 at 02:14:39PM +0530, Krishna chaitanya chundru wrote:
> > > Add suspend and resume syscore ops.
> > > 
> > > When system suspends, and if the link is in L1ss, disable the clocks
> > > and power down the phy so that system enters into low power state by
> > > parking link in L1ss to save the maximum power. And when the system
> > > resumes, enable the clocks back and power on phy if they are disabled
> > > in the suspend path.
> > > 
> > > we are doing this only when link is in l1ss but not in L2/L3 as
> > > nowhere we are forcing link to L2/L3 by sending PME turn off.
> > > 
> > > is_suspended flag indicates if the clocks are disabled in the suspend
> > > path or not.
> > > 
> > > There is access to Ep PCIe space to mask MSI/MSIX after pm suspend ops
> > > (getting hit by affinity changes while making CPUs offline during suspend,
> > > this will happen after devices are suspended (all phases of suspend ops)).
> > > When registered with pm ops there is a crash due to un-clocked access,
> > > as in the pm suspend op clocks are disabled. So, registering with syscore
> > > ops which will called after making CPUs offline.
> > > 
> > > Make GDSC always on to ensure controller and its dependent clocks
> > > won't go down during system suspend.
> > > 
> > > Krishna chaitanya chundru (5):
> > >    PCI: qcom: Add system suspend and resume support
> > >    PCI: qcom: Add retry logic for link to be stable in L1ss
> > >    phy: core: Add support for phy power down & power up
> > >    phy: qcom: Add power down/up callbacks to pcie phy
> > >    clk: qcom: Alwaya on pcie gdsc
> >
> > This seems fairly ugly because it doesn't fit nicely into the PM
> > framework.  Why is this a qcom-specific thing?  What about other
> > DWC-based controllers?
>
> We wanted to allow system S3 state by turning off all PCIe clocks
> but at the same time retaining NVMe device in D0 state and PCIe link
> in l1ss state.
>
> Here nothing really specific to DWC as PCIe controller remains intact.
> 
> And the Qcom PHY allows this scheme  (that is to retain the link
> state in l1ss even though all pcie clocks are turned off).

Is there somewhere in the PCIe spec I can read about how a link with
clocks turned off can remain in L1.1 or L1.2?

> Since clocks are completely managed by qcom platform driver, we are
> trying to manage them during S3/S0 transitions with PM callbacks.

I'm looking at this text in PCIe r6.0, sec 5.4.1:

  Components in the D0 state (i.e., fully active state) normally keep
  their Upstream Link in the active L0 state, as defined in § Section
  5.3.2 . ASPM defines a protocol for components in the D0 state to
  reduce Link power by placing their Links into a low power state and
  instructing the other end of the Link to do likewise. This
  capability allows hardware-autonomous, dynamic Link power reduction
  beyond what is achievable by software-only controlled (i.e., PCI-PM
  software driven) power management.

How does this qcom software management of clocks fit into this scheme?
It seems to me that if you need software to turn clocks off and on,
that is no longer ASPM.

Bjorn