Re: [PATCH v6 0/5] PCI: qcom: Add system suspend & resume support

[Manivannan Sadhasivam]

On Mon, Sep 12, 2022 at 12:08:06PM -0500, Bjorn Helgaas wrote:
> On Mon, Sep 12, 2022 at 09:40:30PM +0530, Krishna Chaitanya Chundru wrote:
> > On 9/10/2022 1:21 AM, Bjorn Helgaas wrote:
> > > On Fri, Sep 09, 2022 at 02:14:39PM +0530, Krishna chaitanya chundru wrote:
> > > > Add suspend and resume syscore ops.
> > > > 
> > > > When system suspends, and if the link is in L1ss, disable the clocks
> > > > and power down the phy so that system enters into low power state by
> > > > parking link in L1ss to save the maximum power. And when the system
> > > > resumes, enable the clocks back and power on phy if they are disabled
> > > > in the suspend path.
> > > > 
> > > > we are doing this only when link is in l1ss but not in L2/L3 as
> > > > nowhere we are forcing link to L2/L3 by sending PME turn off.
> > > > 
> > > > is_suspended flag indicates if the clocks are disabled in the suspend
> > > > path or not.
> > > > 
> > > > There is access to Ep PCIe space to mask MSI/MSIX after pm suspend ops
> > > > (getting hit by affinity changes while making CPUs offline during suspend,
> > > > this will happen after devices are suspended (all phases of suspend ops)).
> > > > When registered with pm ops there is a crash due to un-clocked access,
> > > > as in the pm suspend op clocks are disabled. So, registering with syscore
> > > > ops which will called after making CPUs offline.
> > > > 
> > > > Make GDSC always on to ensure controller and its dependent clocks
> > > > won't go down during system suspend.
> > > > 
> > > > Krishna chaitanya chundru (5):
> > > >    PCI: qcom: Add system suspend and resume support
> > > >    PCI: qcom: Add retry logic for link to be stable in L1ss
> > > >    phy: core: Add support for phy power down & power up
> > > >    phy: qcom: Add power down/up callbacks to pcie phy
> > > >    clk: qcom: Alwaya on pcie gdsc
> > >
> > > This seems fairly ugly because it doesn't fit nicely into the PM
> > > framework.  Why is this a qcom-specific thing?  What about other
> > > DWC-based controllers?
> >
> > We wanted to allow system S3 state by turning off all PCIe clocks
> > but at the same time retaining NVMe device in D0 state and PCIe link
> > in l1ss state.
> >
> > Here nothing really specific to DWC as PCIe controller remains intact.
> > 
> > And the Qcom PHY allows this scheme  (that is to retain the link
> > state in l1ss even though all pcie clocks are turned off).
> 
> Is there somewhere in the PCIe spec I can read about how a link with
> clocks turned off can remain in L1.1 or L1.2?
> 

This part is Qcom specific. On Qcom platforms there are two power domains used,
CX and MX. CX domain is sourcing the PCIe controller and MX is sourcing PCIe
PHY.

Both PCIe and PHY drivers don't control MX domain and only control CX.
So even though this patch is turning off all of the PCIe clocks, that
only helps in powering down the CX domain for achieving the low power
usecase while still keeping MX domain powered on.

So at the end of suspend, the PCIe controller would've turned off but
the link stays in L1SS state due to it being backed by MX.

> > Since clocks are completely managed by qcom platform driver, we are
> > trying to manage them during S3/S0 transitions with PM callbacks.
> 
> I'm looking at this text in PCIe r6.0, sec 5.4.1:
> 
>   Components in the D0 state (i.e., fully active state) normally keep
>   their Upstream Link in the active L0 state, as defined in § Section
>   5.3.2 . ASPM defines a protocol for components in the D0 state to
>   reduce Link power by placing their Links into a low power state and
>   instructing the other end of the Link to do likewise. This
>   capability allows hardware-autonomous, dynamic Link power reduction
>   beyond what is achievable by software-only controlled (i.e., PCI-PM
>   software driven) power management.
> 
> How does this qcom software management of clocks fit into this scheme?
> It seems to me that if you need software to turn clocks off and on,
> that is no longer ASPM.
> 

The PCIe link automatically transitions to L1SS if there is no activity
on the link but still the clocks sourced to the PCIe controller needs to be
turned off. PCIe spec only covers the link specifics and the controller is
platform dependent.

Thanks,
Mani

> Bjorn