Re: [PATCH v2 08/10] clk: renesas: rzg2l-cpg: Add suspend/resume support for power domains

[Ulf Hansson]

On Wed, 17 Apr 2024 at 10:05, claudiu beznea <claudiu.beznea@xxxxxxxxx> wrote:
>
> Hi, Ulf,
>
> On 16.04.2024 15:07, Ulf Hansson wrote:
> > On Thu, 7 Mar 2024 at 15:10, Claudiu <claudiu.beznea@xxxxxxxxx> wrote:
> >>
> >> From: Claudiu Beznea <claudiu.beznea.uj@xxxxxxxxxxxxxx>
> >>
> >> RZ/G3S supports deep sleep states that it can reach with the help of the
> >> TF-A.
> >>
> >> RZ/G3S has a few power domains (e.g. GIC) that need to be always-on while
> >> Linux is running. These domains are initialized (and powered on) when
> >> clock driver is probed.
> >>
> >> As the TF-A takes control at the very last(suspend)/first(resume)
> >> phase of configuring the deep sleep state, it can do it's own settings on
> >> power domains.
> >
> > For my understanding, can you please elaborate on this part a bit.
> > What does the "last suspend/resume phase" mean, more exactly, here?
>
> The RZ/G3S SoC support a power saving mode where most of the SoC parts are
> turned off and the system RAM is switched to retention mode. This is done
> with the help of TF-A. The handshake b/w Linux and TF-A is done though the
> drivers/firmware/psci/psci.c driver.
>
> After Linux finishes the execution of suspend code the control is taken by
> TF-A. TF-A does the final settings on the system (e.g. switching the RAM to
> retention mode) and power off most of the SoC parts.
>
> By the last phase of the suspend I'm referring to the TF-A doing the final
> adjustments for the system to switch to this power saving mode.
>
> When resuming, as the TF-A is the 1st one being executed on the system
> (this is what I called above the 1st phase of the resume), TF-A moves the
> DDR out of retention mode, reconfigure basic IPs (like in boot case as most
> of the SoC parts were powered off) and then give the control to Linux which
> will execute the resume code.

Alright, thanks for clarifying! This makes sense to me now!

>
>
> >
> >>
> >> Thus, to restore the proper Linux state, add rzg2l_cpg_resume() which
> >> powers on the always-on domains and rzg2l_cpg_complete() which activates
> >> the power down mode for the IPs selected through CPG_PWRDN_IP{1, 2}.
> >>
> >> Along with it, added the suspend_check member to the RZ/G2L power domain
> >> data structure whose purpose is to checks if a domain can be powered off
> >> while the system is going to suspend. This is necessary for the serial
> >> console domain which needs to be powered on if no_console_suspend is
> >> available in bootargs.
> >>
> >> Signed-off-by: Claudiu Beznea <claudiu.beznea.uj@xxxxxxxxxxxxxx>
> >> ---
> >>
> >> Changes in v2:
> >> - none; this patch is new
> >>
> >>  drivers/clk/renesas/rzg2l-cpg.c | 66 ++++++++++++++++++++++++++++++---
> >>  drivers/clk/renesas/rzg2l-cpg.h |  1 +
> >>  2 files changed, 62 insertions(+), 5 deletions(-)
> >>
> >> diff --git a/drivers/clk/renesas/rzg2l-cpg.c b/drivers/clk/renesas/rzg2l-cpg.c
> >> index b36700f4a9f5..b18af227177e 100644
> >> --- a/drivers/clk/renesas/rzg2l-cpg.c
> >> +++ b/drivers/clk/renesas/rzg2l-cpg.c
> >> @@ -15,6 +15,7 @@
> >>  #include <linux/clk.h>
> >>  #include <linux/clk-provider.h>
> >>  #include <linux/clk/renesas.h>
> >> +#include <linux/console.h>
> >>  #include <linux/delay.h>
> >>  #include <linux/device.h>
> >>  #include <linux/init.h>
> >> @@ -139,6 +140,7 @@ struct rzg2l_pll5_mux_dsi_div_param {
> >>   * @num_resets: Number of Module Resets in info->resets[]
> >>   * @last_dt_core_clk: ID of the last Core Clock exported to DT
> >>   * @info: Pointer to platform data
> >> + * @domains: generic PM domains
> >>   * @mux_dsi_div_params: pll5 mux and dsi div parameters
> >>   */
> >>  struct rzg2l_cpg_priv {
> >> @@ -155,6 +157,8 @@ struct rzg2l_cpg_priv {
> >>
> >>         const struct rzg2l_cpg_info *info;
> >>
> >> +       struct generic_pm_domain **domains;
> >> +
> >>         struct rzg2l_pll5_mux_dsi_div_param mux_dsi_div_params;
> >>  };
> >>
> >> @@ -1570,12 +1574,14 @@ struct rzg2l_cpg_pm_domains {
> >>   * struct rzg2l_cpg_pd - RZ/G2L power domain data structure
> >>   * @genpd: generic PM domain
> >>   * @priv: pointer to CPG private data structure
> >> + * @suspend_check: check if domain could be powered off in suspend
> >>   * @conf: CPG PM domain configuration info
> >>   * @id: RZ/G2L power domain ID
> >>   */
> >>  struct rzg2l_cpg_pd {
> >>         struct generic_pm_domain genpd;
> >>         struct rzg2l_cpg_priv *priv;
> >> +       int (*suspend_check)(void);
> >>         struct rzg2l_cpg_pm_domain_conf conf;
> >>         u16 id;
> >>  };
> >> @@ -1676,6 +1682,13 @@ static int rzg2l_cpg_power_off(struct generic_pm_domain *domain)
> >>         struct rzg2l_cpg_reg_conf pwrdn = pd->conf.pwrdn;
> >>         struct rzg2l_cpg_priv *priv = pd->priv;
> >>
> >> +       if (pd->suspend_check) {
> >> +               int ret = pd->suspend_check();
> >> +
> >> +               if (ret)
> >> +                       return ret;
> >> +       }
> >> +
> >
> > This should not be needed at all, I think.
> >
> > Instead, genpd should be able to take the correct decision during
> > system-wide suspend and simply avoid calling the ->power_off()
> > callback, when that is needed.
> >
> > If I understand correctly, GENPD_FLAG_ACTIVE_WAKEUP is set for the
> > genpd in question. The only remaining thing would then be to let the
> > console driver, during system suspend, check whether
> > "console_suspend_enabled" is set and then call device_set_awake_path()
> > for its device. In this way, genpd should then keep the corresponding
> > PM domain powered-on.
>
> You're right! I've checked it and all good w/o ->suspend_check() if
> device_set_wakeup_path() is called for the console driver.
>
> I'll send an update for it.

Great! Please keep me posted on the entire series for next version. I
will try to continue to help review this.

>
> >
> >>         /* Set MSTOP. */
> >>         if (mstop.mask)
> >>                 writel(mstop.mask | (mstop.mask << 16), priv->base + mstop.off);
> >> @@ -1687,8 +1700,14 @@ static int rzg2l_cpg_power_off(struct generic_pm_domain *domain)
> >>         return 0;
> >>  }
> >>
> >> -static int __init rzg2l_cpg_pd_setup(struct rzg2l_cpg_pd *pd, bool always_on)
> >> +static int rzg2l_pd_suspend_check_console(void)
> >>  {
> >> +       return console_suspend_enabled ? 0 : -EBUSY;
> >> +}
> >> +
> >> +static int __init rzg2l_cpg_pd_setup(struct rzg2l_cpg_pd *pd, u32 flags)
> >> +{
> >> +       bool always_on = !!(flags & RZG2L_PD_F_ALWAYS_ON);
> >>         struct dev_power_governor *governor;
> >>
> >>         pd->genpd.flags |= GENPD_FLAG_PM_CLK | GENPD_FLAG_ACTIVE_WAKEUP;
> >> @@ -1700,6 +1719,8 @@ static int __init rzg2l_cpg_pd_setup(struct rzg2l_cpg_pd *pd, bool always_on)
> >>         } else {
> >>                 pd->genpd.power_on = rzg2l_cpg_power_on;
> >>                 pd->genpd.power_off = rzg2l_cpg_power_off;
> >> +               if (flags & RZG2L_PD_F_CONSOLE)
> >> +                       pd->suspend_check = rzg2l_pd_suspend_check_console;
> >>                 governor = &simple_qos_governor;
> >>         }
> >>
> >> @@ -1719,7 +1740,7 @@ static int __init rzg2l_cpg_add_clk_domain(struct rzg2l_cpg_priv *priv)
> >>
> >>         pd->genpd.name = np->name;
> >>         pd->priv = priv;
> >> -       ret = rzg2l_cpg_pd_setup(pd, true);
> >> +       ret = rzg2l_cpg_pd_setup(pd, RZG2L_PD_F_ALWAYS_ON);
> >>         if (ret)
> >>                 return ret;
> >>
> >> @@ -1778,13 +1799,13 @@ static int __init rzg2l_cpg_add_pm_domains(struct rzg2l_cpg_priv *priv)
> >>         domains->onecell_data.domains = domains->domains;
> >>         domains->onecell_data.num_domains = info->num_pm_domains;
> >>         domains->onecell_data.xlate = rzg2l_cpg_pm_domain_xlate;
> >> +       priv->domains = domains->domains;
> >>
> >>         ret = devm_add_action_or_reset(dev, rzg2l_cpg_genpd_remove, &domains->onecell_data);
> >>         if (ret)
> >>                 return ret;
> >>
> >>         for (unsigned int i = 0; i < info->num_pm_domains; i++) {
> >> -               bool always_on = !!(info->pm_domains[i].flags & RZG2L_PD_F_ALWAYS_ON);
> >>                 struct rzg2l_cpg_pd *pd;
> >>
> >>                 pd = devm_kzalloc(dev, sizeof(*pd), GFP_KERNEL);
> >> @@ -1796,11 +1817,11 @@ static int __init rzg2l_cpg_add_pm_domains(struct rzg2l_cpg_priv *priv)
> >>                 pd->id = info->pm_domains[i].id;
> >>                 pd->priv = priv;
> >>
> >> -               ret = rzg2l_cpg_pd_setup(pd, always_on);
> >> +               ret = rzg2l_cpg_pd_setup(pd, info->pm_domains[i].flags);
> >>                 if (ret)
> >>                         return ret;
> >>
> >> -               if (always_on) {
> >> +               if (info->pm_domains[i].flags & RZG2L_PD_F_ALWAYS_ON) {
> >>                         ret = rzg2l_cpg_power_on(&pd->genpd);
> >>                         if (ret)
> >>                                 return ret;
> >> @@ -1890,9 +1911,43 @@ static int __init rzg2l_cpg_probe(struct platform_device *pdev)
> >>         if (error)
> >>                 return error;
> >>
> >> +       dev_set_drvdata(dev, priv);
> >> +
> >>         return 0;
> >>  }
> >>
> >> +static int rzg2l_cpg_resume(struct device *dev)
> >> +{
> >> +       struct rzg2l_cpg_priv *priv = dev_get_drvdata(dev);
> >> +       const struct rzg2l_cpg_info *info = priv->info;
> >> +
> >> +       /* Power on always ON domains. */
> >> +       for (unsigned int i = 0; i < info->num_pm_domains; i++) {
> >> +               if (info->pm_domains[i].flags & RZG2L_PD_F_ALWAYS_ON) {
> >> +                       int ret = rzg2l_cpg_power_on(priv->domains[i]);
> >> +
> >> +                       if (ret)
> >> +                               return ret;
> >> +               }
> >> +       }
> >
> > I don't quite understand why this is needed? Is always-on PM domains
> > being powered-off during system wide suspend, so you need to power
> > them on again?
>
> Yes, as power to most of the system parts is cut off during sytem suspend
> (and DDR is kept in retention mode) and the resume is almost like a cold
> boot where the TF-A does basic re-initialization and then pass execution to
>  Linux resume code.

Hmm. If these are really always-on PM domains, why isn't the FW
powering them on again then before returning to Linux after a system
resume?

In a way it sounds to me that they aren't really always-on PM domains,
as Linux seems to be capable of turning them on/off too, right?

That said, perhaps using GENPD_FLAG_RPM_ALWAYS_ON instead of
GENPD_FLAG_ALWAYS_ON for some PM domains can be another way forward?
In this way, the ->power_on|off() callbacks can be used to turn on/off
the PM domains, but only during system suspend/resume. Would that
work?

[...]

Kind regards
Uffe