Re: Fwd: ARM64: CPUIdle driver is not select any Idle state other then WFI

[Dhruva Gole]

Hi Vivek,

On Oct 14, 2024 at 16:06:34 +0530, Vivek yadav wrote:
> ---------- Forwarded message ---------
> From: Vivek yadav <linux.ninja23@xxxxxxxxx>
> Date: Fri, Oct 11, 2024 at 3:14 PM
> Subject: ARM64: CPUIdle driver is not select any Idle state other then WFI
> To: <linux-pm@xxxxxxxxxxxxxxx>

+ Kevin, Vignesh and few colleagues at TI who have been working on this as
well.

> 
> 
> Hi @all,
> 
> I am working on one custom SoC. Where I add one CPUIdle state for
> ``arm,cortex-a55`` processor.

Any further luck on this?

I have also been working on something similar[1] but on an A53 core on
TI-K3 AM62x processor.

> 
> idle-states {
>       entry-method = "psci";
>        cpu_ret_l: cpu-retention-l {
>          compatible = "arm,idle-state";
>          arm,psci-suspend-param = <0x00010001>;
>          local-timer-stop;
>          entry-latency-us = <55>;
>           exit-latency-us = <140>;
>           min-residency-us = <780>;
>     };
> };
> 
> I am using ``Menu governor`` with the ``psci_idle driver`` in its original form.
> After booting Linux I find out that the CPUIdle core is never going
> inside the ``cpu-retention`` state.
> To check time spent by CPU in any state. I am using the below command.
> 
> ``cat /sys/devices/system/cpu/cpu*/cpuidle/state*/time``

What I was seeing is in a multi core system (2 or more) all cores don't
enter the idle-state simultaneously. There's something keeping atleast 1
core always busy. However I could definitely see entry into TF-A from 1
core at a time.
I then switched to a single core system to see if we were atall able to
enter TF-A when only 1 core was available for linux, it turned out that
with the "local-timer-stop" property that we have, this is never
possible.

See this chunk in the kernel cpuidle driver:
	if (broadcast && tick_broadcast_enter()) {

When I dug deeper into tick_broadcast_enter it always returns something
non zero and hence in my case it was entering the if block and tried to
find a deepest state. Then the deepest state would always return WFI and
not the idle-state I had added.

What we found out was on our kernel we end up using 

kernel/time/tick-broadcast-hrtimer.c 

This always seems to be keeping atleast 1 CPU busy and prevents idle.
If we remove the local-timer-stop it was helping us, but we still need
to dig into the full impact of what that entails and I am still
interested in finding out how so many other users of similar idle-state
implementation are able to do so without trouble.

Arm64 recommends to use arch_timer instead of external timers. Once we
enter el3, timer interrupts to el1 is blocked and hence it's equivalent
to local-timer-stop, so it does make sense to keep this property, but
then how are others able to enter idle-states for all plugged CPUs at
the same time?

> 
> OUTPUT:
> 0 ===>CPU0 state0 (WFI)
> 0 ===>CPU0 state1 (cpu-retention)
> 
> increasing some time value ===>CPU1 state0 (WFI)
> 0 ===>CPU1 state1 (cpu-retention)
> 
> increasing some time value
> 0
> 
> increasing some time value
> 0
> 
> What am I doing wrong? Why does ``cpu-retention`` state time not increase?
> Any pointer will be helpful.

I had asked a similar qn. on IRC [2], but didn't get much response. I
am still interested in finding out what the right thing to do here is.

[1] https://github.com/DhruvaG2000/v-linux/commit/0fd088d624276a2e72b8dc6660d261ab6d194f4b#diff-34369928f669d14776f8f5bdbe3fc3d75306196a2ac28b1a4d7e17402b9c3995R160
[2] https://libera.irclog.whitequark.org/armlinux/2024-08-23 

-- 
Best regards,
Dhruva Gole
Texas Instruments Incorporated