This isn't in the fastpath, it's in the rebalancing logic.
the reality is much more complex unfortunately.
C and P states hang together tightly, and even C state on one core
impacts other cores' performance, just like P state selection on one
core impacts other cores.
(at least for x86, we should really stop talking as if the OS picks
the "frequency", that's just not the case anymore)
I agree, the reality is very complex. But we should go back and analyse
what problem we are trying to solve, what each framework is trying to
address.
When viewed separately from the scheduler, cpufreq and cpuidle governors
do the right thing. But they both base their action on the CPU load
(balance) decided by the scheduler and it's the latter that we are
trying to adjust (and we are still debating what the right approach is).
Since such information seems too complex to be moved into the scheduler,
why don't we get cpufreq in charge of restricting the load balancing to
certain CPUs? It already tracks the load/idle time to (gradually) change
the P state. Depending on the governor/policy, it could decide that (for
Cpuidle I think for now can stay the same, gradually entering deeper
sleep states. It could be later unified with cpufreq if there are any
benefits. In deciding the load balancing restrictions, maybe cpufreq
should be aware of C-state latencies.