Re: [PATCH v2 0/9] Add per-core RAPL energy counter support for AMD CPUs

From: K Prateek Nayak
Date: Fri Jun 21 2024 - 04:24:55 EST


Hello Dhananjay,

On 6/20/2024 6:26 PM, Dhananjay Ugwekar wrote:
Currently the energy-cores event in the power PMU aggregates energy
consumption data at a package level. On the other hand the core energy
RAPL counter in AMD CPUs has a core scope (which means the energy
consumption is recorded separately for each core). Earlier efforts to add
the core event in the power PMU had failed [1], due to the difference in
the scope of these two events. Hence, there is a need for a new core scope
PMU.

This patchset adds a new "power_per_core" PMU alongside the existing
"power" PMU, which will be responsible for collecting the new
"energy-per-core" event.

Tested the package level and core level PMU counters with workloads
pinned to different CPUs.

Results with workload pinned to CPU 1 in Core 1 on an AMD Zen4 Genoa
machine:

$ perf stat -a --per-core -e power_per_core/energy-per-core/ sleep 1

When testing this on a 2P 3rd Generation EPYC System (2 x 64/128T), I
ran into an issue where it seems like the energy reporting for the
system is coming from the second socket. Following are the CPUs on each
socket of the system:

Node 0: 0-63, 128-191
Node 1: 64-127, 192-255

Following are the experiments I ran:

$ # Run a busy loop on each thread of the first socket
$ for i in `seq 0 63` `seq 128 191`; do taskset -c $i ~/scripts/loop & done
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5

S0-D0-C0 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C1 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C0 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C1 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/

From the energy data, it looks as if the system is entirely idle.

If I repeat the same, pinning the running busy loop on the threads of
second socket, I see the following:

$ # Run a busy loop on each thread of the second socket
$ for i in `seq 64 127` `seq 192 255`; do taskset -c $i ~/scripts/loop & done
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5

S0-D0-C0 1 11.79 Joules power_per_core/energy-per-core/
S0-D0-C1 1 11.80 Joules power_per_core/energy-per-core/
S0-D0-C2 1 11.90 Joules power_per_core/energy-per-core/
S0-D0-C3 1 11.88 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 11.76 Joules power_per_core/energy-per-core/
S1-D1-C0 1 11.81 Joules power_per_core/energy-per-core/
S1-D1-C1 1 11.80 Joules power_per_core/energy-per-core/
S1-D1-C2 1 11.90 Joules power_per_core/energy-per-core/
S1-D1-C3 1 11.88 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 11.76 Joules power_per_core/energy-per-core/

The whole system seems to be busy this time around. I've verified that
only half the system is busy using htop in either case.

Running some more experiments, I see the following:

$ taskset -c 1 ~/scripts/loop& # First thread from Core 1, Socket
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5

S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C1 1 0.21 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.20 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
(Seemingly idle system)


$ taskset -c 65 ~/scripts/loop&
$ sudo perf stat -a --per-core -e power_per_core/energy-per-core/ -- sleep 5

S0-D0-C0 1 0.01 Joules power_per_core/energy-per-core/
S0-D0-C1 1 16.73 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.00 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S0-D0-C63 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C0 1 0.01 Joules power_per_core/energy-per-core/
S1-D1-C1 1 16.73 Joules power_per_core/energy-per-core/
S1-D1-C2 1 0.00 Joules power_per_core/energy-per-core/
S1-D1-C3 1 0.00 Joules power_per_core/energy-per-core/
...
S1-D1-C63 1 0.00 Joules power_per_core/energy-per-core/

(Core 1 from both sockets look busy reporting identical energy
values)

Hope it helps narrow down the issue.


Performance counter stats for 'system wide':

S0-D0-C0 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C1 1 5.72 Joules power_per_core/energy-per-core/
S0-D0-C2 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C3 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C4 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C5 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C6 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C7 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C8 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C9 1 0.02 Joules power_per_core/energy-per-core/
S0-D0-C10 1 0.02 Joules power_per_core/energy-per-core/

[1]: https://lore.kernel.org/lkml/3e766f0e-37d4-0f82-3868-31b14228868d@xxxxxxxxxxxxxxx/

This patchset applies cleanly on top of v6.10-rc4 as well as latest
tip/master.

P.S. I tested these changes on top of tip:perf/core


[..snip..]


--
Thanks and Regards,
Prateek