Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

[Chun-Tse Shao]

On Tue, Oct 13, 2026 at 2:08 PM Besar Wicaksono <bwicaksono@xxxxxxxxxx> wrote:
>
> Hi Chun-Tse,
>
> I tested this on 2-sockets Tegra410 and found some issues.
> 1. With --per-socket or -A, every lpm_pcie_* metric is only reported for
>    socket 0, so socket 1's PCIe traffic never shows up. Example output:
>
> perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1
>
>  Performance counter stats for 'system wide':
>
> CPU0               15,204      nvidia_pcie_pmu_0_rc_1/wr_bytes/ #      0.0 MB/s  lpm_pcie_wr_bw_1
> CPU0                                                                    #     33.8 MB/s  lpm_pcie_bw_1
> CPU0           33,910,952      nvidia_pcie_pmu_0_rc_1/rd_bytes/ #     33.8 MB/s  lpm_pcie_rd_bw_1
> CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
> CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
> CPU0        1,002,286,615      tool/duration_time/
> CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_wr_bw_1
> CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
> CPU0        1,002,275,059      tool/duration_time/
> CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #      0.0 MB/s  lpm_pcie_rem_rd_bw_1
> CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/
> <snip>
>
> Note CPU88 (socket 1_ has no lpm_pcie_* values.
>
> 2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of
>    socket 1. These are separate root complexes with different devices,
>    so the "Root Complex i" value can combine traffic from two unrelated
>    links.
>    Could the metrics follow the hardware hierarchy instead, starting at
>    the socket and drilling down to the RC, with the same destination
>    split at each level?
>     - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
>         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
>         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
>     - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
>         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
>         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
>       grouped as lpm_pcie_s<s>_rc<i>_bw.

Thanks for testing and for the suggestion. I think both problems come
from how perf expects a PMU to expose its cpumask. Ideally perf wants
a single PMU for both sockets, with one CPU per socket in its cpumask,
e.g.:

```
/sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
```

We can work around the one-PMU-per-socket layout in the metrics, but
would a single PMU per RC be a viable plan for you to implement?

Thanks,
CT

>
> Thanks,
> Besar
>
> > -----Original Message-----
> > From: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > Sent: Monday, September 28, 2026 5:51 PM
> > To: acme@xxxxxxxxxx; namhyung@xxxxxxxxxx; irogers@xxxxxxxxxx
> > Cc: peterz@xxxxxxxxxxxxx; mingo@xxxxxxxxxx; mark.rutland@xxxxxxx;
> > alexander.shishkin@xxxxxxxxxxxxxxx; jolsa@xxxxxxxxxx;
> > adrian.hunter@xxxxxxxxx; james.clark@xxxxxxxxxx; Besar Wicaksono
> > <bwicaksono@xxxxxxxxxx>; linux-perf-users@xxxxxxxxxxxxxxx; linux-
> > kernel@xxxxxxxxxxxxxxx; Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe
> > metrics
> >
> > External email: Use caution opening links or attachments
> >
> >
> > Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas
> > follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:
> >
> >   AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS
> >   AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS
> >
> > There is one PCIE PMU per socket and PCIe Root Complex (RC), named
> > nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each
> > socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:
> >
> >   lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs.
> >   lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one RC.
> >   lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
> >     CMEM, GMEM, PCIe peer and CXL memory.
> >   lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to remote
> >     memory.
> >
> > The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.
> >
> > A socket's event counts as 0 if its PMU isn't present (has_event() is
> > 0), e.g. on a single socket system, or if it isn't counted in the
> > aggregation (source_count() is 0), e.g. the other socket's event with
> > --per-socket. Otherwise the metric fails to parse or is nan.
> >
> > Signed-off-by: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > Assisted-by: Gemini:gemini-3.1-pro-preview
> > ---
> >  tools/perf/pmu-events/arm64_metrics.py | 68
> > ++++++++++++++++++++++++--
> >  1 file changed, 64 insertions(+), 4 deletions(-)
> >
> > diff --git a/tools/perf/pmu-events/arm64_metrics.py b/tools/perf/pmu-
> > events/arm64_metrics.py
> > index b0d0651fec12..1ca3d8ddb4b7 100755
> > --- a/tools/perf/pmu-events/arm64_metrics.py
> > +++ b/tools/perf/pmu-events/arm64_metrics.py
> > @@ -2,9 +2,10 @@
> >  # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >  import argparse
> >  import os
> > -from typing import Optional
> > -from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
> > -                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
> > MetricGroup)
> > +from typing import List, Optional, Union
> > +from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
> > Expression,
> > +                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
> > LoadEvents, Metric,
> > +                    MetricGroup, Select)
> >  from common_metrics import Cycles
> >
> >  # Global command line arguments.
> > @@ -67,7 +68,66 @@ def NvidiaT410Uncore() -> Optional[MetricGroup]:
> >                     rd_cum_outs - rd_cum_outs, "1ns"),
> >          ], description="NVIDIA Tegra410 DDR latency")
> >
> > -    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
> > +    def PcieBw() -> MetricGroup:
> > +        # PCIE PMU, one per socket and PCIe Root Complex (RC):
> > +        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
> > +        loc_dst =
> > "dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci
> > e_cxl=0x1"
> > +        rem_dst = "dst_rem=0x1"
> > +
> > +        def SocketSum(rc: str, event: str) -> Expression:
> > +            # Sum the event over both sockets. A PMU counts as 0 if it isn't
> > +            # present (has_event() is 0) or isn't counted in the aggregation,
> > +            # e.g. the other socket with --per-socket (source_count() is 0).
> > +            def Count(e: Event) -> Expression:
> > +                return Select(Select(e, source_count(e), 0), has_event(e), 0)
> > +
> > +            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
> > +                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
> > +
> > +        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in the
> > socket.
> > +        rd = SocketSum("rc", "rd_bytes")
> > +        wr = SocketSum("rc", "wr_bytes")
> > +        metrics: List[Union[Metric, MetricGroup]] = [
> > +            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root
> > Complexes",
> > +                   d_ratio(rd + wr, interval_sec), mb_per_sec),
> > +            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all
> > Root Complexes",
> > +                   d_ratio(rd, interval_sec), mb_per_sec),
> > +            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all
> > Root Complexes",
> > +                   d_ratio(wr, interval_sec), mb_per_sec),
> > +        ]
> > +        for i in range(6):
> > +            rc = f"rc_{i}"
> > +            rc_rd = SocketSum(rc, "rd_bytes")
> > +            rc_wr = SocketSum(rc, "wr_bytes")
> > +            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
> > +            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
> > +            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
> > +            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
> > +            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
> > +                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root
> > Complex {i}",
> > +                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root
> > Complex {i}",
> > +                       d_ratio(rc_rd, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on Root
> > Complex {i}",
> > +                       d_ratio(rc_wr, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_loc_rd_bw_{i}",
> > +                       f"PCIe local read bandwidth on Root Complex {i}",
> > +                       d_ratio(loc_rd, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_loc_wr_bw_{i}",
> > +                       f"PCIe local write bandwidth on Root Complex {i}",
> > +                       d_ratio(loc_wr, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_rem_rd_bw_{i}",
> > +                       f"PCIe remote read bandwidth on Root Complex {i}",
> > +                       d_ratio(rem_rd, interval_sec), mb_per_sec),
> > +                Metric(f"lpm_pcie_rem_wr_bw_{i}",
> > +                       f"PCIe remote write bandwidth on Root Complex {i}",
> > +                       d_ratio(rem_wr, interval_sec), mb_per_sec),
> > +            ], description=f"PCIe bandwidth on Root Complex {i}"))
> > +
> > +        return MetricGroup("lpm_pcie_bw", metrics,
> > +                           description="NVIDIA Tegra410 PCIe Root Complex
> > bandwidth")
> > +
> > +    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
> >                         description="NVIDIA Tegra410 uncore metrics")
> >
> >
> > --
> > 2.56.0.rc1.315.gc6ed9934b7-goog
>