Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

[Chun-Tse Shao]

Hi Besar,

Thanks, that makes sense. I'll drop my series in favor of yours.

Two things from mine that your JSON doesn't have yet:
1. DDR bandwidth from all sources to local CMEM (mem_bytes_{rd,wr}
with only dst_loc_cmem=1). Today it needs the sum of the local_cpu,
local_noncpu and remote _to_local_cmem metrics.
2. Read + write total bandwidth, for UCF memory and PCIe.

Agreed on handling the PCIe socket grouping in userspace; the
--metric-only issue I reported on your 3/6 is part of that.

Thanks,
CT

On Mon, Oct 19, 2026 at 10:43 AM Besar Wicaksono <bwicaksono@xxxxxxxxxx> wrote:
>
>
>
> > -----Original Message-----
> > From: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > Sent: Friday, October 16, 2026 6:40 PM
> > To: Besar Wicaksono <bwicaksono@xxxxxxxxxx>
> > Cc: acme@xxxxxxxxxx; namhyung@xxxxxxxxxx; irogers@xxxxxxxxxx;
> > peterz@xxxxxxxxxxxxx; mingo@xxxxxxxxxx; mark.rutland@xxxxxxx;
> > alexander.shishkin@xxxxxxxxxxxxxxx; jolsa@xxxxxxxxxx;
> > adrian.hunter@xxxxxxxxx; james.clark@xxxxxxxxxx; linux-perf-
> > users@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Matt Ochs
> > <mochs@xxxxxxxxxx>; Sean Kelley <skelley@xxxxxxxxxx>; Rich Wiley
> > <rwiley@xxxxxxxxxx>; Vikram Sethi <vsethi@xxxxxxxxxx>; Raj Joshi
> > <rajjoshi@xxxxxxxxxx>
> > Subject: Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe
> > metrics
> >
> > External email: Use caution opening links or attachments
> >
> >
> > On Tue, Oct 13, 2026 at 2:08 PM Besar Wicaksono
> > <bwicaksono@xxxxxxxxxx> wrote:
> > >
> > > Hi Chun-Tse,
> > >
> > > I tested this on 2-sockets Tegra410 and found some issues.
> > > 1. With --per-socket or -A, every lpm_pcie_* metric is only reported for
> > >    socket 0, so socket 1's PCIe traffic never shows up. Example output:
> > >
> > > perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1
> > >
> > >  Performance counter stats for 'system wide':
> > >
> > > CPU0               15,204      nvidia_pcie_pmu_0_rc_1/wr_bytes/ #      0.0 MB/s
> > lpm_pcie_wr_bw_1
> > > CPU0                                                                    #     33.8 MB/s  lpm_pcie_bw_1
> > > CPU0           33,910,952      nvidia_pcie_pmu_0_rc_1/rd_bytes/ #     33.8
> > MB/s  lpm_pcie_rd_bw_1
> > > CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
> > > CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
> > > CPU0        1,002,286,615      tool/duration_time/
> > > CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #
> > 0.0 MB/s  lpm_pcie_rem_wr_bw_1
> > > CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
> > > CPU0        1,002,275,059      tool/duration_time/
> > > CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #
> > 0.0 MB/s  lpm_pcie_rem_rd_bw_1
> > > CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/
> > > <snip>
> > >
> > > Note CPU88 (socket 1_ has no lpm_pcie_* values.
> > >
> > > 2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of
> > >    socket 1. These are separate root complexes with different devices,
> > >    so the "Root Complex i" value can combine traffic from two unrelated
> > >    links.
> > >    Could the metrics follow the hardware hierarchy instead, starting at
> > >    the socket and drilling down to the RC, with the same destination
> > >    split at each level?
> > >     - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
> > >         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
> > >         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
> > >     - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
> > >         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
> > >         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
> > >       grouped as lpm_pcie_s<s>_rc<i>_bw.
> >
> > Thanks for testing and for the suggestion. I think both problems come
> > from how perf expects a PMU to expose its cpumask. Ideally perf wants
> > a single PMU for both sockets, with one CPU per socket in its cpumask,
> > e.g.:
> >
> > ```
> > /sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
> > ```
> >
>
> I see this as a driver design choice: expose each physical PMU
> separately, or group several physical instances under one logical PMU.
> Either way, the PMU name and cpumask tell userspace which CPU(s) and
> root complex are associated with the PMU. With the first approach, each
> perf PMU has a cpumask of a single CPU. Tegra410 follows the first
> approach and exposes each physical PMU as a separate perf PMU.
>
> > We can work around the one-PMU-per-socket layout in the metrics, but
> > would a single PMU per RC be a viable plan for you to implement?
> >
>
> Grouping instances across sockets is an interesting alternative, but it
> would require changes to the arm_cspmu driver design. We would also need
> to evaluate how such grouping could be supported generically in
> arm_cspmu, and whether the ACPI APMT would need changes.
>
> For this patch, I think userspace gives us more flexibility in how we
> present and aggregate counts across sockets and root complexes, so I'd
> prefer to address it there and keep the current driver model.
>
> FYI, I've posted the Tegra410 uncore JSON series:
> https://lore.kernel.org/all/20261019164241.403644-1-bwicaksono@xxxxxxxxxx/
> It adds events and metrics for the same counters (e.g. ucf_avg_*,
> cmem_avg_rd_latency, pcie_avg_*), so it would be good to align the two
> and avoid duplicate metrics.
>
> Thanks,
> Besar
>
> > Thanks,
> > CT
> >
> > >
> > > Thanks,
> > > Besar
> > >
> > > > -----Original Message-----
> > > > From: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > > Sent: Monday, September 28, 2026 5:51 PM
> > > > To: acme@xxxxxxxxxx; namhyung@xxxxxxxxxx; irogers@xxxxxxxxxx
> > > > Cc: peterz@xxxxxxxxxxxxx; mingo@xxxxxxxxxx;
> > mark.rutland@xxxxxxx;
> > > > alexander.shishkin@xxxxxxxxxxxxxxx; jolsa@xxxxxxxxxx;
> > > > adrian.hunter@xxxxxxxxx; james.clark@xxxxxxxxxx; Besar Wicaksono
> > > > <bwicaksono@xxxxxxxxxx>; linux-perf-users@xxxxxxxxxxxxxxx; linux-
> > > > kernel@xxxxxxxxxxxxxxx; Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > > Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe
> > > > metrics
> > > >
> > > > External email: Use caution opening links or attachments
> > > >
> > > >
> > > > Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas
> > > > follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:
> > > >
> > > >   AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS
> > > >   AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS
> > > >
> > > > There is one PCIE PMU per socket and PCIe Root Complex (RC), named
> > > > nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each
> > > > socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:
> > > >
> > > >   lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs.
> > > >   lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one
> > RC.
> > > >   lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
> > > >     CMEM, GMEM, PCIe peer and CXL memory.
> > > >   lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to
> > remote
> > > >     memory.
> > > >
> > > > The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.
> > > >
> > > > A socket's event counts as 0 if its PMU isn't present (has_event() is
> > > > 0), e.g. on a single socket system, or if it isn't counted in the
> > > > aggregation (source_count() is 0), e.g. the other socket's event with
> > > > --per-socket. Otherwise the metric fails to parse or is nan.
> > > >
> > > > Signed-off-by: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > > Assisted-by: Gemini:gemini-3.1-pro-preview
> > > > ---
> > > >  tools/perf/pmu-events/arm64_metrics.py | 68
> > > > ++++++++++++++++++++++++--
> > > >  1 file changed, 64 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/tools/perf/pmu-events/arm64_metrics.py b/tools/perf/pmu-
> > > > events/arm64_metrics.py
> > > > index b0d0651fec12..1ca3d8ddb4b7 100755
> > > > --- a/tools/perf/pmu-events/arm64_metrics.py
> > > > +++ b/tools/perf/pmu-events/arm64_metrics.py
> > > > @@ -2,9 +2,10 @@
> > > >  # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> > > >  import argparse
> > > >  import os
> > > > -from typing import Optional
> > > > -from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
> > > > -                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
> > > > MetricGroup)
> > > > +from typing import List, Optional, Union
> > > > +from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
> > > > Expression,
> > > > +                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
> > > > LoadEvents, Metric,
> > > > +                    MetricGroup, Select)
> > > >  from common_metrics import Cycles
> > > >
> > > >  # Global command line arguments.
> > > > @@ -67,7 +68,66 @@ def NvidiaT410Uncore() ->
> > Optional[MetricGroup]:
> > > >                     rd_cum_outs - rd_cum_outs, "1ns"),
> > > >          ], description="NVIDIA Tegra410 DDR latency")
> > > >
> > > > -    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
> > > > +    def PcieBw() -> MetricGroup:
> > > > +        # PCIE PMU, one per socket and PCIe Root Complex (RC):
> > > > +        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
> > > > +        loc_dst =
> > > >
> > "dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci
> > > > e_cxl=0x1"
> > > > +        rem_dst = "dst_rem=0x1"
> > > > +
> > > > +        def SocketSum(rc: str, event: str) -> Expression:
> > > > +            # Sum the event over both sockets. A PMU counts as 0 if it isn't
> > > > +            # present (has_event() is 0) or isn't counted in the aggregation,
> > > > +            # e.g. the other socket with --per-socket (source_count() is 0).
> > > > +            def Count(e: Event) -> Expression:
> > > > +                return Select(Select(e, source_count(e), 0), has_event(e), 0)
> > > > +
> > > > +            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
> > > > +                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
> > > > +
> > > > +        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in
> > the
> > > > socket.
> > > > +        rd = SocketSum("rc", "rd_bytes")
> > > > +        wr = SocketSum("rc", "wr_bytes")
> > > > +        metrics: List[Union[Metric, MetricGroup]] = [
> > > > +            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root
> > > > Complexes",
> > > > +                   d_ratio(rd + wr, interval_sec), mb_per_sec),
> > > > +            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all
> > > > Root Complexes",
> > > > +                   d_ratio(rd, interval_sec), mb_per_sec),
> > > > +            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all
> > > > Root Complexes",
> > > > +                   d_ratio(wr, interval_sec), mb_per_sec),
> > > > +        ]
> > > > +        for i in range(6):
> > > > +            rc = f"rc_{i}"
> > > > +            rc_rd = SocketSum(rc, "rd_bytes")
> > > > +            rc_wr = SocketSum(rc, "wr_bytes")
> > > > +            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
> > > > +            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
> > > > +            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
> > > > +            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
> > > > +            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
> > > > +                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root
> > > > Complex {i}",
> > > > +                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root
> > > > Complex {i}",
> > > > +                       d_ratio(rc_rd, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on
> > Root
> > > > Complex {i}",
> > > > +                       d_ratio(rc_wr, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_loc_rd_bw_{i}",
> > > > +                       f"PCIe local read bandwidth on Root Complex {i}",
> > > > +                       d_ratio(loc_rd, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_loc_wr_bw_{i}",
> > > > +                       f"PCIe local write bandwidth on Root Complex {i}",
> > > > +                       d_ratio(loc_wr, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_rem_rd_bw_{i}",
> > > > +                       f"PCIe remote read bandwidth on Root Complex {i}",
> > > > +                       d_ratio(rem_rd, interval_sec), mb_per_sec),
> > > > +                Metric(f"lpm_pcie_rem_wr_bw_{i}",
> > > > +                       f"PCIe remote write bandwidth on Root Complex {i}",
> > > > +                       d_ratio(rem_wr, interval_sec), mb_per_sec),
> > > > +            ], description=f"PCIe bandwidth on Root Complex {i}"))
> > > > +
> > > > +        return MetricGroup("lpm_pcie_bw", metrics,
> > > > +                           description="NVIDIA Tegra410 PCIe Root Complex
> > > > bandwidth")
> > > > +
> > > > +    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
> > > >                         description="NVIDIA Tegra410 uncore metrics")
> > > >
> > > >
> > > > --
> > > > 2.56.0.rc1.315.gc6ed9934b7-goog
> > >