RE: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe metrics

[Besar Wicaksono]

> -----Original Message-----
> From: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> Sent: Friday, October 16, 2026 6:40 PM
> To: Besar Wicaksono <bwicaksono@xxxxxxxxxx>
> Cc: acme@xxxxxxxxxx; namhyung@xxxxxxxxxx; irogers@xxxxxxxxxx;
> peterz@xxxxxxxxxxxxx; mingo@xxxxxxxxxx; mark.rutland@xxxxxxx;
> alexander.shishkin@xxxxxxxxxxxxxxx; jolsa@xxxxxxxxxx;
> adrian.hunter@xxxxxxxxx; james.clark@xxxxxxxxxx; linux-perf-
> users@xxxxxxxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx; Matt Ochs
> <mochs@xxxxxxxxxx>; Sean Kelley <skelley@xxxxxxxxxx>; Rich Wiley
> <rwiley@xxxxxxxxxx>; Vikram Sethi <vsethi@xxxxxxxxxx>; Raj Joshi
> <rajjoshi@xxxxxxxxxx>
> Subject: Re: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe
> metrics
> 
> External email: Use caution opening links or attachments
> 
> 
> On Tue, Oct 13, 2026 at 2:08 PM Besar Wicaksono
> <bwicaksono@xxxxxxxxxx> wrote:
> >
> > Hi Chun-Tse,
> >
> > I tested this on 2-sockets Tegra410 and found some issues.
> > 1. With --per-socket or -A, every lpm_pcie_* metric is only reported for
> >    socket 0, so socket 1's PCIe traffic never shows up. Example output:
> >
> > perf stat -a -A -M lpm_pcie_rc1_bw -- sleep 1
> >
> >  Performance counter stats for 'system wide':
> >
> > CPU0               15,204      nvidia_pcie_pmu_0_rc_1/wr_bytes/ #      0.0 MB/s
> lpm_pcie_wr_bw_1
> > CPU0                                                                    #     33.8 MB/s  lpm_pcie_bw_1
> > CPU0           33,910,952      nvidia_pcie_pmu_0_rc_1/rd_bytes/ #     33.8
> MB/s  lpm_pcie_rd_bw_1
> > CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes/
> > CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes/
> > CPU0        1,002,286,615      tool/duration_time/
> > CPU0                    0      nvidia_pcie_pmu_0_rc_1/wr_bytes,dst_rem=0x1/ #
> 0.0 MB/s  lpm_pcie_rem_wr_bw_1
> > CPU88                   0      nvidia_pcie_pmu_1_rc_1/wr_bytes,dst_rem=0x1/
> > CPU0        1,002,275,059      tool/duration_time/
> > CPU0                    0      nvidia_pcie_pmu_0_rc_1/rd_bytes,dst_rem=0x1/ #
> 0.0 MB/s  lpm_pcie_rem_rd_bw_1
> > CPU88                   0      nvidia_pcie_pmu_1_rc_1/rd_bytes,dst_rem=0x1/
> > <snip>
> >
> > Note CPU88 (socket 1_ has no lpm_pcie_* values.
> >
> > 2. In the default mode, lpm_pcie_*_<i> adds RC i of socket 0 and RC i of
> >    socket 1. These are separate root complexes with different devices,
> >    so the "Root Complex i" value can combine traffic from two unrelated
> >    links.
> >    Could the metrics follow the hardware hierarchy instead, starting at
> >    the socket and drilling down to the RC, with the same destination
> >    split at each level?
> >     - per socket, all RCs (nvidia_pcie_pmu_<s>_rc):
> >         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}
> >         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw
> >     - per socket and RC (nvidia_pcie_pmu_<s>_rc_<i>):
> >         lpm_pcie_s<s>_{bw,rd_bw,wr_bw}_<i>
> >         lpm_pcie_s<s>_{loc,rem}_{rd,wr}_bw_<i>
> >       grouped as lpm_pcie_s<s>_rc<i>_bw.
> 
> Thanks for testing and for the suggestion. I think both problems come
> from how perf expects a PMU to expose its cpumask. Ideally perf wants
> a single PMU for both sockets, with one CPU per socket in its cpumask,
> e.g.:
> 
> ```
> /sys/bus/event_source/devices/nvidia_pcie_rc_0/cpumask: 0,88
> ```
> 

I see this as a driver design choice: expose each physical PMU
separately, or group several physical instances under one logical PMU.
Either way, the PMU name and cpumask tell userspace which CPU(s) and
root complex are associated with the PMU. With the first approach, each
perf PMU has a cpumask of a single CPU. Tegra410 follows the first
approach and exposes each physical PMU as a separate perf PMU.

> We can work around the one-PMU-per-socket layout in the metrics, but
> would a single PMU per RC be a viable plan for you to implement?
> 

Grouping instances across sockets is an interesting alternative, but it
would require changes to the arm_cspmu driver design. We would also need
to evaluate how such grouping could be supported generically in
arm_cspmu, and whether the ACPI APMT would need changes.

For this patch, I think userspace gives us more flexibility in how we
present and aggregate counts across sockets and root complexes, so I'd
prefer to address it there and keep the current driver model.

FYI, I've posted the Tegra410 uncore JSON series:
https://lore.kernel.org/all/20261019164241.403644-1-bwicaksono@xxxxxxxxxx/
It adds events and metrics for the same counters (e.g. ucf_avg_*,
cmem_avg_rd_latency, pcie_avg_*), so it would be good to align the two
and avoid duplicate metrics.

Thanks,
Besar

> Thanks,
> CT
> 
> >
> > Thanks,
> > Besar
> >
> > > -----Original Message-----
> > > From: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > Sent: Monday, September 28, 2026 5:51 PM
> > > To: acme@xxxxxxxxxx; namhyung@xxxxxxxxxx; irogers@xxxxxxxxxx
> > > Cc: peterz@xxxxxxxxxxxxx; mingo@xxxxxxxxxx;
> mark.rutland@xxxxxxx;
> > > alexander.shishkin@xxxxxxxxxxxxxxx; jolsa@xxxxxxxxxx;
> > > adrian.hunter@xxxxxxxxx; james.clark@xxxxxxxxxx; Besar Wicaksono
> > > <bwicaksono@xxxxxxxxxx>; linux-perf-users@xxxxxxxxxxxxxxx; linux-
> > > kernel@xxxxxxxxxxxxxxx; Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > Subject: [PATCH 2/2] perf jevents: Add NVIDIA Tegra410 uncore PCIe
> > > metrics
> > >
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > Add PCIe bandwidth metrics for the NVIDIA Tegra410 SoC. The formulas
> > > follow Documentation/admin-guide/perf/nvidia-tegra410-pmu.rst:
> > >
> > >   AVG_RD_BANDWIDTH_IN_GBPS = RD_BYTES / ELAPSED_TIME_IN_NS
> > >   AVG_WR_BANDWIDTH_IN_GBPS = WR_BYTES / ELAPSED_TIME_IN_NS
> > >
> > > There is one PCIE PMU per socket and PCIe Root Complex (RC), named
> > > nvidia_pcie_pmu_<socket-id>_rc_<pcie-rc-id>, with RCs 0 to 5 in each
> > > socket. The metrics sum the rd_bytes/wr_bytes events of both sockets:
> > >
> > >   lpm_pcie_bw, lpm_pcie_rd_bw, lpm_pcie_wr_bw: all RCs.
> > >   lpm_pcie_bw_<rc>, lpm_pcie_rd_bw_<rc>, lpm_pcie_wr_bw_<rc>: one
> RC.
> > >   lpm_pcie_loc_rd_bw_<rc>, lpm_pcie_loc_wr_bw_<rc>: one RC to local
> > >     CMEM, GMEM, PCIe peer and CXL memory.
> > >   lpm_pcie_rem_rd_bw_<rc>, lpm_pcie_rem_wr_bw_<rc>: one RC to
> remote
> > >     memory.
> > >
> > > The per RC metrics are in the lpm_pcie_rc<rc>_bw metric groups.
> > >
> > > A socket's event counts as 0 if its PMU isn't present (has_event() is
> > > 0), e.g. on a single socket system, or if it isn't counted in the
> > > aggregation (source_count() is 0), e.g. the other socket's event with
> > > --per-socket. Otherwise the metric fails to parse or is nan.
> > >
> > > Signed-off-by: Chun-Tse Shao <ctshao@xxxxxxxxxx>
> > > Assisted-by: Gemini:gemini-3.1-pro-preview
> > > ---
> > >  tools/perf/pmu-events/arm64_metrics.py | 68
> > > ++++++++++++++++++++++++--
> > >  1 file changed, 64 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/tools/perf/pmu-events/arm64_metrics.py b/tools/perf/pmu-
> > > events/arm64_metrics.py
> > > index b0d0651fec12..1ca3d8ddb4b7 100755
> > > --- a/tools/perf/pmu-events/arm64_metrics.py
> > > +++ b/tools/perf/pmu-events/arm64_metrics.py
> > > @@ -2,9 +2,10 @@
> > >  # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> > >  import argparse
> > >  import os
> > > -from typing import Optional
> > > -from metric import (aggr_nr, d_ratio, Event, JsonEncodeMetric,
> > > -                    JsonEncodeMetricGroupDescriptions, LoadEvents, Metric,
> > > MetricGroup)
> > > +from typing import List, Optional, Union
> > > +from metric import (aggr_nr, d_ratio, has_event, source_count, Event,
> > > Expression,
> > > +                    JsonEncodeMetric, JsonEncodeMetricGroupDescriptions,
> > > LoadEvents, Metric,
> > > +                    MetricGroup, Select)
> > >  from common_metrics import Cycles
> > >
> > >  # Global command line arguments.
> > > @@ -67,7 +68,66 @@ def NvidiaT410Uncore() ->
> Optional[MetricGroup]:
> > >                     rd_cum_outs - rd_cum_outs, "1ns"),
> > >          ], description="NVIDIA Tegra410 DDR latency")
> > >
> > > -    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat()],
> > > +    def PcieBw() -> MetricGroup:
> > > +        # PCIE PMU, one per socket and PCIe Root Complex (RC):
> > > +        # AVG_{RD,WR}_BANDWIDTH = {RD,WR}_BYTES / ELAPSED_TIME
> > > +        loc_dst =
> > >
> "dst_loc_cmem=0x1,dst_loc_gmem=0x1,dst_loc_pcie_p2p=0x1,dst_loc_pci
> > > e_cxl=0x1"
> > > +        rem_dst = "dst_rem=0x1"
> > > +
> > > +        def SocketSum(rc: str, event: str) -> Expression:
> > > +            # Sum the event over both sockets. A PMU counts as 0 if it isn't
> > > +            # present (has_event() is 0) or isn't counted in the aggregation,
> > > +            # e.g. the other socket with --per-socket (source_count() is 0).
> > > +            def Count(e: Event) -> Expression:
> > > +                return Select(Select(e, source_count(e), 0), has_event(e), 0)
> > > +
> > > +            return (Count(Event(f"nvidia_pcie_pmu_0_{rc}/{event}/")) +
> > > +                    Count(Event(f"nvidia_pcie_pmu_1_{rc}/{event}/")))
> > > +
> > > +        # nvidia_pcie_pmu_<socket-id>_rc matches the PMUs of all RCs in
> the
> > > socket.
> > > +        rd = SocketSum("rc", "rd_bytes")
> > > +        wr = SocketSum("rc", "wr_bytes")
> > > +        metrics: List[Union[Metric, MetricGroup]] = [
> > > +            Metric("lpm_pcie_bw", "Total PCIe bandwidth across all Root
> > > Complexes",
> > > +                   d_ratio(rd + wr, interval_sec), mb_per_sec),
> > > +            Metric("lpm_pcie_rd_bw", "Total PCIe read bandwidth across all
> > > Root Complexes",
> > > +                   d_ratio(rd, interval_sec), mb_per_sec),
> > > +            Metric("lpm_pcie_wr_bw", "Total PCIe write bandwidth across all
> > > Root Complexes",
> > > +                   d_ratio(wr, interval_sec), mb_per_sec),
> > > +        ]
> > > +        for i in range(6):
> > > +            rc = f"rc_{i}"
> > > +            rc_rd = SocketSum(rc, "rd_bytes")
> > > +            rc_wr = SocketSum(rc, "wr_bytes")
> > > +            loc_rd = SocketSum(rc, f"rd_bytes,{loc_dst}")
> > > +            loc_wr = SocketSum(rc, f"wr_bytes,{loc_dst}")
> > > +            rem_rd = SocketSum(rc, f"rd_bytes,{rem_dst}")
> > > +            rem_wr = SocketSum(rc, f"wr_bytes,{rem_dst}")
> > > +            metrics.append(MetricGroup(f"lpm_pcie_rc{i}_bw", [
> > > +                Metric(f"lpm_pcie_bw_{i}", f"PCIe total bandwidth on Root
> > > Complex {i}",
> > > +                       d_ratio(rc_rd + rc_wr, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_rd_bw_{i}", f"PCIe read bandwidth on Root
> > > Complex {i}",
> > > +                       d_ratio(rc_rd, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_wr_bw_{i}", f"PCIe write bandwidth on
> Root
> > > Complex {i}",
> > > +                       d_ratio(rc_wr, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_loc_rd_bw_{i}",
> > > +                       f"PCIe local read bandwidth on Root Complex {i}",
> > > +                       d_ratio(loc_rd, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_loc_wr_bw_{i}",
> > > +                       f"PCIe local write bandwidth on Root Complex {i}",
> > > +                       d_ratio(loc_wr, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_rem_rd_bw_{i}",
> > > +                       f"PCIe remote read bandwidth on Root Complex {i}",
> > > +                       d_ratio(rem_rd, interval_sec), mb_per_sec),
> > > +                Metric(f"lpm_pcie_rem_wr_bw_{i}",
> > > +                       f"PCIe remote write bandwidth on Root Complex {i}",
> > > +                       d_ratio(rem_wr, interval_sec), mb_per_sec),
> > > +            ], description=f"PCIe bandwidth on Root Complex {i}"))
> > > +
> > > +        return MetricGroup("lpm_pcie_bw", metrics,
> > > +                           description="NVIDIA Tegra410 PCIe Root Complex
> > > bandwidth")
> > > +
> > > +    return MetricGroup("lpm_uncore", [DdrBw(), DdrLat(), PcieBw()],
> > >                         description="NVIDIA Tegra410 uncore metrics")
> > >
> > >
> > > --
> > > 2.56.0.rc1.315.gc6ed9934b7-goog
> >