[PATCH 4/5] perf: add MIPS Coherence Manager 3 (CM3) PMU driver
From: Vladimir Kondratiev
Date: Sun Nov 08 2026 - 08:21:16 EST
The CM3 of the MIPS I6500 multiprocessing system serializes the requests
of all cores, IOCUs and the hardware prefetcher before they reach the
shared L2 and the memory interface. It provides two programmable 32-bit
event counters and a 32-bit cycle counter to observe this traffic:
coherence manager requests, I/O and memory interface requests, AXI bus
utilization and PDTrace statistics. Each event is refined by a 64-bit
qualifier split into groups of bits (port, thread, command, CCA, L1/L2
state, ...), where a request is counted when it matches a set bit in
every group.
Expose the counters as uncore PMUs bound to the coherency-manager DT
node, using the GCR block already mapped by mips_cm_probe(). Events are
described by 'event' (config:0-8, 0x100 selecting the cycle counter)
and 'qual' (config1:0-63); a qualifier of 0 picks a per-event default
that counts every occurrence. Named events are provided for the common
combinations and double as templates for custom qualifiers.
Every cluster has its own CM3 and its own counters, so one PMU named
mips_cm3_pmu_<cluster> is registered per cluster found in GCR_CONFIG.
A cluster does not need CPUs, online or at all, for its L2 and memory
interface to be busy, so the counters of other clusters are accessed
through the GCR redirect block from whichever CPU runs the PMU. This
only requires the CM of that cluster to be powered, which is checked
in the CPC (CM core STAT_CONF) when an event is opened: the counters
of a powered down cluster are initialized lazily on first use and
events on it fail with -ENODEV meanwhile. The counters are shared by
the whole cluster: events are system-wide only, pinned to a single CPU
which follows CPU hotplug; sampling, task events and the exclude_*
attributes are rejected.
Counter overflow is handled through the CM interrupt when the DT node
provides one, and by periodic polling otherwise, giving 64-bit counts
in both cases. Each cluster's CM drives its own interrupt output into
its own GIC, and a GIC shared interrupt is only routed to the cluster
of its affinity CPU, so the interrupt serves the cluster the CPUs live
in, and only when they all live in that one cluster; everything else
is polled.
Document the perf interface, the event numbers and the qualifier layouts
in Documentation/admin-guide/perf/mips_cm3_pmu.rst.
Signed-off-by: Vladimir Kondratiev <vladimir.kondratiev@xxxxxxxxxxxx>
---
Documentation/admin-guide/perf/index.rst | 1 +
Documentation/admin-guide/perf/mips_cm3_pmu.rst | 337 ++++++++
drivers/perf/Kconfig | 12 +
drivers/perf/Makefile | 1 +
drivers/perf/mips_cm3_pmu.c | 988 ++++++++++++++++++++++++
5 files changed, 1339 insertions(+)
diff --git a/Documentation/admin-guide/perf/index.rst b/Documentation/admin-guide/perf/index.rst
index aa12708ddb96..97620aa20465 100644
--- a/Documentation/admin-guide/perf/index.rst
+++ b/Documentation/admin-guide/perf/index.rst
@@ -14,6 +14,7 @@ Performance monitor support
qcom_l2_pmu
qcom_l3_pmu
starfive_starlink_pmu
+ mips_cm3_pmu
mrvl-odyssey-ddr-pmu
mrvl-odyssey-tad-pmu
arm-ccn
diff --git a/Documentation/admin-guide/perf/mips_cm3_pmu.rst b/Documentation/admin-guide/perf/mips_cm3_pmu.rst
new file mode 100644
index 000000000000..33cba0aca494
--- /dev/null
+++ b/Documentation/admin-guide/perf/mips_cm3_pmu.rst
@@ -0,0 +1,337 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================================================
+MIPS Coherence Manager 3 (CM3) Performance Monitoring Unit (PMU)
+=================================================================
+
+The Coherence Manager 3 (CM3) of the MIPS I6500 multiprocessing system is
+the point where the requests of all cores, I/O coherence units (IOCU) and
+the hardware prefetcher are serialized before reaching the shared L2 cache
+and the memory interface. The CM3 implements a small set of performance
+counters which observe this traffic:
+
+- two programmable 32-bit event counters, and
+- one 32-bit cycle counter clocked at the CM3 clock.
+
+The counters live in the global debug block of the CM Global Configuration
+Registers (``GCR_DB_PC_*``), are shared by all CPUs of the cluster, and count
+regardless of which CPU, privilege level or task generated the traffic. They
+are exposed by the ``mips_cm3_pmu`` driver (``CONFIG_MIPS_CM3_PMU``, built-in
+or as the ``mips_cm3_pmu`` module) as uncore PMUs.
+
+Clusters
+========
+
+A multi-cluster system has one CM3, and so one set of counters, per
+cluster. The number of clusters is read from ``GCR_CONFIG`` and the driver
+registers one PMU per cluster, named after the cluster number::
+
+ /sys/bus/event_source/devices/mips_cm3_pmu_0/
+ /sys/bus/event_source/devices/mips_cm3_pmu_1/
+
+A single-cluster system only has ``mips_cm3_pmu_0``. Each PMU counts the
+traffic of its own cluster: a request from a core in cluster 1 to the memory
+interface of cluster 1 is seen by ``mips_cm3_pmu_1`` only.
+
+A cluster does not need CPUs for its CM to be busy: its L2 cache and memory
+interface may serve I/O devices behind its IOCUs while its cores are absent,
+offline or powered down. The counters of any cluster are therefore reached
+through the GCR redirect block from whichever CPU runs the PMU, which can be
+in any cluster. What is needed is that the CM of the counted cluster is
+powered: the driver checks the CM power sequencer state in the Cluster Power
+Controller (``CPC_STAT_CONF`` of the CM "core") when an event is opened and
+fails with ``-ENODEV`` if the cluster is powered down. The PMU of the
+cluster the CPUs run in is always usable.
+
+Because the counters are shared, events are system wide only. Sampling
+(``perf record``), attaching to a task, and the ``exclude_*`` attributes are
+not supported; ``perf stat -a`` is the intended interface. The ``cpumask``
+attribute names the single CPU on which the counters are programmed and
+read; it migrates to another online CPU when that one goes offline::
+
+ $ cat /sys/bus/event_source/devices/mips_cm3_pmu_1/cpumask
+ 0
+
+Counter overflow
+================
+
+The CM3 raises its ``so_cm_perf_cnt_int`` output when a counter reaches
+0xFFFFFFFF. Whether that output reaches an interrupt controller input is a
+decision of the SoC integrator. If the ``coherency-manager`` device tree node
+carries an ``interrupts`` property, the driver uses it to extend the counters
+to 64 bits. Otherwise the driver polls the counters every 250 ms, which is
+well below the time the fastest event needs to wrap a 32-bit counter. Either
+way, the values reported by perf are 64-bit.
+
+The interrupt can only serve the cluster the CPUs live in, and only when
+they all live in that one cluster. Every cluster's CM drives its own
+interrupt output into its own GIC, and a GIC shared interrupt is routed to
+the cluster of its affinity CPU, so a single ``interrupts`` entry cannot
+serve several clusters at once. The PMUs of all other clusters, and all PMUs
+when CPUs are spread over several clusters, are polled.
+
+The mode chosen is printed at probe time, or the fact that the cluster was
+powered down when the driver probed; the counters are then initialized when
+the first event is opened::
+
+ mips_cm3_pmu coherency-manager: mips_cm3_pmu_0: 2 counters + cycles, polled
+ mips_cm3_pmu coherency-manager: mips_cm3_pmu_1: cluster powered down
+
+Event encoding
+==============
+
+An event is described by two fields, documented in the ``format`` directory::
+
+ $ cat /sys/bus/event_source/devices/mips_cm3_pmu_0/format/event
+ config:0-8
+ $ cat /sys/bus/event_source/devices/mips_cm3_pmu_0/format/qual
+ config1:0-63
+
+``event`` selects the event number written to ``GCR_DB_PC_EVENT``. The
+value ``0x100`` is a pseudo event selecting the cycle counter.
+
+``qual`` is the 64-bit qualifier written to ``GCR_DB_PC_QUALn``. It refines
+the event: the qualifier is split into groups of bits, each group covering
+one attribute of a request (originating port, command, cacheability, L2
+state, ...). A request is counted when, in **every** group, the bit
+matching its attribute is set. Setting several bits of a group ORs the
+alternatives; setting all bits of a group makes it a "don't care". A ``qual``
+of 0 (or omitting it) selects a per-event default that counts every
+occurrence of the event.
+
+Event numbers
+-------------
+
+=========== ========================== ======================================
+Event Name Description
+=========== ========================== ======================================
+0x01 CM requests Requests serialized by the coherence
+ manager (qualifier: Table 1)
+0x02 I/O traffic requests Requests received by an IOCU
+ (qualifier: Table 2)
+0x03 Memory interface requests Requests issued on the memory
+ interface (qualifier: Table 3)
+0x07 MEM AXI utilization Cycles on the memory AXI/ACE bus
+ (qualifier: Table 4)
+0x08 - 0x0f IOCU0-7 AXI utilization Cycles on the AXI bus of IOCU
+ (event - 0x08) (qualifier: Table 4)
+0x11 PDTrace dropped messages Messages dropped by CM PDTrace
+ (qualifier: Table 5)
+0x12 PDTrace overflow cycles Cycles spent in CM PDTrace overflow
+0x100 cycles CM3 cycle counter (pseudo event)
+=========== ========================== ======================================
+
+Table 1: Coherence Manager Requests qualifier (event 0x01)
+----------------------------------------------------------
+
+====== =========== ===================================================
+Bits Group Values (one bit each, LSB first)
+====== =========== ===================================================
+9:0 Port Input port 0..7, prefetcher (8), intervention (9).
+ Ports are numbered cores first, then IOCUs.
+13:10 Thread Thread 0..3
+15:14 Target Memory, register bus (GCR, GIC, CPC, ...)
+39:16 Command Legacy read, legacy write, CohReadOwn,
+ CohReadShare, CohReadDiscard, CohEvict,
+ CohUpgrade, CohUpgradeSC, CohWriteBack,
+ CohWriteInvalidate, CohReadDiscardAllocate,
+ prefetch own, prefetch share, prefetch write
+ invalidate, GetToOwn, Tag_Err, RegRead,
+ RegWrite, Sync, L1I cacheop, L1D cacheop,
+ L2 cacheop, L3 cacheop, other
+42:40 CCA Uncached, uncached accelerated, other
+44:43 Size Less than a line, one line
+47:45 L1 state Invalid, shared, exclusive/modified
+51:48 L2 state Invalid, shared, exclusive, modified
+53:52 L2 locked Not locked, locked
+56:54 Eviction None, L2 and L1 eviction, L2 eviction only
+58:57 Bank L2 bank 0, bank 1
+60:59 Scheduler Scheduler 0, scheduler 1
+63:61 Reserved Must be set
+====== =========== ===================================================
+
+Default: all bits set.
+
+Table 2: I/O Traffic Requests qualifier (event 0x02)
+----------------------------------------------------
+
+====== =============== ===============================================
+Bits Group Values
+====== =============== ===============================================
+2:0 IOCU number Encoded 0..7 (a value, not a mask)
+7:3 Direction/CCA Uncached read, coherent read with allocate,
+ coherent read without allocate, uncached
+ write, coherent write
+15:8 Size 1, 2, 4, 8, 16, 32, 64, 128 bytes per beat
+24:16 Length 1, 2, 3-4, 5-8, 9-16, 17-32, 33-64, 65-128,
+ 129-256 beats
+26:25 Alignment Aligned, misaligned
+28:27 Prefetch Not prefetch, prefetch
+29 All device IDs Set to ignore bits 35:30
+35:30 Device ID Encoded device ID
+36 All txn IDs Set to ignore bits 41:37
+41:37 Transaction ID Encoded transaction ID
+====== =============== ===============================================
+
+Default: IOCU 0, every other group "don't care" (``0x3fffffffff8``).
+
+Table 3: Memory Interface Requests qualifier (event 0x03)
+---------------------------------------------------------
+
+====== =============== ===============================================
+Bits Group Values
+====== =============== ===============================================
+1:0 Direction Read, write
+9:2 Size 1, 2, 4, 8, 16, 32, 64, 128 bytes per beat
+17:10 Length 0..7 beats
+21:18 Cacheability Uncached, uncached accelerated, cacheable read
+ discard, cacheable not read discard
+29:22 Source Input port 0..7
+35:32 Thread Thread 0..3
+37:36 Code/data Data, code
+39:38 Prefetch Not prefetch, prefetch
+40 All guest IDs Set to ignore bits 45:41
+45:41 Guest ID Encoded guest ID
+====== =============== ===============================================
+
+Default: all groups "don't care" (``0x3fffffffffff``).
+
+Table 4: AXI Bus Utilization qualifier (events 0x07 - 0x0f)
+-----------------------------------------------------------
+
+These events count cycles, qualified by the state of the handshake signals
+of one AXI channel.
+
+====== =============== ===============================================
+Bits Group Values
+====== =============== ===============================================
+0 Not valid Count cycles where xVALID is deasserted
+1 Valid Count cycles where xVALID is asserted
+2 Not ready Count cycles where xREADY is deasserted
+3 Ready Count cycles where xREADY is asserted
+6:4 Channel Encoded: 0 AR, 1 AW, 2 W, 3 R, 4 B
+====== =============== ===============================================
+
+Setting both ``valid`` and ``ready`` counts completed transfers. Default:
+transfers on the AR channel (``0xa``).
+
+Table 5: CM PDTrace Dropped Messages qualifier (event 0x11)
+-----------------------------------------------------------
+
+====== =============== ===============================================
+Bits Group Values
+====== =============== ===============================================
+5:0 Trace port ubrh, prsh, tmh0, tmh0_mulp, tmh1, tmh1_mulp
+7:6 Trace type Request, response
+====== =============== ===============================================
+
+Default: all bits set.
+
+Named events
+============
+
+The ``events`` directory provides ready-made event/qualifier pairs for the
+most useful combinations::
+
+ $ perf list mips_cm3_pmu_0
+
+ mips_cm3_pmu_0/cm_requests/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_coh_writeback/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_l2_hit/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_l2_miss/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_legacy_read/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_legacy_write/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_mem/ [Kernel PMU event]
+ mips_cm3_pmu_0/cm_requests_regbus/ [Kernel PMU event]
+ mips_cm3_pmu_0/cycles/ [Kernel PMU event]
+ mips_cm3_pmu_0/iocu0_requests/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_axi_ar/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_axi_aw/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_axi_b/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_axi_r/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_axi_w/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_reads/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_requests/ [Kernel PMU event]
+ mips_cm3_pmu_0/mem_writes/ [Kernel PMU event]
+ mips_cm3_pmu_0/pdtrace_dropped/ [Kernel PMU event]
+ mips_cm3_pmu_0/pdtrace_overflow_cycles/ [Kernel PMU event]
+
+============================== =============================================
+Name Counts
+============================== =============================================
+cycles CM3 clock cycles
+cm_requests Every request serialized by the CM
+cm_requests_mem Requests targeting memory
+cm_requests_regbus Requests targeting the register bus
+cm_requests_legacy_read Non-coherent reads, including code fetches
+cm_requests_legacy_write Non-coherent writes
+cm_requests_coh_writeback Coherent write backs
+cm_requests_l2_hit Memory requests whose line is present in L2
+cm_requests_l2_miss Memory requests whose line is not in L2
+iocu0_requests Requests received by IOCU 0
+mem_requests Requests issued on the memory interface
+mem_reads Memory interface reads
+mem_writes Memory interface writes
+mem_axi_ar/aw/w/r/b Completed transfers on the given channel
+ of the memory AXI bus
+pdtrace_dropped Messages dropped by CM PDTrace
+pdtrace_overflow_cycles Cycles spent in CM PDTrace overflow
+============================== =============================================
+
+The named events are plain ``event=...,qual=...`` strings, so they serve as
+templates: read one from sysfs and clear the bits of the group you want to
+narrow down.
+
+Examples
+========
+
+Count all CM requests and the share coming from port 1 (core 1), together
+with cycles, for one second::
+
+ $ perf stat -a -e mips_cm3_pmu_0/cycles/ \
+ -e mips_cm3_pmu_0/cm_requests/ \
+ -e mips_cm3_pmu_0/event=0x1,qual=0xfffffffffffffc02/ \
+ sleep 1
+
+The second qualifier is the ``cm_requests`` default with the Port group
+(bits 9:0) reduced to bit 1.
+
+L2 hit rate of memory requests::
+
+ $ perf stat -a -e mips_cm3_pmu_0/cm_requests_l2_hit/ \
+ -e mips_cm3_pmu_0/cm_requests_l2_miss/ sleep 1
+
+Utilization of the memory read data channel: cycles with a transfer versus
+cycles where the CM waits for data (valid deasserted)::
+
+ $ perf stat -a -e mips_cm3_pmu_0/cycles/ \
+ -e mips_cm3_pmu_0/mem_axi_r/ \
+ -e mips_cm3_pmu_0/event=0x7,qual=0x3d/ sleep 1
+
+Requests received by IOCU 1 (IOCU number is an encoded value in bits 2:0)::
+
+ $ perf stat -a -e mips_cm3_pmu_0/event=0x2,qual=0x3fffffffff9/ sleep 1
+
+Memory interface traffic of every cluster of a two-cluster system::
+
+ $ perf stat -a -e mips_cm3_pmu_0/mem_requests/ \
+ -e mips_cm3_pmu_1/mem_requests/ sleep 1
+
+Only two programmable counters exist; a group requesting more than two
+non-cycle events, or more than one ``cycles`` event, is rejected with
+``-EINVAL``.
+
+Device tree
+===========
+
+The driver binds to the ``coherency-manager`` node (``mti,mips-cm`` or
+``mobileye,eyeq6-cm``) and uses the GCR base address already discovered by
+the kernel from the ``CMGCRBase`` register. An ``interrupts`` property on
+that node, when the SoC routes ``so_cm_perf_cnt_int`` to an interrupt
+controller, enables interrupt-driven overflow handling::
+
+ coherency-manager {
+ compatible = "mobileye,eyeq6-cm";
+ interrupt-parent = <&gic>;
+ interrupts = <GIC_SHARED 42 IRQ_TYPE_LEVEL_HIGH>;
+ };
diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
index 245e7bb763b9..2dbed2fc7b03 100644
--- a/drivers/perf/Kconfig
+++ b/drivers/perf/Kconfig
@@ -75,6 +75,18 @@ config ARM_XSCALE_PMU
depends on ARM_PMU && CPU_XSCALE
def_bool y
+config MIPS_CM3_PMU
+ tristate "MIPS Coherence Manager 3 (CM3) PMU"
+ depends on MIPS_CM
+ help
+ Support for the performance counters of the MIPS Coherence Manager 3
+ (CM3) found in the I6500 multiprocessing system. The CM3 provides two
+ programmable event counters and a cycle counter which can count
+ requests serialized by the coherence manager, I/O and memory interface
+ traffic, and AXI bus utilization, each refined by a qualifier.
+
+ If compiled as a module, it will be called mips_cm3_pmu.
+
config RISCV_PMU
depends on RISCV
bool "RISC-V PMU framework"
diff --git a/drivers/perf/Makefile b/drivers/perf/Makefile
index eb8a022dad9a..5f0920ba7058 100644
--- a/drivers/perf/Makefile
+++ b/drivers/perf/Makefile
@@ -17,6 +17,7 @@ obj-$(CONFIG_FUJITSU_UNCORE_PMU) += fujitsu_uncore_pmu.o
obj-$(CONFIG_HISI_PMU) += hisilicon/
obj-$(CONFIG_QCOM_L2_PMU) += qcom_l2_pmu.o
obj-$(CONFIG_QCOM_L3_PMU) += qcom_l3_pmu.o
+obj-$(CONFIG_MIPS_CM3_PMU) += mips_cm3_pmu.o
obj-$(CONFIG_RISCV_PMU) += riscv_pmu.o
obj-$(CONFIG_RISCV_PMU_LEGACY) += riscv_pmu_legacy.o
obj-$(CONFIG_RISCV_PMU_SBI) += riscv_pmu_sbi.o
diff --git a/drivers/perf/mips_cm3_pmu.c b/drivers/perf/mips_cm3_pmu.c
new file mode 100644
index 000000000000..d4be23e79d6f
--- /dev/null
+++ b/drivers/perf/mips_cm3_pmu.c
@@ -0,0 +1,988 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * MIPS Coherence Manager 3 (CM3) performance counter PMU driver
+ *
+ * The CM3 found in the MIPS I6500 multiprocessing system implements two
+ * programmable 32-bit event counters and a 32-bit cycle counter. They are
+ * controlled through the GCR global debug block (GCR_DB_PC_*) and count
+ * events happening at the coherence manager: serialized requests, I/O and
+ * memory traffic, AXI bus utilization, ...
+ *
+ * perf interface:
+ *
+ * config [8:0] event number (GCR_DB_PC_EVENT.Px_Event), 0x100 selects
+ * the cycle counter.
+ * config1 [63:0] event qualifier (GCR_DB_PC_QUALn). A qualifier is split
+ * into groups of bits, each selecting one attribute of the
+ * request; an event is counted when it matches at least one
+ * set bit of every group. Setting all bits of a group makes
+ * it a "don't care". A qualifier of 0 selects a sensible
+ * default which counts every occurrence of the event.
+ *
+ * Every cluster has its own CM3 and therefore its own set of counters, and
+ * one PMU instance (mips_cm3_pmu_<cluster>) is registered per cluster. A
+ * cluster does not need CPUs, online or at all, for its L2 and memory
+ * interface to be busy, so the counters of other clusters are reached
+ * through the GCR redirect block from whichever CPU runs the PMU. This
+ * requires the CM of that cluster to be powered, which is checked in the
+ * CPC before an event is opened.
+ *
+ * The counters are shared by the whole cluster, so events are system wide,
+ * pinned to a single CPU, and cannot be sampled.
+ */
+
+#define CM3_PMU_NAME "mips_cm3_pmu"
+
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/perf_event.h>
+#include <linux/platform_device.h>
+
+#include <asm/cpu-info.h>
+#include <asm/mips-cps.h>
+
+/* Two programmable counters followed by the cycle counter */
+#define CM3_PMU_MAX_PCS 2
+#define CM3_PMU_IDX_CYCLES CM3_PMU_MAX_PCS
+#define CM3_PMU_NUM_COUNTERS (CM3_PMU_MAX_PCS + 1)
+#define CM3_PMU_COUNTER_MASK GENMASK_ULL(31, 0)
+
+/*
+ * Without an overflow interrupt the 32-bit counters are read periodically.
+ * The fastest events count a couple of occurrences per CM clock cycle, so
+ * even at 2GHz a counter takes more than a second to wrap.
+ */
+#define CM3_PMU_POLL_PERIOD_NS (250 * NSEC_PER_MSEC)
+
+/* Event numbers (GCR_DB_PC_EVENT.Px_Event) */
+#define CM3_PMU_EV_NONE 0x00
+#define CM3_PMU_EV_CM_REQ 0x01 /* Coherence Manager requests */
+#define CM3_PMU_EV_IO_REQ 0x02 /* I/O traffic requests (IOCU) */
+#define CM3_PMU_EV_MEM_REQ 0x03 /* Memory interface requests */
+#define CM3_PMU_EV_MEM_AXI 0x07 /* MEM AXI bus utilization */
+#define CM3_PMU_EV_IOCU_AXI(n) (0x08 + (n)) /* IOCUn AXI bus utilization */
+#define CM3_PMU_EV_IOCU_AXI_LAST CM3_PMU_EV_IOCU_AXI(7)
+#define CM3_PMU_EV_PDTRACE_DROPPED 0x11 /* CM PDTrace dropped messages */
+#define CM3_PMU_EV_PDTRACE_OVF_CYCLES 0x12 /* CM PDTrace overflow cycles */
+#define CM3_PMU_EV_HW_MASK GENMASK(7, 0)
+/* Pseudo event selecting the cycle counter */
+#define CM3_PMU_EV_CYCLES 0x100
+#define CM3_PMU_EV_MASK GENMASK(8, 0)
+
+/*
+ * Qualifier of the Coherence Manager Requests event. Ports are numbered
+ * with the cores first, followed by the IOCUs; port bit 8 is the prefetcher
+ * and bit 9 interventions.
+ */
+#define CM3_PMU_Q_CMREQ_PORT GENMASK_ULL(9, 0)
+#define CM3_PMU_Q_CMREQ_THREAD GENMASK_ULL(13, 10)
+#define CM3_PMU_Q_CMREQ_TARGET GENMASK_ULL(15, 14)
+#define CM3_PMU_Q_CMREQ_TARGET_MEM BIT_ULL(14)
+#define CM3_PMU_Q_CMREQ_TARGET_REGBUS BIT_ULL(15)
+#define CM3_PMU_Q_CMREQ_CMD GENMASK_ULL(39, 16)
+#define CM3_PMU_Q_CMREQ_CMD_LEGACY_RD BIT_ULL(16)
+#define CM3_PMU_Q_CMREQ_CMD_LEGACY_WR BIT_ULL(17)
+#define CM3_PMU_Q_CMREQ_CMD_COH_WB BIT_ULL(24)
+#define CM3_PMU_Q_CMREQ_CCA GENMASK_ULL(42, 40) /* UC, UCA, other */
+#define CM3_PMU_Q_CMREQ_SIZE GENMASK_ULL(44, 43) /* sub-line, line */
+#define CM3_PMU_Q_CMREQ_L1_STATE GENMASK_ULL(47, 45) /* I, S, E/M */
+#define CM3_PMU_Q_CMREQ_L2_STATE GENMASK_ULL(51, 48) /* I, S, E, M */
+#define CM3_PMU_Q_CMREQ_L2_INVALID BIT_ULL(48)
+#define CM3_PMU_Q_CMREQ_L2_LOCKED GENMASK_ULL(53, 52)
+#define CM3_PMU_Q_CMREQ_EVICTION GENMASK_ULL(56, 54)
+#define CM3_PMU_Q_CMREQ_BANK GENMASK_ULL(58, 57)
+#define CM3_PMU_Q_CMREQ_SCHEDULER GENMASK_ULL(60, 59)
+#define CM3_PMU_Q_CMREQ_RESERVED GENMASK_ULL(63, 61) /* must be set */
+#define CM3_PMU_Q_CMREQ_DEFAULT GENMASK_ULL(63, 0)
+
+/*
+ * Qualifier of the I/O Traffic Requests event. The IOCU number is an
+ * encoded value, not a mask. Direction bits: UC read, coherent read with
+ * allocate, coherent read without allocate, UC write, coherent write.
+ */
+#define CM3_PMU_Q_IO_IOCU GENMASK_ULL(2, 0)
+#define CM3_PMU_Q_IO_DIR GENMASK_ULL(7, 3)
+#define CM3_PMU_Q_IO_SIZE GENMASK_ULL(15, 8) /* 1 .. 128 bytes */
+#define CM3_PMU_Q_IO_LENGTH GENMASK_ULL(24, 16) /* 1 .. 129-256 beats */
+#define CM3_PMU_Q_IO_ALIGNED GENMASK_ULL(26, 25)
+#define CM3_PMU_Q_IO_PREFETCH GENMASK_ULL(28, 27)
+#define CM3_PMU_Q_IO_ALL_DEVID BIT_ULL(29)
+#define CM3_PMU_Q_IO_DEVID GENMASK_ULL(35, 30)
+#define CM3_PMU_Q_IO_ALL_TXNID BIT_ULL(36)
+#define CM3_PMU_Q_IO_TXNID GENMASK_ULL(41, 37)
+#define CM3_PMU_Q_IO_DEFAULT GENMASK_ULL(41, 3) /* IOCU 0 */
+
+/* Qualifier of the Memory Interface Requests event */
+#define CM3_PMU_Q_MEM_DIR GENMASK_ULL(1, 0)
+#define CM3_PMU_Q_MEM_DIR_READ BIT_ULL(0)
+#define CM3_PMU_Q_MEM_DIR_WRITE BIT_ULL(1)
+#define CM3_PMU_Q_MEM_SIZE GENMASK_ULL(9, 2) /* 1 .. 128 bytes */
+#define CM3_PMU_Q_MEM_LENGTH GENMASK_ULL(17, 10) /* 0 .. 7 */
+#define CM3_PMU_Q_MEM_CCA GENMASK_ULL(21, 18) /* UC, UCA, read discard, cacheable */
+#define CM3_PMU_Q_MEM_SOURCE GENMASK_ULL(29, 22) /* input port */
+#define CM3_PMU_Q_MEM_THREAD GENMASK_ULL(35, 32)
+#define CM3_PMU_Q_MEM_CODE_DATA GENMASK_ULL(37, 36)
+#define CM3_PMU_Q_MEM_PREFETCH GENMASK_ULL(39, 38)
+#define CM3_PMU_Q_MEM_ALL_GUESTID BIT_ULL(40)
+#define CM3_PMU_Q_MEM_GUESTID GENMASK_ULL(45, 41)
+#define CM3_PMU_Q_MEM_DEFAULT GENMASK_ULL(45, 0)
+
+/* Qualifier of the AXI Bus Utilization events, counts cycles */
+#define CM3_PMU_Q_AXI_NOT_VALID BIT_ULL(0)
+#define CM3_PMU_Q_AXI_VALID BIT_ULL(1)
+#define CM3_PMU_Q_AXI_NOT_READY BIT_ULL(2)
+#define CM3_PMU_Q_AXI_READY BIT_ULL(3)
+#define CM3_PMU_Q_AXI_CHANNEL GENMASK_ULL(6, 4)
+#define CM3_PMU_Q_AXI_CHANNEL_AR 0
+#define CM3_PMU_Q_AXI_CHANNEL_AW 1
+#define CM3_PMU_Q_AXI_CHANNEL_W 2
+#define CM3_PMU_Q_AXI_CHANNEL_R 3
+#define CM3_PMU_Q_AXI_CHANNEL_B 4
+/* Completed transfers (VALID && READY) on a channel */
+#define CM3_PMU_Q_AXI_XFER(ch) (FIELD_PREP_CONST(CM3_PMU_Q_AXI_CHANNEL, ch) | \
+ CM3_PMU_Q_AXI_VALID | CM3_PMU_Q_AXI_READY)
+#define CM3_PMU_Q_AXI_DEFAULT CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_AR)
+
+/* Qualifier of the CM PDTrace Dropped Messages event */
+#define CM3_PMU_Q_PDTRACE_PORT GENMASK_ULL(5, 0)
+#define CM3_PMU_Q_PDTRACE_TYPE GENMASK_ULL(7, 6) /* request, response */
+#define CM3_PMU_Q_PDTRACE_DEFAULT GENMASK_ULL(7, 0)
+
+/* Restrict a single qualifier group of a default qualifier to @bits */
+#define CM3_PMU_QUAL(def, group, bits) (((def) & ~(group)) | (bits))
+
+struct cm3_pmu {
+ struct pmu pmu;
+ struct hlist_node node;
+ struct hrtimer hrtimer;
+ struct perf_event *events[CM3_PMU_NUM_COUNTERS];
+ struct mutex init_lock; /* serializes cm3_pmu_hw_init() */
+ bool initialized;
+ unsigned int cluster;
+ unsigned int num_pcs;
+ unsigned int num_active;
+ int on_cpu;
+ int irq;
+};
+
+struct cm3_pmu_dev {
+ unsigned int num_pmus;
+ struct cm3_pmu pmus[] __counted_by(num_pmus);
+};
+
+#define to_cm3_pmu(p) container_of(p, struct cm3_pmu, pmu)
+
+static int cm3_pmu_cpuhp_state;
+
+/* sysfs attributes */
+
+static ssize_t cm3_pmu_format_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct dev_ext_attribute *eattr;
+
+ eattr = container_of(attr, struct dev_ext_attribute, attr);
+ return sysfs_emit(buf, "%s\n", (char *)eattr->var);
+}
+
+#define CM3_PMU_FORMAT_ATTR(_name, _config) \
+ (&((struct dev_ext_attribute[]) { \
+ { .attr = __ATTR(_name, 0444, cm3_pmu_format_show, NULL), \
+ .var = (void *)_config, } \
+ })[0].attr.attr)
+
+static struct attribute *cm3_pmu_format_attrs[] = {
+ CM3_PMU_FORMAT_ATTR(event, "config:0-8"),
+ CM3_PMU_FORMAT_ATTR(qual, "config1:0-63"),
+ NULL
+};
+
+static const struct attribute_group cm3_pmu_format_attr_group = {
+ .name = "format",
+ .attrs = cm3_pmu_format_attrs,
+};
+
+struct cm3_pmu_event_attr {
+ struct device_attribute attr;
+ u16 event;
+ u64 qual;
+};
+
+static ssize_t cm3_pmu_event_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cm3_pmu_event_attr *eattr;
+
+ eattr = container_of(attr, struct cm3_pmu_event_attr, attr);
+ if (!eattr->qual)
+ return sysfs_emit(buf, "event=0x%x\n", eattr->event);
+
+ return sysfs_emit(buf, "event=0x%x,qual=0x%llx\n",
+ eattr->event, eattr->qual);
+}
+
+#define CM3_PMU_EVENT_ATTR(_name, _event, _qual) \
+ (&((struct cm3_pmu_event_attr[]) { \
+ { .attr = __ATTR(_name, 0444, cm3_pmu_event_show, NULL), \
+ .event = _event, .qual = _qual, } \
+ })[0].attr.attr)
+
+static struct attribute *cm3_pmu_event_attrs[] = {
+ CM3_PMU_EVENT_ATTR(cycles, CM3_PMU_EV_CYCLES, 0),
+
+ /* Requests serialized by the coherence manager */
+ CM3_PMU_EVENT_ATTR(cm_requests, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_Q_CMREQ_DEFAULT),
+ CM3_PMU_EVENT_ATTR(cm_requests_mem, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_TARGET,
+ CM3_PMU_Q_CMREQ_TARGET_MEM)),
+ CM3_PMU_EVENT_ATTR(cm_requests_regbus, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_TARGET,
+ CM3_PMU_Q_CMREQ_TARGET_REGBUS)),
+ CM3_PMU_EVENT_ATTR(cm_requests_legacy_read, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_CMD,
+ CM3_PMU_Q_CMREQ_CMD_LEGACY_RD)),
+ CM3_PMU_EVENT_ATTR(cm_requests_legacy_write, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_CMD,
+ CM3_PMU_Q_CMREQ_CMD_LEGACY_WR)),
+ CM3_PMU_EVENT_ATTR(cm_requests_coh_writeback, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_CMD,
+ CM3_PMU_Q_CMREQ_CMD_COH_WB)),
+ /* Memory requests whose line is (not) present in the L2 */
+ CM3_PMU_EVENT_ATTR(cm_requests_l2_hit, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_TARGET,
+ CM3_PMU_Q_CMREQ_TARGET_MEM),
+ CM3_PMU_Q_CMREQ_L2_STATE,
+ CM3_PMU_Q_CMREQ_L2_STATE &
+ ~CM3_PMU_Q_CMREQ_L2_INVALID)),
+ CM3_PMU_EVENT_ATTR(cm_requests_l2_miss, CM3_PMU_EV_CM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_QUAL(CM3_PMU_Q_CMREQ_DEFAULT,
+ CM3_PMU_Q_CMREQ_TARGET,
+ CM3_PMU_Q_CMREQ_TARGET_MEM),
+ CM3_PMU_Q_CMREQ_L2_STATE,
+ CM3_PMU_Q_CMREQ_L2_INVALID)),
+
+ /* Requests received by IOCU 0 */
+ CM3_PMU_EVENT_ATTR(iocu0_requests, CM3_PMU_EV_IO_REQ,
+ CM3_PMU_Q_IO_DEFAULT),
+
+ /* Requests issued on the memory interface */
+ CM3_PMU_EVENT_ATTR(mem_requests, CM3_PMU_EV_MEM_REQ,
+ CM3_PMU_Q_MEM_DEFAULT),
+ CM3_PMU_EVENT_ATTR(mem_reads, CM3_PMU_EV_MEM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_MEM_DEFAULT,
+ CM3_PMU_Q_MEM_DIR,
+ CM3_PMU_Q_MEM_DIR_READ)),
+ CM3_PMU_EVENT_ATTR(mem_writes, CM3_PMU_EV_MEM_REQ,
+ CM3_PMU_QUAL(CM3_PMU_Q_MEM_DEFAULT,
+ CM3_PMU_Q_MEM_DIR,
+ CM3_PMU_Q_MEM_DIR_WRITE)),
+
+ /* Transfers on each channel of the memory AXI bus */
+ CM3_PMU_EVENT_ATTR(mem_axi_ar, CM3_PMU_EV_MEM_AXI,
+ CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_AR)),
+ CM3_PMU_EVENT_ATTR(mem_axi_aw, CM3_PMU_EV_MEM_AXI,
+ CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_AW)),
+ CM3_PMU_EVENT_ATTR(mem_axi_w, CM3_PMU_EV_MEM_AXI,
+ CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_W)),
+ CM3_PMU_EVENT_ATTR(mem_axi_r, CM3_PMU_EV_MEM_AXI,
+ CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_R)),
+ CM3_PMU_EVENT_ATTR(mem_axi_b, CM3_PMU_EV_MEM_AXI,
+ CM3_PMU_Q_AXI_XFER(CM3_PMU_Q_AXI_CHANNEL_B)),
+
+ CM3_PMU_EVENT_ATTR(pdtrace_dropped, CM3_PMU_EV_PDTRACE_DROPPED,
+ CM3_PMU_Q_PDTRACE_DEFAULT),
+ CM3_PMU_EVENT_ATTR(pdtrace_overflow_cycles,
+ CM3_PMU_EV_PDTRACE_OVF_CYCLES, 0),
+ NULL
+};
+
+static const struct attribute_group cm3_pmu_events_attr_group = {
+ .name = "events",
+ .attrs = cm3_pmu_event_attrs,
+};
+
+static ssize_t cpumask_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(dev_get_drvdata(dev));
+
+ return sysfs_emit(buf, "%*pbl\n",
+ cpumask_pr_args(cpumask_of(cm3_pmu->on_cpu)));
+}
+static DEVICE_ATTR_RO(cpumask);
+
+static struct attribute *cm3_pmu_cpumask_attrs[] = {
+ &dev_attr_cpumask.attr,
+ NULL
+};
+
+static const struct attribute_group cm3_pmu_cpumask_attr_group = {
+ .attrs = cm3_pmu_cpumask_attrs,
+};
+
+static const struct attribute_group *cm3_pmu_attr_groups[] = {
+ &cm3_pmu_format_attr_group,
+ &cm3_pmu_events_attr_group,
+ &cm3_pmu_cpumask_attr_group,
+ NULL
+};
+
+/*
+ * Hardware access
+ *
+ * The debug block of the local cluster is accessed directly. That of another
+ * cluster is accessed through the redirect block, pointed at the debug block
+ * of the target cluster, from whichever CPU this runs on. Both paths are
+ * valid in any context; the redirect path takes the per-CPU redirect lock
+ * with interrupts disabled.
+ */
+
+static bool cm3_pmu_lock(struct cm3_pmu *cm3_pmu)
+{
+ preempt_disable();
+ if (cm3_pmu->cluster == cpu_cluster(¤t_cpu_data))
+ return false;
+
+ mips_cm_lock_other(cm3_pmu->cluster, 0, 0, CM_GCR_Cx_OTHER_BLOCK_DEBUG);
+ return true;
+}
+
+static void cm3_pmu_unlock(bool other)
+{
+ if (other)
+ mips_cm_unlock_other();
+ preempt_enable();
+}
+
+#define CM3_PMU_READ(cm3_pmu, reg) \
+({ \
+ bool __other = cm3_pmu_lock(cm3_pmu); \
+ typeof(read_gcr_db_##reg()) __val; \
+ \
+ __val = __other ? read_gcr_redir_db_##reg() : read_gcr_db_##reg(); \
+ cm3_pmu_unlock(__other); \
+ __val; \
+})
+
+#define CM3_PMU_WRITE(cm3_pmu, reg, val) \
+do { \
+ typeof(read_gcr_db_##reg()) __val = (val); \
+ bool __other = cm3_pmu_lock(cm3_pmu); \
+ \
+ if (__other) \
+ write_gcr_redir_db_##reg(__val); \
+ else \
+ write_gcr_db_##reg(__val); \
+ cm3_pmu_unlock(__other); \
+} while (0)
+
+/*
+ * Whether the CM of the cluster is powered. The CPC is always powered and
+ * reports the CM power sequencer state in the STAT_CONF of its "CM core".
+ * Accessing the GCRs of a powered down cluster would fault.
+ */
+static bool cm3_pmu_cluster_powered(struct cm3_pmu *cm3_pmu)
+{
+ u32 stat;
+
+ /* No other cluster before CM 3.5, and ours is obviously powered */
+ if (cm3_pmu->cluster == cpu_cluster(&raw_current_cpu_data))
+ return true;
+
+ if (!mips_cpc_present())
+ return false;
+
+ mips_cm_lock_other(cm3_pmu->cluster, CM_GCR_Cx_OTHER_CORE_CM, 0,
+ CM_GCR_Cx_OTHER_BLOCK_LOCAL);
+ stat = read_cpc_co_stat_conf();
+ mips_cm_unlock_other();
+
+ return FIELD_GET(CPC_Cx_STAT_CONF_SEQSTATE, stat) ==
+ CPC_Cx_STAT_CONF_SEQSTATE_U5;
+}
+
+static u32 cm3_pmu_counton_bit(int idx)
+{
+ switch (idx) {
+ case 0:
+ return CM_GCR_DB_PC_CTL_P0_COUNTON;
+ case 1:
+ return CM_GCR_DB_PC_CTL_P1_COUNTON;
+ default:
+ return CM_GCR_DB_PC_CTL_CYCL_CNT_COUNTON;
+ }
+}
+
+static u32 cm3_pmu_reset_bit(int idx)
+{
+ switch (idx) {
+ case 0:
+ return CM_GCR_DB_PC_CTL_P0_RESET;
+ case 1:
+ return CM_GCR_DB_PC_CTL_P1_RESET;
+ default:
+ return CM_GCR_DB_PC_CTL_CYCL_CNT_RESET;
+ }
+}
+
+static u32 cm3_pmu_overflow_bit(int idx)
+{
+ switch (idx) {
+ case 0:
+ return CM_GCR_DB_PC_OV_P0_OF;
+ case 1:
+ return CM_GCR_DB_PC_OV_P1_OF;
+ default:
+ return CM_GCR_DB_PC_OV_CYCL_CNT_OF;
+ }
+}
+
+static u32 cm3_pmu_read_counter(struct cm3_pmu *cm3_pmu, int idx)
+{
+ switch (idx) {
+ case 0:
+ return CM3_PMU_READ(cm3_pmu, pc_cnt0);
+ case 1:
+ return CM3_PMU_READ(cm3_pmu, pc_cnt1);
+ default:
+ return CM3_PMU_READ(cm3_pmu, pc_cycl);
+ }
+}
+
+static void cm3_pmu_program_counter(struct cm3_pmu *cm3_pmu, int idx,
+ u32 event, u64 qual)
+{
+ u32 val = CM3_PMU_READ(cm3_pmu, pc_event);
+
+ if (idx == 0) {
+ val &= ~CM_GCR_DB_PC_EVENT_P0_EVENT;
+ val |= FIELD_PREP(CM_GCR_DB_PC_EVENT_P0_EVENT, event);
+ CM3_PMU_WRITE(cm3_pmu, pc_event, val);
+ CM3_PMU_WRITE(cm3_pmu, pc_qual0, qual);
+ } else {
+ val &= ~CM_GCR_DB_PC_EVENT_P1_EVENT;
+ val |= FIELD_PREP(CM_GCR_DB_PC_EVENT_P1_EVENT, event);
+ CM3_PMU_WRITE(cm3_pmu, pc_event, val);
+ CM3_PMU_WRITE(cm3_pmu, pc_qual1, qual);
+ }
+}
+
+static void cm3_pmu_set_ctl(struct cm3_pmu *cm3_pmu, u32 clear, u32 set)
+{
+ u32 val = CM3_PMU_READ(cm3_pmu, pc_ctl);
+
+ CM3_PMU_WRITE(cm3_pmu, pc_ctl, (val & ~clear) | set);
+}
+
+/* Stop everything, clear pending overflows, enable the interrupt if used */
+static void cm3_pmu_hw_reset(struct cm3_pmu *cm3_pmu)
+{
+ CM3_PMU_WRITE(cm3_pmu, pc_ctl,
+ cm3_pmu->irq > 0 ? CM_GCR_DB_PC_CTL_PERF_INT_EN : 0);
+ CM3_PMU_WRITE(cm3_pmu, pc_event, 0);
+ CM3_PMU_WRITE(cm3_pmu, pc_ov, CM_GCR_DB_PC_OV_P1_OF |
+ CM_GCR_DB_PC_OV_P0_OF |
+ CM_GCR_DB_PC_OV_CYCL_CNT_OF);
+}
+
+/*
+ * Bring the counters of the cluster to a known state the first time they
+ * are needed. The CM may not have been powered at probe time.
+ */
+static int cm3_pmu_hw_init(struct cm3_pmu *cm3_pmu)
+{
+ guard(mutex)(&cm3_pmu->init_lock);
+
+ if (cm3_pmu->initialized)
+ return 0;
+
+ if (!cm3_pmu_cluster_powered(cm3_pmu))
+ return -ENODEV;
+
+ cm3_pmu->num_pcs = FIELD_GET(CM_GCR_DB_PC_CTL_PERF_NUM_CNT,
+ CM3_PMU_READ(cm3_pmu, pc_ctl));
+ cm3_pmu->num_pcs = min_t(unsigned int, cm3_pmu->num_pcs,
+ CM3_PMU_MAX_PCS);
+ cm3_pmu_hw_reset(cm3_pmu);
+ cm3_pmu->initialized = true;
+
+ return 0;
+}
+
+static u64 cm3_pmu_default_qual(u32 event)
+{
+ switch (event) {
+ case CM3_PMU_EV_CM_REQ:
+ return CM3_PMU_Q_CMREQ_DEFAULT;
+ case CM3_PMU_EV_IO_REQ:
+ return CM3_PMU_Q_IO_DEFAULT;
+ case CM3_PMU_EV_MEM_REQ:
+ return CM3_PMU_Q_MEM_DEFAULT;
+ case CM3_PMU_EV_MEM_AXI:
+ case CM3_PMU_EV_IOCU_AXI(0) ... CM3_PMU_EV_IOCU_AXI_LAST:
+ return CM3_PMU_Q_AXI_DEFAULT;
+ case CM3_PMU_EV_PDTRACE_DROPPED:
+ return CM3_PMU_Q_PDTRACE_DEFAULT;
+ default:
+ /* All qualifier groups set to "don't care" */
+ return GENMASK_ULL(63, 0);
+ }
+}
+
+/* perf callbacks */
+
+static void cm3_pmu_update(struct perf_event *event)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ u64 prev, new;
+
+ do {
+ prev = local64_read(&hwc->prev_count);
+ new = cm3_pmu_read_counter(cm3_pmu, hwc->idx);
+ } while (local64_cmpxchg(&hwc->prev_count, prev, new) != prev);
+
+ local64_add((new - prev) & CM3_PMU_COUNTER_MASK, &event->count);
+}
+
+static void cm3_pmu_start(struct perf_event *event, int flags)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx = hwc->idx;
+
+ if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
+ return;
+
+ hwc->state = 0;
+
+ if (idx != CM3_PMU_IDX_CYCLES)
+ cm3_pmu_program_counter(cm3_pmu, idx, hwc->config,
+ hwc->config_base);
+
+ /*
+ * Setting the reset bit together with the count-on bit resets the
+ * counter and its overflow status before counting is started.
+ */
+ local64_set(&hwc->prev_count, 0);
+ cm3_pmu_set_ctl(cm3_pmu, 0,
+ cm3_pmu_reset_bit(idx) | cm3_pmu_counton_bit(idx));
+
+ perf_event_update_userpage(event);
+}
+
+static void cm3_pmu_stop(struct perf_event *event, int flags)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+
+ if (hwc->state & PERF_HES_STOPPED)
+ return;
+
+ cm3_pmu_set_ctl(cm3_pmu, cm3_pmu_counton_bit(hwc->idx), 0);
+
+ cm3_pmu_update(event);
+ hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE;
+}
+
+static int cm3_pmu_add(struct perf_event *event, int flags)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ int idx;
+
+ if (hwc->config == CM3_PMU_EV_CYCLES) {
+ idx = CM3_PMU_IDX_CYCLES;
+ if (cm3_pmu->events[idx])
+ return -EAGAIN;
+ } else {
+ for (idx = 0; idx < cm3_pmu->num_pcs; idx++)
+ if (!cm3_pmu->events[idx])
+ break;
+ if (idx == cm3_pmu->num_pcs)
+ return -EAGAIN;
+ }
+
+ hwc->idx = idx;
+ hwc->state = PERF_HES_STOPPED | PERF_HES_UPTODATE;
+ cm3_pmu->events[idx] = event;
+
+ if (!cm3_pmu->num_active++ && cm3_pmu->irq <= 0)
+ hrtimer_start(&cm3_pmu->hrtimer,
+ ns_to_ktime(CM3_PMU_POLL_PERIOD_NS),
+ HRTIMER_MODE_REL_PINNED);
+
+ if (flags & PERF_EF_START)
+ cm3_pmu_start(event, PERF_EF_RELOAD);
+
+ perf_event_update_userpage(event);
+
+ return 0;
+}
+
+static void cm3_pmu_del(struct perf_event *event, int flags)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+
+ cm3_pmu_stop(event, PERF_EF_UPDATE);
+
+ cm3_pmu->events[hwc->idx] = NULL;
+ if (!--cm3_pmu->num_active && cm3_pmu->irq <= 0)
+ hrtimer_cancel(&cm3_pmu->hrtimer);
+
+ perf_event_update_userpage(event);
+}
+
+static void cm3_pmu_count_group_event(struct perf_event *event,
+ struct perf_event *member,
+ unsigned int *cycles, unsigned int *pcs)
+{
+ if (member->pmu != event->pmu)
+ return;
+
+ if (member->hw.config == CM3_PMU_EV_CYCLES)
+ (*cycles)++;
+ else
+ (*pcs)++;
+}
+
+static bool cm3_pmu_validate_group(struct perf_event *event)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct perf_event *leader = event->group_leader;
+ struct perf_event *sibling;
+ unsigned int cycles = 0, pcs = 0;
+
+ /* Hardware events of the group must all be ours */
+ if (leader->pmu != event->pmu && !is_software_event(leader))
+ return false;
+
+ cm3_pmu_count_group_event(event, event, &cycles, &pcs);
+ if (leader != event)
+ cm3_pmu_count_group_event(event, leader, &cycles, &pcs);
+
+ for_each_sibling_event(sibling, leader) {
+ if (sibling->pmu != event->pmu && !is_software_event(sibling))
+ return false;
+
+ cm3_pmu_count_group_event(event, sibling, &cycles, &pcs);
+ }
+
+ return cycles <= 1 && pcs <= cm3_pmu->num_pcs;
+}
+
+static int cm3_pmu_event_init(struct perf_event *event)
+{
+ struct cm3_pmu *cm3_pmu = to_cm3_pmu(event->pmu);
+ struct hw_perf_event *hwc = &event->hw;
+ u64 config = event->attr.config;
+ int ret;
+
+ if (event->attr.type != event->pmu->type)
+ return -ENOENT;
+
+ /* Counters are shared by all CPUs: no sampling, no per-task events */
+ if (is_sampling_event(event) ||
+ event->attach_state & PERF_ATTACH_TASK)
+ return -EOPNOTSUPP;
+
+ if (event->cpu < 0)
+ return -EINVAL;
+
+ if (config & ~CM3_PMU_EV_MASK)
+ return -EINVAL;
+
+ if (config != CM3_PMU_EV_CYCLES &&
+ (config & ~CM3_PMU_EV_HW_MASK || config == CM3_PMU_EV_NONE))
+ return -EINVAL;
+
+ /* The cluster's CM must be powered for its counters to be usable */
+ ret = cm3_pmu_hw_init(cm3_pmu);
+ if (ret)
+ return ret;
+
+ if (!cm3_pmu_validate_group(event))
+ return -EINVAL;
+
+ hwc->idx = -1;
+ hwc->config = config;
+ hwc->config_base = event->attr.config1 ?: cm3_pmu_default_qual(config);
+ event->cpu = cm3_pmu->on_cpu;
+
+ return 0;
+}
+
+static enum hrtimer_restart cm3_pmu_poll(struct hrtimer *hrtimer)
+{
+ struct cm3_pmu *cm3_pmu = container_of(hrtimer, struct cm3_pmu,
+ hrtimer);
+ int idx;
+
+ for (idx = 0; idx < CM3_PMU_NUM_COUNTERS; idx++) {
+ struct perf_event *event = cm3_pmu->events[idx];
+
+ if (event && !(event->hw.state & PERF_HES_STOPPED))
+ cm3_pmu_update(event);
+ }
+
+ hrtimer_forward_now(hrtimer, ns_to_ktime(CM3_PMU_POLL_PERIOD_NS));
+
+ return HRTIMER_RESTART;
+}
+
+static irqreturn_t cm3_pmu_handle_irq(int irq, void *data)
+{
+ struct cm3_pmu *cm3_pmu = data;
+ u32 ov = CM3_PMU_READ(cm3_pmu, pc_ov);
+ int idx;
+
+ if (!ov)
+ return IRQ_NONE;
+
+ /* Status bits are write-1-to-clear, this deasserts the interrupt */
+ CM3_PMU_WRITE(cm3_pmu, pc_ov, ov);
+
+ for (idx = 0; idx < CM3_PMU_NUM_COUNTERS; idx++) {
+ struct perf_event *event = cm3_pmu->events[idx];
+
+ if (event && ov & cm3_pmu_overflow_bit(idx))
+ cm3_pmu_update(event);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* CPU hotplug */
+
+static int cm3_pmu_online_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct cm3_pmu *cm3_pmu = hlist_entry_safe(node, struct cm3_pmu, node);
+
+ if (cm3_pmu->on_cpu >= 0)
+ return 0;
+
+ cm3_pmu->on_cpu = cpu;
+ if (cm3_pmu->irq > 0)
+ WARN_ON(irq_set_affinity(cm3_pmu->irq, cpumask_of(cpu)));
+
+ return 0;
+}
+
+static int cm3_pmu_offline_cpu(unsigned int cpu, struct hlist_node *node)
+{
+ struct cm3_pmu *cm3_pmu = hlist_entry_safe(node, struct cm3_pmu, node);
+ unsigned int target;
+
+ if (cm3_pmu->on_cpu != cpu)
+ return 0;
+
+ target = cpumask_any_but(cpu_online_mask, cpu);
+ if (target >= nr_cpu_ids) {
+ cm3_pmu->on_cpu = -1;
+ return 0;
+ }
+
+ perf_pmu_migrate_context(&cm3_pmu->pmu, cpu, target);
+ cm3_pmu->on_cpu = target;
+ if (cm3_pmu->irq > 0)
+ WARN_ON(irq_set_affinity(cm3_pmu->irq, cpumask_of(target)));
+
+ return 0;
+}
+
+/* Platform driver */
+
+static void cm3_pmu_unregister(struct cm3_pmu *cm3_pmu)
+{
+ perf_pmu_unregister(&cm3_pmu->pmu);
+ cpuhp_state_remove_instance(cm3_pmu_cpuhp_state, &cm3_pmu->node);
+ hrtimer_cancel(&cm3_pmu->hrtimer);
+ if (cm3_pmu->initialized && cm3_pmu_cluster_powered(cm3_pmu)) {
+ cm3_pmu->irq = 0;
+ cm3_pmu_hw_reset(cm3_pmu);
+ }
+}
+
+static int cm3_pmu_register(struct device *dev, struct cm3_pmu *cm3_pmu)
+{
+ const char *name;
+ int ret;
+
+ cm3_pmu->on_cpu = -1;
+ cm3_pmu->num_pcs = CM3_PMU_MAX_PCS;
+ mutex_init(&cm3_pmu->init_lock);
+ hrtimer_setup(&cm3_pmu->hrtimer, cm3_pmu_poll, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+
+ cm3_pmu->pmu = (struct pmu) {
+ .parent = dev,
+ .module = THIS_MODULE,
+ .task_ctx_nr = perf_invalid_context,
+ .capabilities = PERF_PMU_CAP_NO_EXCLUDE |
+ PERF_PMU_CAP_NO_INTERRUPT,
+ .event_init = cm3_pmu_event_init,
+ .add = cm3_pmu_add,
+ .del = cm3_pmu_del,
+ .start = cm3_pmu_start,
+ .stop = cm3_pmu_stop,
+ .read = cm3_pmu_update,
+ .attr_groups = cm3_pmu_attr_groups,
+ };
+
+ name = devm_kasprintf(dev, GFP_KERNEL, CM3_PMU_NAME "_%u",
+ cm3_pmu->cluster);
+ if (!name)
+ return -ENOMEM;
+
+ /* Initialize now if the cluster is powered, else on first use */
+ ret = cm3_pmu_hw_init(cm3_pmu);
+ if (ret && ret != -ENODEV)
+ return ret;
+
+ ret = cpuhp_state_add_instance(cm3_pmu_cpuhp_state, &cm3_pmu->node);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to register hotplug\n");
+
+ ret = perf_pmu_register(&cm3_pmu->pmu, name, -1);
+ if (ret) {
+ cpuhp_state_remove_instance(cm3_pmu_cpuhp_state, &cm3_pmu->node);
+ return dev_err_probe(dev, ret, "failed to register %s\n", name);
+ }
+
+ if (cm3_pmu->initialized)
+ dev_info(dev, "%s: %u counters + cycles, %s\n", name,
+ cm3_pmu->num_pcs,
+ cm3_pmu->irq > 0 ? "overflow interrupt" : "polled");
+ else
+ dev_info(dev, "%s: cluster powered down\n", name);
+
+ return 0;
+}
+
+static int cm3_pmu_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct cm3_pmu_dev *cm3_dev;
+ unsigned int nclusters, i;
+ int irq, ret;
+
+ if (!mips_cm_present() || mips_cm_revision() < CM_REV_CM3)
+ return -ENODEV;
+
+ /* One PMU per cluster, whether or not it holds CPUs */
+ nclusters = mips_cps_numclusters();
+ cm3_dev = devm_kzalloc(dev, struct_size(cm3_dev, pmus, nclusters),
+ GFP_KERNEL);
+ if (!cm3_dev)
+ return -ENOMEM;
+
+ cm3_dev->num_pmus = nclusters;
+ for (i = 0; i < nclusters; i++)
+ cm3_dev->pmus[i].cluster = i;
+
+ /*
+ * Each cluster's CM raises its own overflow interrupt into its own
+ * GIC, and a GIC shared interrupt is routed to a single cluster at a
+ * time (the one of its affinity CPU). The interrupt can therefore
+ * only serve the cluster the CPUs live in, and only if they all live
+ * in that one cluster so that any CPU may handle it. Other clusters
+ * are polled.
+ */
+ irq = platform_get_irq_optional(pdev, 0);
+ if (irq > 0 && !mips_cps_multicluster_cpus()) {
+ struct cm3_pmu *cm3_pmu;
+
+ cm3_pmu = &cm3_dev->pmus[cpu_cluster(&boot_cpu_data)];
+ ret = devm_request_irq(dev, irq, cm3_pmu_handle_irq,
+ IRQF_NOBALANCING | IRQF_NO_THREAD,
+ CM3_PMU_NAME, cm3_pmu);
+ if (ret)
+ return dev_err_probe(dev, ret, "failed to request IRQ\n");
+
+ cm3_pmu->irq = irq;
+ } else if (irq > 0) {
+ dev_info(dev, "CPUs in several clusters: overflow interrupt not usable, polling\n");
+ } else if (irq != -ENXIO) {
+ return dev_err_probe(dev, irq, "failed to get IRQ\n");
+ }
+
+ for (i = 0; i < cm3_dev->num_pmus; i++) {
+ ret = cm3_pmu_register(dev, &cm3_dev->pmus[i]);
+ if (ret)
+ goto err_unregister;
+ }
+
+ platform_set_drvdata(pdev, cm3_dev);
+
+ return 0;
+
+err_unregister:
+ while (i--)
+ cm3_pmu_unregister(&cm3_dev->pmus[i]);
+
+ return ret;
+}
+
+static void cm3_pmu_remove(struct platform_device *pdev)
+{
+ struct cm3_pmu_dev *cm3_dev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ for (i = 0; i < cm3_dev->num_pmus; i++)
+ cm3_pmu_unregister(&cm3_dev->pmus[i]);
+}
+
+static const struct of_device_id cm3_pmu_of_match[] = {
+ { .compatible = "mti,mips-cm" },
+ { .compatible = "mobileye,eyeq6-cm" },
+ {}
+};
+MODULE_DEVICE_TABLE(of, cm3_pmu_of_match);
+
+static struct platform_driver cm3_pmu_driver = {
+ .driver = {
+ .name = CM3_PMU_NAME,
+ .of_match_table = cm3_pmu_of_match,
+ .suppress_bind_attrs = true,
+ },
+ .probe = cm3_pmu_probe,
+ .remove = cm3_pmu_remove,
+};
+
+static int __init cm3_pmu_init(void)
+{
+ int ret;
+
+ ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN,
+ "perf/mips/cm3_pmu:online",
+ cm3_pmu_online_cpu, cm3_pmu_offline_cpu);
+ if (ret < 0)
+ return ret;
+
+ cm3_pmu_cpuhp_state = ret;
+
+ ret = platform_driver_register(&cm3_pmu_driver);
+ if (ret)
+ cpuhp_remove_multi_state(cm3_pmu_cpuhp_state);
+
+ return ret;
+}
+module_init(cm3_pmu_init);
+
+static void __exit cm3_pmu_exit(void)
+{
+ platform_driver_unregister(&cm3_pmu_driver);
+ cpuhp_remove_multi_state(cm3_pmu_cpuhp_state);
+}
+module_exit(cm3_pmu_exit);
+
+MODULE_DESCRIPTION("MIPS Coherence Manager 3 (CM3) PMU driver");
+MODULE_LICENSE("GPL");
--
2.43.0