Re: [PATCH v10 7/9] perf: RISC-V: add support for SSE event

[Himanshu Chauhan]

Hi Zhanpeng Zhang,

On Fri, Sep 11, 2026 at 08:14:54PM +0800, Zhanpeng Zhang wrote:
> From: Clément Léger <cleger@xxxxxxxxxxxx>
> 
> Register a handler for the local PMU overflow SSE event so that RISC-V
> perf can receive overflows even when normal S-mode interrupts are masked.
> Reuse the existing overflow handler and pass it the interrupted pt_regs
> rebuilt by the architecture SSE entry path.
> 
> Select the delivery mechanism once during PMU probe. Prefer SSE when its
> event can be registered and enabled. If the extension or PMU event is
> explicitly unsupported, use the ordinary PMU interrupt. Do not enable the
> interrupt after any other SSE setup failure or when a crash kernel may
> have inherited firmware state. Install the PMU enable and disable
> callbacks only after SSE delivery becomes active.
> 
> Keep the local PMU SSE event disabled across CPU power management. On
> entry, the generic SSE notifier masks the hart before the lower-priority
> PMU notifier disables the event and stops the counters. On exit, the SSE
> notifier first unmasks the hart while the event remains disabled. The PMU
> notifier then restores counters and event userpage state before enabling
> the event. An unmask failure stops the notifier chain and leaves the
> counters stopped. Ordinary PMU interrupts retain their existing notifier
> ordering.
> 
> An SSE overflow can arrive as soon as the event is enabled during probe.
> Publish the counter mask before SSE setup, so an early handler can stop
> the counter source even before perf starts admitting normal samples.
> After a real overflow, restart only counters whose perf state is still
> running. Honor a non-zero return from perf_event_overflow() and leave
> throttled events stopped.
> 
> Guest attribution is not part of this version. Detect an interrupted guest
> from hstatus.SPV and skip its sample while still updating the period and
> counter state, rather than exposing guest state as a host sample.
> 
> The perf PMU callbacks cannot return errors. If an SSE transition or
> interrupted-context read fails, latch the failure per CPU and stop its
> mapped events through the normal perf state transitions. Do not reset the
> firmware counter mapping behind perf, restart a failed event, or attempt a
> runtime switch to IRQ delivery.
> 
> During cleanup, close callback admission and synchronously drain each CPU
> before disabling and unregistering the SSE event. If firmware refuses the
> cleanup, stop the counter source and transfer the event to the SSE core so
> later CPU hotplug or shutdown processing can retry without using freed PMU
> callback state.
> 
> Signed-off-by: Clément Léger <cleger@xxxxxxxxxxxx>
> Co-developed-by: Himanshu Chauhan <himanshu.chauhan@xxxxxxxxxxxxxxxx>
> Signed-off-by: Himanshu Chauhan <himanshu.chauhan@xxxxxxxxxxxxxxxx>
> Co-developed-by: Zhanpeng Zhang <zhangzhanpeng.jasper@xxxxxxxxxxxxx>
> Signed-off-by: Zhanpeng Zhang <zhangzhanpeng.jasper@xxxxxxxxxxxxx>
> ---
>  Documentation/arch/riscv/index.rst     |   1 +
>  Documentation/arch/riscv/pmu-sse.rst   |  55 +++
>  MAINTAINERS                            |   1 +
>  drivers/firmware/riscv/riscv_sbi_sse.c |   9 +
>  drivers/perf/Kconfig                   |  11 +
>  drivers/perf/riscv_pmu.c               |  14 +-
>  drivers/perf/riscv_pmu_sbi.c           | 544 ++++++++++++++++++++-----
>  include/linux/perf/riscv_pmu.h         |  20 +-
>  include/linux/riscv_sbi_sse.h          |   6 +
>  9 files changed, 566 insertions(+), 95 deletions(-)
>  create mode 100644 Documentation/arch/riscv/pmu-sse.rst
>

Thanks for sending v10. I was waiting on your series so that I can send the RAS series
based on it. I was trying to apply the series on v7.3-rc3 but this particular patch fails.
I used b4 shazam to pull the series.

Could you once check?

Thanks
Regards
Himanshu
 
> diff --git a/Documentation/arch/riscv/index.rst b/Documentation/arch/riscv/index.rst
> index ac535c52d509..5cb909c83a88 100644
> --- a/Documentation/arch/riscv/index.rst
> +++ b/Documentation/arch/riscv/index.rst
> @@ -11,6 +11,7 @@ RISC-V architecture
>      vm-layout
>      hwprobe
>      patch-acceptance
> +    pmu-sse
>      uabi
>      vector
>      cmodx
> diff --git a/Documentation/arch/riscv/pmu-sse.rst b/Documentation/arch/riscv/pmu-sse.rst
> new file mode 100644
> index 000000000000..b7458a9d0116
> --- /dev/null
> +++ b/Documentation/arch/riscv/pmu-sse.rst
> @@ -0,0 +1,55 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +========================================
> +RISC-V PMU overflow delivery through SSE
> +========================================
> +
> +When ``CONFIG_RISCV_PMU_SBI_SSE`` is enabled and firmware provides the local
> +PMU overflow event, the RISC-V SBI PMU driver uses Supervisor Software Events
> +(SSE) to deliver counter overflows.
> +
> +Delivery selection
> +==================
> +
> +The delivery mechanism is selected once while the PMU device is probed.  The
> +driver first tries to register and enable the local PMU overflow SSE event.  It
> +tries the ordinary PMU interrupt only when the SSE extension or the local PMU
> +event is explicitly unsupported.  It does not change the delivery mechanism
> +after the PMU has been registered.
> +
> +Other SSE setup errors do not prove that firmware released the overflow route,
> +so the driver does not enable the ordinary PMU interrupt.  The PMU remains
> +available for counting but not sampling.  If setup retained an SSE event, the
> +PMU keeps quiescing that event around perf scheduling changes without rearming
> +it.  This preserves callback ownership without creating two possible delivery
> +mechanisms for the same hardware overflow.
> +
> +Interrupted context
> +===================
> +
> +SSE enters Linux through a supervisor handler context constructed by firmware.
> +The entry contract preserves the interrupted GPRs except ``a6`` and ``a7``,
> +which Linux reads from the interrupted-register event attributes.  Linux then
> +reconstructs the interrupted ``pt_regs`` and publishes it while dispatching the
> +event.
> +
> +Perf uses that context for register samples and callchains.  Kernel stack
> +walking verifies that the interrupted frame belongs to the current task stack
> +before dereferencing it.  For sensitive entry paths and IRQ stacks, Linux
> +records the interrupted PC without walking a stack whose bounds cannot be
> +proved.  User callchains use the existing no-fault user unwinder.  A DWARF
> +raw user-stack copy from an SSE handler must not take a page fault, so it
> +walks the current task's page tables with fast-only GUP, copies each resident
> +page through its kernel mapping, and stops at the first non-resident page,
> +preserving perf's truncated-user-stack semantics.
> +
> +CPU power management
> +====================
> +
> +The PMU and SSE CPU power-management callbacks are ordered according to the
> +selected delivery mechanism.  With SSE delivery, SSE events are masked before
> +the lower-priority PMU callback disables the local PMU event and stops the
> +counters on entry.  On exit, the hart is unmasked while the local PMU event is
> +still disabled.  The PMU callback then restores the counters and enables the
> +event, so an unmask failure leaves the counters stopped.  The ordinary
> +interrupt path retains the existing PMU ordering.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index d11cd44d2449..91b4e32a1fb2 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -23392,6 +23392,7 @@ M:	Atish Patra <atish.patra@xxxxxxxxx>
>  R:	Anup Patel <anup@xxxxxxxxxxxxxx>
>  L:	linux-riscv@xxxxxxxxxxxxxxxxxxx
>  S:	Supported
> +F:	Documentation/arch/riscv/pmu-sse.rst
>  F:	drivers/perf/riscv_pmu.c
>  F:	drivers/perf/riscv_pmu_legacy.c
>  F:	drivers/perf/riscv_pmu_sbi.c
> diff --git a/drivers/firmware/riscv/riscv_sbi_sse.c b/drivers/firmware/riscv/riscv_sbi_sse.c
> index e81cfb4b0c56..c7d2a2def00a 100644
> --- a/drivers/firmware/riscv/riscv_sbi_sse.c
> +++ b/drivers/firmware/riscv/riscv_sbi_sse.c
> @@ -161,6 +161,15 @@ static int sse_sbi_event_func(struct sse_event *event, unsigned long func)
>  	return 0;
>  }
>  
> +bool sse_event_is_enabled_local(struct sse_event *event)
> +{
> +	if (!sse_event_is_global(event->evt_id))
> +		lockdep_assert_preemption_disabled();
> +
> +	return sse_get_reg_evt(event)->is_enabled;
> +}
> +EXPORT_SYMBOL_GPL(sse_event_is_enabled_local);
> +
>  int sse_event_disable_local(struct sse_event *event)
>  {
>  	if (!sse_event_is_global(event->evt_id))
> diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig
> index 245e7bb763b9..10b047856121 100644
> --- a/drivers/perf/Kconfig
> +++ b/drivers/perf/Kconfig
> @@ -105,6 +105,17 @@ config RISCV_PMU_SBI
>  	  full perf feature support i.e. counter overflow, privilege mode
>  	  filtering, counter configuration.
>  
> +config RISCV_PMU_SBI_SSE
> +	depends on RISCV_PMU_SBI && RISCV_SBI_SSE
> +	bool "RISC-V PMU SSE events"
> +	default n
> +	help
> +	  Say y if you want to use SSE events to deliver PMU interrupts. This
> +	  provides a way to profile the kernel at any level by using NMI-like
> +	  SSE events. SSE events being really intrusive, this option allows
> +	  to select it only if needed. See Documentation/arch/riscv/pmu-sse.rst
> +	  for interrupted-context limitations.
> +
>  config STARFIVE_STARLINK_PMU
>  	depends on ARCH_STARFIVE || COMPILE_TEST
>  	depends on 64BIT
> diff --git a/drivers/perf/riscv_pmu.c b/drivers/perf/riscv_pmu.c
> index 8e3cd0f35336..1f65c640b693 100644
> --- a/drivers/perf/riscv_pmu.c
> +++ b/drivers/perf/riscv_pmu.c
> @@ -13,6 +13,7 @@
>  #include <linux/irqdesc.h>
>  #include <linux/perf/riscv_pmu.h>
>  #include <linux/printk.h>
> +#include <linux/riscv_sbi_sse.h>
>  #include <linux/smp.h>
>  #include <linux/sched_clock.h>
>  
> @@ -247,6 +248,11 @@ void riscv_pmu_start(struct perf_event *event, int flags)
>  	if (flags & PERF_EF_RELOAD)
>  		WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE));
>  
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	if (unlikely(this_cpu_ptr(rvpmu->hw_events)->sse_failed))
> +		return;
> +#endif
> +
>  	hwc->state = 0;
>  	riscv_pmu_event_set_period(event);
>  	init_val = local64_read(&hwc->prev_count) & max_period;
> @@ -306,6 +312,7 @@ static int riscv_pmu_event_init(struct perf_event *event)
>  	struct hw_perf_event *hwc = &event->hw;
>  	struct riscv_pmu *rvpmu = to_riscv_pmu(event->pmu);
>  	int mapped_event;
> +	int ret;
>  	u64 event_config = 0;
>  	uint64_t cmask;
>  
> @@ -331,8 +338,11 @@ static int riscv_pmu_event_init(struct perf_event *event)
>  	hwc->idx = -1;
>  	hwc->event_base = mapped_event;
>  
> -	if (rvpmu->event_init)
> -		rvpmu->event_init(event);
> +	if (rvpmu->event_init) {
> +		ret = rvpmu->event_init(event);
> +		if (ret)
> +			return ret;
> +	}
>  
>  	if (!is_sampling_event(event)) {
>  		/*
> diff --git a/drivers/perf/riscv_pmu_sbi.c b/drivers/perf/riscv_pmu_sbi.c
> index 50220f7b46d9..7e971b47730c 100644
> --- a/drivers/perf/riscv_pmu_sbi.c
> +++ b/drivers/perf/riscv_pmu_sbi.c
> @@ -16,13 +16,16 @@
>  #include <linux/irqdomain.h>
>  #include <linux/of_irq.h>
>  #include <linux/of.h>
> +#include <linux/riscv_sbi_sse.h>
>  #include <linux/cpu_pm.h>
> +#include <linux/crash_dump.h>
>  #include <linux/sched/clock.h>
>  #include <linux/soc/andes/irq.h>
>  #include <linux/workqueue.h>
>  
>  #include <asm/errata_list.h>
>  #include <asm/sbi.h>
> +#include <asm/sse.h>
>  #include <asm/cpufeature.h>
>  #include <asm/vendor_extensions.h>
>  #include <asm/vendor_extensions/andes.h>
> @@ -95,7 +98,6 @@ static bool riscv_pmu_use_irq;
>  static unsigned int riscv_pmu_irq_num;
>  static unsigned int riscv_pmu_irq_mask;
>  static unsigned int riscv_pmu_irq;
> -
>  /* Cache the available counters in a bitmask */
>  static unsigned long cmask;
>  
> @@ -896,14 +898,24 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask)
>  	return 0;
>  }
>  
> -static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu)
> +static inline void pmu_sbi_stop_all_mask(unsigned long ctr_mask)
>  {
>  	/*
>  	 * No need to check the error because we are disabling all the counters
>  	 * which may include counters that are not enabled yet.
>  	 */
>  	sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP,
> -		  0, pmu->cmask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0);
> +		  0, ctr_mask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0);
> +}
> +
> +static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu)
> +{
> +	pmu_sbi_stop_all_mask(pmu->cmask);
> +}
> +
> +static void pmu_sbi_stop_all_cpu(void *info)
> +{
> +	pmu_sbi_stop_all(info);
>  }
>  
>  static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu)
> @@ -953,7 +965,7 @@ static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu)
>  static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt,
>  					      u64 ctr_ovf_mask)
>  {
> -	int idx = 0, i;
> +	int idx, i;
>  	struct perf_event *event;
>  	unsigned long flag = SBI_PMU_START_FLAG_SET_INIT_VALUE;
>  	unsigned long ctr_start_mask = 0;
> @@ -962,7 +974,19 @@ static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt,
>  	u64 init_val = 0;
>  
>  	for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) {
> -		ctr_start_mask = cpu_hw_evt->used_hw_ctrs[i] & ~ctr_ovf_mask;
> +		unsigned long ctr_ovf_mask_word;
> +		int lidx;
> +
> +		ctr_ovf_mask_word = ctr_ovf_mask >> (i * BITS_PER_LONG);
> +		ctr_start_mask = 0;
> +		for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG) {
> +			lidx = idx + i * BITS_PER_LONG;
> +			event = cpu_hw_evt->events[lidx];
> +			if (event && !(event->hw.state & PERF_HES_STOPPED))
> +				ctr_start_mask |= BIT(idx);
> +		}
> +		ctr_start_mask &= ~ctr_ovf_mask_word;
> +
>  		/* Start all the counters that did not overflow in a single shot */
>  		if (ctr_start_mask) {
>  			sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, i * BITS_PER_LONG,
> @@ -971,23 +995,25 @@ static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt,
>  	}
>  
>  	/* Reinitialize and start all the counter that overflowed */
> -	while (ctr_ovf_mask) {
> -		if (ctr_ovf_mask & 0x01) {
> -			event = cpu_hw_evt->events[idx];
> -			hwc = &event->hw;
> -			max_period = riscv_pmu_ctr_get_width_mask(event);
> -			init_val = local64_read(&hwc->prev_count) & max_period;
> +	for (idx = 0; idx < RISCV_MAX_COUNTERS; idx++) {
> +		if (!(ctr_ovf_mask & BIT_ULL(idx)))
> +			continue;
> +
> +		event = cpu_hw_evt->events[idx];
> +		if (!event || event->hw.state & PERF_HES_STOPPED)
> +			continue;
> +
> +		hwc = &event->hw;
> +		max_period = riscv_pmu_ctr_get_width_mask(event);
> +		init_val = local64_read(&hwc->prev_count) & max_period;
>  #if defined(CONFIG_32BIT)
> -			sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1,
> -				  flag, init_val, init_val >> 32, 0);
> +		sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1,
> +			  flag, init_val, init_val >> 32, 0);
>  #else
> -			sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1,
> -				  flag, init_val, 0, 0);
> +		sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1,
> +			  flag, init_val, 0, 0);
>  #endif
> -			perf_event_update_userpage(event);
> -		}
> -		ctr_ovf_mask = ctr_ovf_mask >> 1;
> -		idx++;
> +		perf_event_update_userpage(event);
>  	}
>  }
>  
> @@ -1002,7 +1028,7 @@ static inline void pmu_sbi_start_ovf_ctrs_snapshot(struct cpu_hw_events *cpu_hw_
>  	struct riscv_pmu_snapshot_data *sdata = cpu_hw_evt->snapshot_addr;
>  
>  	for_each_set_bit(idx, cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS) {
> -		if (ctr_ovf_mask & BIT(idx)) {
> +		if (ctr_ovf_mask & BIT_ULL(idx)) {
>  			event = cpu_hw_evt->events[idx];
>  			hwc = &event->hw;
>  			max_period = riscv_pmu_ctr_get_width_mask(event);
> @@ -1016,31 +1042,111 @@ static inline void pmu_sbi_start_ovf_ctrs_snapshot(struct cpu_hw_events *cpu_hw_
>  	}
>  
>  	for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) {
> +		unsigned long ctr_start_mask = 0;
> +		int lidx;
> +
>  		/* Restore the counter values to relative indices for used hw counters */
> -		for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG)
> -			sdata->ctr_values[idx] =
> -					cpu_hw_evt->snapshot_cval_shcopy[idx + i * BITS_PER_LONG];
> +		for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG) {
> +			lidx = idx + i * BITS_PER_LONG;
> +			event = cpu_hw_evt->events[lidx];
> +			if (event && !(event->hw.state & PERF_HES_STOPPED))
> +				ctr_start_mask |= BIT(idx);
> +
> +			sdata->ctr_values[idx] = cpu_hw_evt->snapshot_cval_shcopy[lidx];
> +		}
> +
>  		/* Start all the counters in a single shot */
> -		sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx * BITS_PER_LONG,
> -			  cpu_hw_evt->used_hw_ctrs[i], flag, 0, 0, 0);
> +		if (ctr_start_mask)
> +			sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START,
> +				  i * BITS_PER_LONG, ctr_start_mask, flag, 0, 0, 0);
>  	}
>  }
>  
> +static bool pmu_sbi_sse_failed(struct cpu_hw_events *cpu_hw_evt)
> +{
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	return READ_ONCE(cpu_hw_evt->sse_failed);
> +#else
> +	return false;
> +#endif
> +}
> +
>  static void pmu_sbi_start_overflow_mask(struct riscv_pmu *pmu,
>  					u64 ctr_ovf_mask)
>  {
>  	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events);
>  
> +	if (unlikely(pmu_sbi_sse_failed(cpu_hw_evt)))
> +		return;
> +
>  	if (sbi_pmu_snapshot_available())
>  		pmu_sbi_start_ovf_ctrs_snapshot(cpu_hw_evt, ctr_ovf_mask);
>  	else
>  		pmu_sbi_start_ovf_ctrs_sbi(cpu_hw_evt, ctr_ovf_mask);
>  }
>  
> -static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +/*
> + * A local SSE delivery failure makes the current PMU state unsafe to resume.
> + * Latch the failure before stopping mapped events so the SSE transition and
> + * overflow restart paths cannot undo the fail-safe while they are quiesced.
> + */
> +static void pmu_sbi_fail_sse(struct riscv_pmu *pmu, const char *op, int ret)
> +{
> +	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events);
> +	struct perf_event *event;
> +	int idx;
> +
> +	if (READ_ONCE(cpu_hw_evt->sse_failed))
> +		return;
> +
> +	WRITE_ONCE(cpu_hw_evt->sse_failed, true);
> +	pr_err_ratelimited("failed to %s local PMU SSE event: %d; stopping counters\n",
> +			   op, ret);
> +
> +	for (idx = 0; idx < RISCV_MAX_COUNTERS; idx++) {
> +		event = cpu_hw_evt->events[idx];
> +		if (event)
> +			riscv_pmu_stop(event, PERF_EF_UPDATE);
> +	}
> +}
> +
> +static void pmu_sbi_sse_disable(struct pmu *pmu)
> +{
> +	struct riscv_pmu *rvpmu = to_riscv_pmu(pmu);
> +	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(rvpmu->hw_events);
> +	int ret;
> +
> +	if (!READ_ONCE(rvpmu->sse_active) ||
> +	    READ_ONCE(cpu_hw_evt->sse_failed))
> +		return;
> +
> +	ret = sse_event_disable_local(rvpmu->sse_evt);
> +	if (unlikely(ret))
> +		pmu_sbi_fail_sse(rvpmu, "disable", ret);
> +}
> +
> +static void pmu_sbi_sse_enable(struct pmu *pmu)
> +{
> +	struct riscv_pmu *rvpmu = to_riscv_pmu(pmu);
> +	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(rvpmu->hw_events);
> +	int ret;
> +
> +	if (!READ_ONCE(rvpmu->sse_active) ||
> +	    READ_ONCE(cpu_hw_evt->sse_failed))
> +		return;
> +
> +	ret = sse_event_enable_local(rvpmu->sse_evt);
> +	if (unlikely(ret))
> +		pmu_sbi_fail_sse(rvpmu, "enable", ret);
> +}
> +#endif
> +
> +static irqreturn_t pmu_sbi_ovf_handler(struct cpu_hw_events *cpu_hw_evt,
> +				       struct pt_regs *regs, bool from_sse,
> +				       bool from_guest)
>  {
>  	struct perf_sample_data data;
> -	struct pt_regs *regs;
>  	struct hw_perf_event *hw_evt;
>  	union sbi_pmu_ctr_info *info;
>  	int lidx, hidx, fidx;
> @@ -1048,28 +1154,38 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
>  	struct perf_event *event;
>  	u64 overflow;
>  	u64 overflowed_ctrs = 0;
> -	struct cpu_hw_events *cpu_hw_evt = dev;
>  	u64 start_clock = sched_clock();
>  	struct riscv_pmu_snapshot_data *sdata = cpu_hw_evt->snapshot_addr;
>  
>  	if (WARN_ON_ONCE(!cpu_hw_evt))
>  		return IRQ_NONE;
>  
> -	/* Firmware counter don't support overflow yet */
> +	/*
> +	 * SSE can arrive before perf installs an event. The early exits below
> +	 * must stop the PMU source before firmware completes the SSE.
> +	 */
>  	fidx = find_first_bit(cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS);
>  	if (fidx == RISCV_MAX_COUNTERS) {
> -		csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num));
> +		if (from_sse)
> +			pmu_sbi_stop_all_mask(cmask);
> +		else
> +			csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num));
>  		return IRQ_NONE;
>  	}
>  
>  	event = cpu_hw_evt->events[fidx];
>  	if (!event) {
> -		ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
> +		if (from_sse)
> +			pmu_sbi_stop_all_mask(cmask);
> +		else
> +			ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
>  		return IRQ_NONE;
>  	}
>  
>  	pmu = to_riscv_pmu(event->pmu);
>  	pmu_sbi_stop_hw_ctrs(pmu);
> +	if (unlikely(pmu_sbi_sse_failed(cpu_hw_evt)))
> +		return IRQ_NONE;
>  
>  	/* Overflow status register should only be read after counter are stopped */
>  	if (sbi_pmu_snapshot_available())
> @@ -1079,15 +1195,17 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
>  
>  	/*
>  	 * Overflow interrupt pending bit should only be cleared after stopping
> -	 * all the counters to avoid any race condition.
> +	 * all the counters to avoid any race condition. When using SSE,
> +	 * interrupt is cleared when stopping counters.
>  	 */
> -	ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
> +	if (!from_sse)
> +		ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
>  
>  	/* No overflow bit is set */
> -	if (!overflow)
> +	if (!overflow) {
> +		pmu_sbi_start_overflow_mask(pmu, 0);
>  		return IRQ_NONE;
> -
> -	regs = get_irq_regs();
> +	}
>  
>  	for_each_set_bit(lidx, cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS) {
>  		struct perf_event *event = cpu_hw_evt->events[lidx];
> @@ -1109,14 +1227,20 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
>  			hidx = info->csr - CSR_CYCLE;
>  
>  		/* check if the corresponding bit is set in scountovf or overflow mask in shmem */
> -		if (!(overflow & BIT(hidx)))
> +		if (!(overflow & BIT_ULL(hidx)))
>  			continue;
>  
> +#ifdef CONFIG_CPU_PM
> +		/* Do not let CPU-PM resume override this overflow decision. */
> +		if (from_sse)
> +			clear_bit(lidx, cpu_hw_evt->pm_resume_hw_ctrs);
> +#endif
> +
>  		/*
>  		 * Keep a track of overflowed counters so that they can be started
>  		 * with updated initial value.
>  		 */
> -		overflowed_ctrs |= BIT(lidx);
> +		overflowed_ctrs |= BIT_ULL(lidx);
>  		hw_evt = &event->hw;
>  		/* Update the event states here so that we know the state while reading */
>  		hw_evt->state |= PERF_HES_STOPPED;
> @@ -1124,6 +1248,14 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
>  		hw_evt->state |= PERF_HES_UPTODATE;
>  		perf_sample_data_init(&data, 0, hw_evt->last_period);
>  		if (riscv_pmu_event_set_period(event)) {
> +			int overflow_ret;
> +
> +			/* Guest attribution and guest stack sampling are not supported yet. */
> +			if (from_guest) {
> +				hw_evt->state = 0;
> +				continue;
> +			}
> +
>  			/*
>  			 * Unlike other ISAs, RISC-V don't have to disable interrupts
>  			 * to avoid throttling here. As per the specification, the
> @@ -1132,128 +1264,291 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
>  			 * TODO: We will need to stop the guest counters once
>  			 * virtualization support is added.
>  			 */
> -			perf_event_overflow(event, &data, regs);
> +			overflow_ret = perf_event_overflow(event, &data, regs);
> +			if (!overflow_ret)
> +				hw_evt->state = 0;
> +		} else {
> +			hw_evt->state = 0;
>  		}
> -		/* Reset the state as we are going to start the counter after the loop */
> -		hw_evt->state = 0;
>  	}
>  
>  	pmu_sbi_start_overflow_mask(pmu, overflowed_ctrs);
> +
>  	perf_sample_event_took(sched_clock() - start_clock);
>  
>  	return IRQ_HANDLED;
>  }
>  
> -static int pmu_sbi_starting_cpu(unsigned int cpu, struct hlist_node *node)
> +static irqreturn_t pmu_sbi_ovf_irq_handler(int irq, void *dev)
>  {
> -	struct riscv_pmu *pmu = hlist_entry_safe(node, struct riscv_pmu, node);
> -	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events);
> -
> -	/*
> -	 * We keep enabling userspace access to CYCLE, TIME and INSTRET via the
> -	 * legacy option but that will be removed in the future.
> -	 */
> -	if (sysctl_perf_user_access == SYSCTL_LEGACY)
> -		csr_write(CSR_SCOUNTEREN, 0x7);
> -	else
> -		csr_write(CSR_SCOUNTEREN, 0x2);
> +	return pmu_sbi_ovf_handler(dev, get_irq_regs(), false, false);
> +}
>  
> -	/* Stop all the counters so that they can be enabled from perf */
> -	pmu_sbi_stop_all(pmu);
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +static int pmu_sbi_ovf_sse_handler(u32 evt, void *arg, struct pt_regs *regs)
> +{
> +	const struct riscv_sse_interrupted_context *context;
> +	struct riscv_pmu *pmu = arg;
> +	struct cpu_hw_events *hw_event = raw_cpu_ptr(pmu->hw_events);
> +	bool from_guest;
> +
> +	if (unlikely(!READ_ONCE(pmu->sse_active))) {
> +		pmu_sbi_stop_all(pmu);
> +		return -EIO;
> +	}
>  
> -	if (riscv_pmu_use_irq) {
> -		cpu_hw_evt->irq = riscv_pmu_irq;
> -		ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
> -		enable_percpu_irq(riscv_pmu_irq, IRQ_TYPE_NONE);
> +	if (unlikely(!regs)) {
> +		pmu_sbi_fail_sse(pmu, "read interrupted context for", -EIO);
> +		return -EIO;
>  	}
>  
> -	if (sbi_pmu_snapshot_available())
> -		return pmu_sbi_snapshot_setup(pmu, cpu);
> +	context = riscv_sse_get_interrupted_context();
> +	from_guest = context && context->regs == regs &&
> +		     (context->hstatus & HSTATUS_SPV);
> +	pmu_sbi_ovf_handler(hw_event, regs, true, from_guest);
>  
>  	return 0;
>  }
>  
> -static int pmu_sbi_dying_cpu(unsigned int cpu, struct hlist_node *node)
> +static int pmu_sbi_setup_sse(struct riscv_pmu *pmu)
>  {
> -	if (riscv_pmu_use_irq) {
> -		disable_percpu_irq(riscv_pmu_irq);
> -	}
> +	int ret;
> +	struct sse_event *evt;
>  
> -	/* Disable all counters access for user mode now */
> -	csr_write(CSR_SCOUNTEREN, 0x0);
> +	evt = sse_event_register(SBI_SSE_EVENT_LOCAL_PMU_OVERFLOW, 0,
> +				 pmu_sbi_ovf_sse_handler, pmu);
> +	if (IS_ERR(evt))
> +		return PTR_ERR(evt);
> +	pmu->sse_evt = evt;
>  
> -	if (sbi_pmu_snapshot_available())
> -		return pmu_sbi_snapshot_disable();
> +	ret = sse_event_enable(evt);
> +	if (ret) {
> +		int cleanup_ret;
>  
> -	return 0;
> +		cleanup_ret = sse_event_disable(evt);
> +		if (cleanup_ret) {
> +			pr_warn("failed to disable SSE event after setup error: %d\n",
> +				cleanup_ret);
> +			return cleanup_ret;
> +		}
> +
> +		cleanup_ret = sse_event_unregister(evt);
> +
> +		if (cleanup_ret) {
> +			pr_warn("failed to unregister SSE event: %d\n",
> +				cleanup_ret);
> +		} else {
> +			pmu->sse_evt = NULL;
> +		}
> +		return cleanup_ret ?: ret;
> +	}
> +
> +	WRITE_ONCE(pmu->sse_active, true);
> +	pr_info("using SSE for PMU event delivery\n");
> +
> +	return ret;
>  }
>  
> -static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pdev)
> +static void pmu_sbi_cleanup_sse(struct riscv_pmu *pmu)
>  {
> +	struct sse_event *sse_evt;
>  	int ret;
> -	struct cpu_hw_events __percpu *hw_events = pmu->hw_events;
> -	struct irq_domain *domain = NULL;
>  
> +	sse_evt = pmu->sse_evt;
> +	if (!sse_evt)
> +		return;
> +	/*
> +	 * Close callback admission before draining each CPU. PMU callbacks run
> +	 * with local interrupts disabled, so the synchronous IPI cannot complete
> +	 * until a callback that observed the old state has returned.
> +	 */
> +	WRITE_ONCE(pmu->sse_active, false);
> +	on_each_cpu(pmu_sbi_stop_all_cpu, pmu, 1);
> +
> +	ret = sse_event_disable(sse_evt);
> +	if (ret) {
> +		pr_warn("failed to disable SSE event: %d\n", ret);
> +		goto retain;
> +	}
> +
> +	ret = sse_event_unregister(sse_evt);
> +	if (ret) {
> +		pr_warn("failed to unregister SSE event: %d\n", ret);
> +		goto retain;
> +	}
> +
> +	pmu->sse_evt = NULL;
> +	return;
> +
> +retain:
> +	sse_event_cleanup(sse_evt);
> +	pmu->sse_evt = NULL;
> +}
> +
> +static bool pmu_sbi_sse_state_retained(struct riscv_pmu *pmu)
> +{
> +	return pmu->sse_evt;
> +}
> +#else
> +static int pmu_sbi_setup_sse(struct riscv_pmu *pmu)
> +{
> +	return -EOPNOTSUPP;
> +}
> +
> +static void pmu_sbi_cleanup_sse(struct riscv_pmu *pmu) {}
> +
> +static bool pmu_sbi_sse_state_retained(struct riscv_pmu *pmu)
> +{
> +	return false;
> +}
> +#endif
> +
> +static bool pmu_sbi_select_irq(void)
> +{
>  	if (riscv_isa_extension_available(NULL, SSCOFPMF)) {
>  		riscv_pmu_irq_num = RV_IRQ_PMU;
> -		riscv_pmu_use_irq = true;
> +		return true;
>  	} else if (IS_ENABLED(CONFIG_ERRATA_THEAD_PMU) &&
>  		   riscv_cached_mvendorid(0) == THEAD_VENDOR_ID &&
>  		   riscv_cached_marchid(0) == 0 &&
>  		   riscv_cached_mimpid(0) == 0) {
>  		riscv_pmu_irq_num = THEAD_C9XX_RV_IRQ_PMU;
> -		riscv_pmu_use_irq = true;
> +		return true;
>  	} else if (riscv_has_vendor_extension_unlikely(ANDES_VENDOR_ID,
>  						       RISCV_ISA_VENDOR_EXT_XANDESPMU) &&
>  		   IS_ENABLED(CONFIG_ANDES_CUSTOM_PMU)) {
>  		riscv_pmu_irq_num = ANDES_SLI_CAUSE_BASE + ANDES_RV_IRQ_PMOVI;
> -		riscv_pmu_use_irq = true;
> +		return true;
>  	}
>  
> -	riscv_pmu_irq_mask = BIT(riscv_pmu_irq_num % BITS_PER_LONG);
> +	return false;
> +}
>  
> -	if (!riscv_pmu_use_irq)
> +static int pmu_sbi_setup_irq(struct riscv_pmu *pmu)
> +{
> +	struct cpu_hw_events __percpu *hw_events = pmu->hw_events;
> +	struct irq_domain *domain;
> +	int ret;
> +
> +	if (!pmu_sbi_select_irq())
>  		return -EOPNOTSUPP;
>  
> +	riscv_pmu_irq_mask = BIT(riscv_pmu_irq_num % BITS_PER_LONG);
> +
>  	domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(),
>  					  DOMAIN_BUS_ANY);
>  	if (!domain) {
>  		pr_err("Failed to find INTC IRQ root domain\n");
> -		ret = -ENODEV;
> -		goto err;
> +		return -ENODEV;
>  	}
>  
>  	riscv_pmu_irq = irq_create_mapping(domain, riscv_pmu_irq_num);
>  	if (!riscv_pmu_irq) {
>  		pr_err("Failed to map PMU interrupt for node\n");
> -		ret = -ENODEV;
> -		goto err;
> +		return -ENODEV;
>  	}
>  
> -	ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_handler, "riscv-pmu", hw_events);
> +	ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_irq_handler,
> +				 "riscv-pmu", hw_events);
>  	if (ret) {
>  		pr_err("registering percpu irq failed [%d]\n", ret);
>  		irq_dispose_mapping(riscv_pmu_irq);
>  		riscv_pmu_irq = 0;
> -		goto err;
> +		return ret;
>  	}
>  
>  	return 0;
> -err:
> -	riscv_pmu_use_irq = false;
> -	return ret;
> +}
> +
> +static int pmu_sbi_starting_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> +	struct riscv_pmu *pmu = hlist_entry_safe(node, struct riscv_pmu, node);
> +	struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events);
> +
> +	/*
> +	 * We keep enabling userspace access to CYCLE, TIME and INSTRET via the
> +	 * legacy option but that will be removed in the future.
> +	 */
> +	if (sysctl_perf_user_access == SYSCTL_LEGACY)
> +		csr_write(CSR_SCOUNTEREN, 0x7);
> +	else
> +		csr_write(CSR_SCOUNTEREN, 0x2);
> +
> +	/* Stop all the counters so that they can be enabled from perf */
> +	pmu_sbi_stop_all(pmu);
> +
> +	if (riscv_pmu_use_irq) {
> +		cpu_hw_evt->irq = riscv_pmu_irq;
> +		ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask);
> +		enable_percpu_irq(riscv_pmu_irq, IRQ_TYPE_NONE);
> +	}
> +
> +	if (sbi_pmu_snapshot_available())
> +		return pmu_sbi_snapshot_setup(pmu, cpu);
> +
> +	return 0;
> +}
> +
> +static int pmu_sbi_dying_cpu(unsigned int cpu, struct hlist_node *node)
> +{
> +	if (riscv_pmu_use_irq)
> +		disable_percpu_irq(riscv_pmu_irq);
> +
> +	/* Disable all counters access for user mode now */
> +	csr_write(CSR_SCOUNTEREN, 0x0);
> +
> +	if (sbi_pmu_snapshot_available())
> +		return pmu_sbi_snapshot_disable();
> +
> +	return 0;
> +}
> +
> +static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pdev)
> +{
> +	int irq_ret;
> +	int sse_ret;
> +
> +	/* Do not claim an IRQ route while a crash kernel leaves SSE state intact. */
> +	if (is_kdump_kernel() && riscv_sse_available()) {
> +		pr_warn("PMU delivery unavailable with retained crash-kernel SSE state\n");
> +		return -EUCLEAN;
> +	}
> +
> +	sse_ret = pmu_sbi_setup_sse(pmu);
> +	if (!sse_ret) {
> +		riscv_pmu_use_irq = false;
> +		return 0;
> +	}
> +	if (pmu_sbi_sse_state_retained(pmu)) {
> +		pr_err("PMU-SSE setup failed with firmware state retained: %d\n",
> +		       sse_ret);
> +		return -EUCLEAN;
> +	}
> +	/* Only an explicitly unsupported SSE path proves IRQ fallback is safe. */
> +	if (sse_ret != -EOPNOTSUPP) {
> +		pr_err("PMU-SSE setup failed: %d\n", sse_ret);
> +		return sse_ret;
> +	}
> +
> +	irq_ret = pmu_sbi_setup_irq(pmu);
> +	if (!irq_ret) {
> +		riscv_pmu_use_irq = true;
> +		return 0;
> +	}
> +	return irq_ret;
>  }
>  
>  #ifdef CONFIG_CPU_PM
> -static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd,
> -				void *v)
> +static int riscv_pm_pmu_update(struct riscv_pmu *rvpmu, unsigned long cmd)
>  {
> -	struct riscv_pmu *rvpmu = container_of(b, struct riscv_pmu, riscv_pm_nb);
>  	struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events);
>  	bool enabled = !bitmap_empty(cpuc->used_hw_ctrs, RISCV_MAX_COUNTERS);
>  	struct perf_event *event;
>  	int idx;
>  
> +	if (cmd == CPU_PM_ENTER)
> +		bitmap_zero(cpuc->pm_resume_hw_ctrs, RISCV_MAX_COUNTERS);
> +
>  	if (!enabled)
>  		return NOTIFY_OK;
>  
> @@ -1264,6 +1559,8 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd,
>  
>  		switch (cmd) {
>  		case CPU_PM_ENTER:
> +			if (!(event->hw.state & PERF_HES_STOPPED))
> +				set_bit(idx, cpuc->pm_resume_hw_ctrs);
>  			/*
>  			 * Stop and update the counter
>  			 */
> @@ -1271,6 +1568,8 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd,
>  			break;
>  		case CPU_PM_EXIT:
>  		case CPU_PM_ENTER_FAILED:
> +			if (!test_and_clear_bit(idx, cpuc->pm_resume_hw_ctrs))
> +				break;
>  			/*
>  			 * Restore and enable the counter.
>  			 */
> @@ -1284,9 +1583,59 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd,
>  	return NOTIFY_OK;
>  }
>  
> +static int riscv_pm_pmu_notify(struct notifier_block *b,
> +			       unsigned long cmd, void *v)
> +{
> +	struct riscv_pmu *rvpmu = container_of(b, struct riscv_pmu,
> +						 riscv_pm_nb);
> +
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events);
> +	int ret;
> +
> +	if (!riscv_pmu_use_irq && READ_ONCE(rvpmu->sse_active)) {
> +		switch (cmd) {
> +		case CPU_PM_ENTER:
> +			cpuc->pm_resume_sse =
> +				sse_event_is_enabled_local(rvpmu->sse_evt);
> +			if (cpuc->pm_resume_sse) {
> +				ret = sse_event_disable_local(rvpmu->sse_evt);
> +				if (ret) {
> +					cpuc->pm_resume_sse = false;
> +					pmu_sbi_fail_sse(rvpmu, "disable for CPU PM",
> +							 ret);
> +					return notifier_from_errno(ret);
> +				}
> +			}
> +			break;
> +		case CPU_PM_EXIT:
> +		case CPU_PM_ENTER_FAILED:
> +			ret = riscv_pm_pmu_update(rvpmu, cmd);
> +			if (!cpuc->pm_resume_sse)
> +				return ret;
> +
> +			cpuc->pm_resume_sse = false;
> +			ret = sse_event_enable_local(rvpmu->sse_evt);
> +			if (ret) {
> +				pmu_sbi_fail_sse(rvpmu, "enable after CPU PM", ret);
> +				return notifier_from_errno(ret);
> +			}
> +			return NOTIFY_OK;
> +		default:
> +			break;
> +		}
> +	}
> +#endif
> +
> +	return riscv_pm_pmu_update(rvpmu, cmd);
> +}
> +
>  static int riscv_pm_pmu_register(struct riscv_pmu *pmu)
>  {
>  	pmu->riscv_pm_nb.notifier_call = riscv_pm_pmu_notify;
> +	/* Keep PMU-SSE disabled until counters and userpage state are restored. */
> +	pmu->riscv_pm_nb.priority = riscv_pmu_use_irq ? 1 : -1;
> +
>  	return cpu_pm_register_notifier(&pmu->riscv_pm_nb);
>  }
>  
> @@ -1301,6 +1650,8 @@ static inline void riscv_pm_pmu_unregister(struct riscv_pmu *pmu) { }
>  
>  static void riscv_pmu_destroy(struct riscv_pmu *pmu)
>  {
> +	pmu_sbi_cleanup_sse(pmu);
> +
>  	if (sbi_v2_available) {
>  		if (sbi_pmu_snapshot_available()) {
>  			pmu_sbi_snapshot_disable();
> @@ -1312,7 +1663,7 @@ static void riscv_pmu_destroy(struct riscv_pmu *pmu)
>  		cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_STARTING, &pmu->node);
>  }
>  
> -static void pmu_sbi_event_init(struct perf_event *event)
> +static int pmu_sbi_event_init(struct perf_event *event)
>  {
>  	/*
>  	 * The permissions are set at event_init so that we do not depend
> @@ -1324,6 +1675,8 @@ static void pmu_sbi_event_init(struct perf_event *event)
>  		event->hw.flags |= PERF_EVENT_FLAG_USER_ACCESS;
>  	else
>  		event->hw.flags |= PERF_EVENT_FLAG_LEGACY;
> +
> +	return 0;
>  }
>  
>  static void pmu_sbi_event_mapped(struct perf_event *event, struct mm_struct *mm)
> @@ -1453,6 +1806,7 @@ static int pmu_sbi_device_probe(struct platform_device *pdev)
>  	/* cache all the information about counters now */
>  	if (pmu_sbi_get_ctrinfo(num_counters, &cmask))
>  		goto out_free;
> +	pmu->cmask = cmask;
>  
>  	ret = pmu_sbi_setup_irqs(pmu, pdev);
>  	if (ret < 0) {
> @@ -1462,9 +1816,15 @@ static int pmu_sbi_device_probe(struct platform_device *pdev)
>  	}
>  	irq_requested = (ret == 0);
>  
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	if (pmu->sse_active) {
> +		pmu->pmu.pmu_enable = pmu_sbi_sse_enable;
> +		pmu->pmu.pmu_disable = pmu_sbi_sse_disable;
> +	}
> +#endif
> +
>  	pmu->pmu.attr_groups = riscv_pmu_attr_groups;
>  	pmu->pmu.parent = &pdev->dev;
> -	pmu->cmask = cmask;
>  	pmu->ctr_start = pmu_sbi_ctr_start;
>  	pmu->ctr_stop = pmu_sbi_ctr_stop;
>  	pmu->event_map = pmu_sbi_event_map;
> diff --git a/include/linux/perf/riscv_pmu.h b/include/linux/perf/riscv_pmu.h
> index f82a28040594..fb913ce7f65e 100644
> --- a/include/linux/perf/riscv_pmu.h
> +++ b/include/linux/perf/riscv_pmu.h
> @@ -28,6 +28,8 @@
>  
>  #define RISCV_PMU_CONFIG1_GUEST_EVENTS 0x1
>  
> +struct sse_event;
> +
>  struct cpu_hw_events {
>  	/* currently enabled events */
>  	int			n_events;
> @@ -39,6 +41,18 @@ struct cpu_hw_events {
>  	DECLARE_BITMAP(used_hw_ctrs, RISCV_MAX_COUNTERS);
>  	/* currently enabled firmware counters */
>  	DECLARE_BITMAP(used_fw_ctrs, RISCV_MAX_COUNTERS);
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	/* Keep counters stopped after an unrecoverable SSE transition failure. */
> +	bool sse_failed;
> +#endif
> +#ifdef CONFIG_CPU_PM
> +	/* Counters stopped by CPU PM and still waiting to be restored. */
> +	DECLARE_BITMAP(pm_resume_hw_ctrs, RISCV_MAX_COUNTERS);
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	/* Restore the local PMU SSE event after counters and userpage state. */
> +	bool pm_resume_sse;
> +#endif
> +#endif
>  	/* The virtual address of the shared memory where counter snapshot will be taken */
>  	void *snapshot_addr;
>  	/* The physical address of the shared memory where counter snapshot will be taken */
> @@ -54,6 +68,10 @@ struct riscv_pmu {
>  	char		*name;
>  
>  	irqreturn_t	(*handle_irq)(int irq_num, void *dev);
> +#ifdef CONFIG_RISCV_PMU_SBI_SSE
> +	struct sse_event	*sse_evt;
> +	bool			sse_active;
> +#endif
>  
>  	unsigned long	cmask;
>  	u64		(*ctr_read)(struct perf_event *event);
> @@ -63,7 +81,7 @@ struct riscv_pmu {
>  	void		(*ctr_start)(struct perf_event *event, u64 init_val);
>  	void		(*ctr_stop)(struct perf_event *event, unsigned long flag);
>  	int		(*event_map)(struct perf_event *event, u64 *config);
> -	void		(*event_init)(struct perf_event *event);
> +	int		(*event_init)(struct perf_event *event);
>  	void		(*event_mapped)(struct perf_event *event, struct mm_struct *mm);
>  	void		(*event_unmapped)(struct perf_event *event, struct mm_struct *mm);
>  	uint8_t		(*csr_index)(struct perf_event *event);
> diff --git a/include/linux/riscv_sbi_sse.h b/include/linux/riscv_sbi_sse.h
> index 774a782e556d..f6b3881c8884 100644
> --- a/include/linux/riscv_sbi_sse.h
> +++ b/include/linux/riscv_sbi_sse.h
> @@ -42,6 +42,7 @@ int sse_event_enable(struct sse_event *sse_evt);
>  int sse_event_disable(struct sse_event *sse_evt);
>  
>  /* Local events require the caller to remain on the current CPU. */
> +bool sse_event_is_enabled_local(struct sse_event *sse_evt);
>  int sse_event_enable_local(struct sse_event *sse_evt);
>  int sse_event_disable_local(struct sse_event *sse_evt);
>  
> @@ -76,6 +77,11 @@ static inline int sse_event_disable(struct sse_event *sse_evt)
>  	return -EOPNOTSUPP;
>  }
>  
> +static inline bool sse_event_is_enabled_local(struct sse_event *sse_evt)
> +{
> +	return false;
> +}
> +
>  static inline int sse_event_enable_local(struct sse_event *sse_evt)
>  {
>  	return -EOPNOTSUPP;
> -- 
> 2.50.1 (Apple Git-155)
>