Re: [External] [RFC PATCH 0/3] riscv: add support for SBI Supervisor Software Events

[Clément Léger]

On 07/12/2023 10:09, Xu Lu wrote:
> Pardon. It seems that the code in opensbi[1] is not complete for PMU NMI now.
> For example, the pmu ovf irq is still delegated to supervisor mode and
> thus can not really play a role as NMI. And neither the kernel nor
> opensbi will inject a pmu event.
> 
> To complete the work, we think maybe 'enable_cb' and 'disable_cb'
> functions can be supplied for sbi_sse_cb_ops.
> When sbi_sse_enable is called to enable pmu event, the enable_cb will
> be called to revoke the delegation of pmu ovf irq and enable this irq
> in CSR_MIE.

Hi Xu,

Indeed, this part has been developed but was left out for the RFC. But
your understanding is correct.

> When pmu ovf irq occurs, kernel traps into opensbi and opensbi injects
> the event via sbi_sse_inject_event and eret back to kernel.
> 
> Please point it out if we have any misunderstanding.
> 
> By the way, how is SSE going now? We will appreciate it very much if
> we can participate in some development work in kernel or opensbi and
> be of any help.

The development is almost complete, Anup/Himanchu will send a new
revision of the spec addressing various comments and I'll resend the RFC
following that spec update.

Regards,

Clément

> 
> Regards!
> 
> Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [1]
> 
> On Thu, Oct 26, 2023 at 10:31 PM Clément Léger <cleger@xxxxxxxxxxxx> wrote:
>>
>> The SBI Supervisor Software Events (SSE) extensions provides a mechanism
>> to inject software events from an SBI implementation to supervisor
>> software such that it preempts all other supervisor level traps and
>> interrupts [1].
>>
>> Various events are defined and can be send asynchronously to supervisor
>> software (RAS, PMU, DEBUG, Asynchronous page fault) from SBI as well
>> as platform specific events. Events can be either local (per-hart) or
>> global. Events can be nested on top of each other based on priority and
>> can interrupt the kernel at any time.
>>
>> First patch adds the SSE definitions. Second one adds support for SSE
>> itself. Implementation is split between arch specific code and generic
>> part (similarly to what is done for ARM SDEI). Finally, the last patch
>> add support fro SSE event in the SBI PMU driver. If the SSE event is
>> available from the SBI then, it will be used instead of the normal
>> interrupt.
>>
>> Amongst the specific points that needs to be handle is the interruption
>> at any point of the kernel execution and more specifically during
>> exception handling. Due to the fact that the exception entry
>> implementation uses the SCRATCH CSR as both the current task struct and
>> as the temporary register to switch the stack and save register, it is
>> difficult to reliably get the current task struct if we get interrupted
>> at this specific moment. A fixup-like mechanism allows to mark the
>> location of the current task struct depending on the entry level
>> (user/kernel) and the location. This is then used in the SSE assembly to
>> determine where is located the current task_struct.
>>
>> Contrary to pseudo NMI [2], SSE does not modifies the way interrupts are
>> handled and does not adds any overhead to existing code. Moreover, it
>> provides "true" NMI-like interrupts which can interrupt the kernel at
>> any time (even in exception handling). This is particularly crucial for
>> RAS errors which needs to be handled as fast as possible to avoid any
>> fault propagation. Additionally, SSE event handling is faster that the
>> standard IRQ handling path with almost half executed instruction (700 vs
>> 1590). Some complementary tests/perf measurements will be done.
>>
>> For testing purpose, one can use the provided SBI implementation at [3].
>> This series also needs patch [4] to fix a bug in the PMU driver.
>>
>> Link: https://lists.riscv.org/g/tech-prs/message/515 [1]
>> Link: https://lore.kernel.org/lkml/20231023082911.23242-10-luxu.kernel@xxxxxxxxxxxxx/T/ [2]
>> Link: https://github.com/rivosinc/opensbi/tree/dev/cleger/sse [3]
>> Link: https://lore.kernel.org/linux-arm-kernel/20231026084010.11888-1-alexghiti@xxxxxxxxxxxx/ [4]
>>
>> ---
>>
>> Clément Léger (3):
>>   riscv: add SBI SSE extension definitions
>>   riscv: add support for SBI Supervisor Software Events extension
>>   perf: RISC-V: add support for SSE event
>>
>>  arch/riscv/include/asm/asm-prototypes.h |   5 +
>>  arch/riscv/include/asm/sbi.h            |  40 ++
>>  arch/riscv/include/asm/sse.h            |  94 +++++
>>  arch/riscv/kernel/Makefile              |   1 +
>>  arch/riscv/kernel/asm-offsets.c         |  17 +
>>  arch/riscv/kernel/entry.S               | 156 ++++++++
>>  arch/riscv/kernel/sbi.c                 |   4 +
>>  arch/riscv/kernel/sse.c                 |  97 +++++
>>  arch/riscv/kernel/stacktrace.c          |  13 +
>>  arch/riscv/kernel/vmlinux.lds.S         |   6 +
>>  drivers/firmware/Kconfig                |  10 +
>>  drivers/firmware/Makefile               |   1 +
>>  drivers/firmware/riscv_sse.c            | 496 ++++++++++++++++++++++++
>>  drivers/perf/riscv_pmu_sbi.c            |  51 ++-
>>  include/linux/riscv_sse.h               |  56 +++
>>  15 files changed, 1038 insertions(+), 9 deletions(-)
>>  create mode 100644 arch/riscv/include/asm/sse.h
>>  create mode 100644 arch/riscv/kernel/sse.c
>>  create mode 100644 drivers/firmware/riscv_sse.c
>>  create mode 100644 include/linux/riscv_sse.h
>>
>> --
>> 2.42.0
>>