[PATCH 1/2] arm64: sdei: Run the event handler on the kernel vectors

[Kiryl Shutsemau (Meta)]

With KPTI, VBAR_EL1 holds the trampoline vectors from the return to EL0
until the next exception from EL0 maps the kernel and installs the
kernel vectors. The trampoline vectors only handle exceptions from EL0;
their current-EL half is zero-filled.

An SDEI event is dispatched regardless of PSTATE.DAIF, so it lands there
whenever the CPU is in user space. The SDEI entry trampoline maps the
kernel but leaves VBAR_EL1 alone. An exception taken by the handler then
executes the zeros, takes an undefined instruction exception to the same
vector, and loops there forever. The event never completes and the CPU
is lost; RCU stall reports are the first sign of it.

RAS notifications land in user space whenever a task consumes the
error, and NMI backtraces land wherever the CPU happens to be. The GHES
handler takes no exception on its own, but any exception in the handler
path does it: a WARN, a kprobe hit, or a fault fixed up by a _nofault()
accessor.

Install the kernel vectors in do_sdei_event() when VBAR_EL1 holds the
trampoline vectors, and restore it before returning to firmware,
including when the handler fails. The IRQ-like resume into user space
keeps using the interrupted value, as it must go through the trampoline
to map the kernel.

Fixes: 79e9aa59dc29 ("arm64: sdei: Add trampoline code for remapping the kernel")
Cc: stable@xxxxxxxxxxxxxxx
Signed-off-by: Kiryl Shutsemau (Meta) <kas@xxxxxxxxxx>
---
 arch/arm64/kernel/sdei.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/sdei.c b/arch/arm64/kernel/sdei.c
index 778f2a1faac8..9fa2a3aeed48 100644
--- a/arch/arm64/kernel/sdei.c
+++ b/arch/arm64/kernel/sdei.c
@@ -18,6 +18,7 @@
 #include <asm/sections.h>
 #include <asm/stacktrace.h>
 #include <asm/sysreg.h>
+#include <asm/vectors.h>
 #include <asm/vmap_stack.h>
 
 unsigned long sdei_exit_mode;
@@ -214,6 +215,20 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	if (arm64_kernel_unmapped_at_el0())
 		clobbered_registers++;
 
+	/*
+	 * With KPTI, VBAR_EL1 holds the trampoline vectors from the return to
+	 * EL0 until the next exception from EL0 maps the kernel. They only
+	 * handle exceptions from EL0: an exception taken by the handler would
+	 * run their empty current-EL half and never come back. Run the handler
+	 * on the kernel vectors. The IRQ-like resume below keeps using the
+	 * interrupted VBAR, so it still goes through the trampoline.
+	 */
+	if (arm64_kernel_unmapped_at_el0() &&
+	    vbar == (unsigned long)this_cpu_read(this_cpu_vector)) {
+		write_sysreg((unsigned long)vectors, vbar_el1);
+		isb();
+	}
+
 	/* Retrieve the missing registers values */
 	for (i = 0; i < clobbered_registers; i++) {
 		/* from within the handler, this call always succeeds */
@@ -221,8 +236,6 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 	}
 
 	err = sdei_event_handler(regs, arg);
-	if (err)
-		return SDEI_EV_FAILED;
 
 	if (elr != read_sysreg(elr_el1)) {
 		/*
@@ -233,6 +246,11 @@ unsigned long __kprobes do_sdei_event(struct pt_regs *regs,
 		pr_warn("unsafe: exception during handler\n");
 	}
 
+	write_sysreg(vbar, vbar_el1);
+
+	if (err)
+		return SDEI_EV_FAILED;
+
 	mode = regs->pstate & (PSR_MODE32_BIT | PSR_MODE_MASK);
 
 	/*
-- 
2.54.0