[PATCH v2] softirq: Preserve interrupt context during IRQ exit

[Karl Mehltretter]

__irq_exit_rcu() drops HARDIRQ_OFFSET before deferred hrtimer rearm,
entry into softirq dispatch, and timersd wakeup. The rearm, wakeup, and
softirq entry code before softirq_handle_begin() are still on the IRQ
return path, but in_task() reports task context.

Context-sensitive code called from this window therefore sees task
context. ftrace records normal-context flags and selects its normal
recursion slot. KCSAN attributes IRQ-exit accesses to the interrupted
task, while KMSAN can select and modify that task's metadata.

Keep HARDIRQ_OFFSET across this IRQ-exit window. For direct softirq
handling, replace it with SOFTIRQ_OFFSET and restore it afterwards. Other
__do_softirq() call paths keep their existing accounting.

With hardirq context retained, ftrace records hardirq context, KCSAN uses
interrupt attribution, and KMSAN no longer uses the interrupted task's
state.

Test softirq eligibility before removing HARDIRQ_OFFSET with
irq_count() == HARDIRQ_OFFSET. This preserves the old !in_interrupt()
semantics, including PREEMPT_RT's task-local softirq-disable state. For
timersd, require exactly one hardirq nesting level and no NMI, preserving
the old predicate.

Drop HARDIRQ_OFFSET before tick_irq_exit(), as before. Preempt-off tracing
now covers the IRQ-exit work continuously instead of showing an artificial
on/off transition.

Suggested-by: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
Link: https://lore.kernel.org/r/20260813130826.GW687043@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Assisted-by: LLM
Signed-off-by: Karl Mehltretter <kmehltretter@xxxxxxxxx>
---
Changes since v1:
- Warn if the hardirq-to-softirq transition does not produce exactly one
  SOFTIRQ_OFFSET, and update lockdep state unconditionally (Frederic).
- Tighten the commit message around the affected IRQ-exit window, the
  timersd predicate and preempt-off tracing.
- Add exact-v2 QEMU, sanitizer, hardware and idle-latency results to the
  review notes.

v1: https://lore.kernel.org/r/20260903112737.49551-1-kmehltretter@xxxxxxxxx

The exact v2 source passed non-RT, threadirqs and PREEMPT_RT x86-64 QEMU
boot/stress, including CPU hotplug, with panic_on_warn=1, lockdep and IRQ
tracing. The same three modes passed a continuous context-invariant test;
neither new warning fired.

KCSAN attributed all 25 target reports to interrupt context. KMSAN selected
or changed task state zero times in 24K IRQ-exit windows and passed all 28
KUnit tests.

vmlinux linked successfully for arm64, ARM, RISC-V and s390.

The exact v2 TIP source passed config-identical A/B boot/stress on a Pi 400
(Cortex-A72, arm64) with panic_on_warn=1: 2000/2000 ping, no boot splat and
no new dmesg output. Focused ftrace, printk/fault-injection and continuous-
invariant diagnostics also passed on the same source.

A second A/B on the Pi 400 with threadirqs enabled also passed. Four
ktimers threads were active on each side, 2000/2000 ping completed, and the
dmesg buffer remained empty after the workload.

A four-boot ABBA cyclictest run on the Pi found no idle timer-latency
regression. Across 6M samples per side, p99 was 5 us on both kernels and
p99.9 was 7 us on the baseline versus 5 us patched.

For additional portability coverage, the mainline v2 adaptation passed
config-identical A/B boot/stress on a SAM9X75 (ARM926EJ-S/ARMv5TEJ) with
panic_on_warn=1, 1000/1000 ping, no boot splat and no new dmesg output.

 kernel/softirq.c | 48 ++++++++++++++++++++++++++++++++++++------------
 1 file changed, 36 insertions(+), 12 deletions(-)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 5d02c36c40e3..7381b6b4d551 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -350,8 +350,8 @@ static inline void ksoftirqd_run_end(void)
 	local_irq_enable();
 }
 
-static inline void softirq_handle_begin(void) { }
-static inline void softirq_handle_end(void) { }
+static inline bool softirq_handle_begin(void) { return false; }
+static inline void softirq_handle_end(bool from_hardirq) { }
 
 static inline bool should_wake_ksoftirqd(void)
 {
@@ -481,15 +481,35 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
 }
 EXPORT_SYMBOL(__local_bh_enable_ip);
 
-static inline void softirq_handle_begin(void)
+static inline bool softirq_handle_begin(void)
 {
-	__local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET);
+	bool from_hardirq = in_hardirq();
+
+	if (!from_hardirq) {
+		__local_bh_disable_ip(_RET_IP_, SOFTIRQ_OFFSET);
+		return false;
+	}
+
+	/* Replace the retained hardirq context with normal softirq context. */
+	__preempt_count_add((int)SOFTIRQ_OFFSET - (int)HARDIRQ_OFFSET);
+	WARN_ON_ONCE(softirq_count() != SOFTIRQ_OFFSET);
+	lockdep_softirqs_off(_RET_IP_);
+
+	return true;
 }
 
-static inline void softirq_handle_end(void)
+static inline void softirq_handle_end(bool from_hardirq)
 {
-	__local_bh_enable(SOFTIRQ_OFFSET);
-	WARN_ON_ONCE(in_interrupt());
+	if (!from_hardirq) {
+		__local_bh_enable(SOFTIRQ_OFFSET);
+		WARN_ON_ONCE(in_interrupt());
+		return;
+	}
+
+	WARN_ON_ONCE(softirq_count() != SOFTIRQ_OFFSET);
+	lockdep_softirqs_on(_RET_IP_);
+	__preempt_count_sub((int)SOFTIRQ_OFFSET - (int)HARDIRQ_OFFSET);
+	WARN_ON_ONCE(!in_hardirq());
 }
 
 static inline void ksoftirqd_run_begin(void)
@@ -605,6 +625,7 @@ static void handle_softirqs(bool ksirqd)
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	struct softirq_action *h;
+	bool from_hardirq;
 	bool in_hardirq;
 	__u32 pending;
 	int softirq_bit;
@@ -618,7 +639,7 @@ static void handle_softirqs(bool ksirqd)
 
 	pending = local_softirq_pending();
 
-	softirq_handle_begin();
+	from_hardirq = softirq_handle_begin();
 	in_hardirq = lockdep_softirq_start();
 	account_softirq_enter(current);
 
@@ -670,7 +691,7 @@ static void handle_softirqs(bool ksirqd)
 
 	account_softirq_exit(current);
 	lockdep_softirq_end(in_hardirq);
-	softirq_handle_end();
+	softirq_handle_end(from_hardirq);
 	current_restore_flags(old_flags, PF_MEMALLOC);
 }
 
@@ -740,6 +761,8 @@ static inline void wake_timersd(void) { }
 
 #endif
 
+#define IRQ_EXIT_TIMERS	(NMI_MASK | HARDIRQ_MASK)
+
 static inline void __irq_exit_rcu(void)
 {
 #ifndef __ARCH_IRQ_EXIT_IRQS_DISABLED
@@ -748,8 +771,7 @@ static inline void __irq_exit_rcu(void)
 	lockdep_assert_irqs_disabled();
 #endif
 	account_hardirq_exit(current);
-	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt() && local_softirq_pending()) {
+	if (irq_count() == HARDIRQ_OFFSET && local_softirq_pending()) {
 		/*
 		 * If we left hrtimers unarmed, make sure to arm them now,
 		 * before enabling interrupts to run softirq.
@@ -759,9 +781,11 @@ static inline void __irq_exit_rcu(void)
 	}
 
 	if (IS_ENABLED(CONFIG_IRQ_FORCED_THREADING) && force_irqthreads() &&
-	    local_timers_pending_force_th() && !(in_nmi() | in_hardirq()))
+	    local_timers_pending_force_th() &&
+	    (preempt_count() & IRQ_EXIT_TIMERS) == HARDIRQ_OFFSET)
 		wake_timersd();
 
+	preempt_count_sub(HARDIRQ_OFFSET);
 	tick_irq_exit();
 }
 

base-commit: 2af470916a208b576ac9975d221d9a378cf8ace9
-- 
2.53.0