Re: [WARNING] kernel/rcu/tree.c:1058 rcu_irq_enter+0x15/0x20

[Peter Zijlstra]

On Thu, Sep 17, 2020 at 01:16:46PM -0400, Steven Rostedt wrote:
> Hi Peter,
> 
> I ran my tests on a series of patches on top of 5.9-rc4, and hit the
> following splat:
> 
>  ------------[ cut here ]------------
>  WARNING: CPU: 0 PID: 2557 at kernel/rcu/tree.c:1058 rcu_irq_enter+0x15/0x20
>  RIP: 0010:rcu_irq_enter+0x15/0x20

>  Call Trace:
>   rcu_irq_enter_irqson+0x21/0x40
>   trace_preempt_off+0x6e/0xd0
>   unwind_next_frame+0x41/0x560
>   __unwind_start+0x153/0x1e0
>   arch_stack_walk+0x76/0x100
>   stack_trace_save+0x4b/0x70
>   save_trace+0x42/0x350
>   __lock_acquire+0x1858/0x2460
>   lock_acquire+0xdc/0x3b0
>   cpuset_read_lock+0x26/0xc0

> What looks to have happened was:
> 
>   cpuset_read_lock()
>      lockdep called
>        save stack trace
>           preempt_disable()
>              trace_preempt_disable();
>                 rcu_irq_enter_irqson();
>                    local_irq_save() (ignored by lockdep due to recursion set)
>                       rcu_irq_enter();
>                          lockdep_assert_irqs_disabled() (no, because it was ignored by recursion being set)
> 
>                           BOOM!
> 
> 
> Thoughts?

> diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
> index 6a584b3e5c74..3e5bc1dd71c6 100644
> --- a/include/linux/lockdep.h
> +++ b/include/linux/lockdep.h
> @@ -550,7 +550,8 @@ do {									\
>  
>  #define lockdep_assert_irqs_disabled()					\
>  do {									\
> -	WARN_ON_ONCE(debug_locks && raw_cpu_read(hardirqs_enabled));	\
> +	WARN_ON_ONCE(debug_locks && raw_cpu_read(hardirqs_enabled) &&	\
> +           likely(!(current->lockdep_recursion & LOCKDEP_RECURSION_MASK)));\
>  } while (0)

Blergh, IIRC there's header hell that way. The sane fix is killing off
that trace_*_rcuidle() disease.

But I think this will also cure it.

---
diff --git a/arch/x86/kernel/unwind_orc.c b/arch/x86/kernel/unwind_orc.c
index 6a339ce328e0..4f90293d170b 100644
--- a/arch/x86/kernel/unwind_orc.c
+++ b/arch/x86/kernel/unwind_orc.c
@@ -432,7 +432,7 @@ bool unwind_next_frame(struct unwind_state *state)
 		return false;
 
 	/* Don't let modules unload while we're reading their ORC data. */
-	preempt_disable();
+	preempt_disable_notrace();
 
 	/* End-of-stack check for user tasks: */
 	if (state->regs && user_mode(state->regs))
@@ -612,14 +612,14 @@ bool unwind_next_frame(struct unwind_state *state)
 		goto err;
 	}
 
-	preempt_enable();
+	preempt_enable_notrace();
 	return true;
 
 err:
 	state->error = true;
 
 the_end:
-	preempt_enable();
+	preempt_enable_notrace();
 	state->stack_info.type = STACK_TYPE_UNKNOWN;
 	return false;
 }