[PATCH 14/14] rcu: Introduce lockdep_assume_in_rcu_*()

[Bart Van Assche]

Remove the context assumptions from lockdep_assert_rcu_helper() and from
the !CONFIG_PROVE_RCU definitions of lockdep_assert_in_rcu_*() in
include/linux/rcupdate.h. Introduce the following macros and document
them with kernel-doc comments:
- lockdep_assume_in_rcu_read_lock(): assume RCU and call
  lockdep_assert_in_rcu_read_lock().
- lockdep_assume_in_rcu_read_lock_bh(): assume RCU and RCU_BH and call
  lockdep_assert_in_rcu_read_lock_bh().
- lockdep_assume_in_rcu_read_lock_sched(): assume RCU and RCU_SCHED and
  call lockdep_assert_in_rcu_read_lock_sched().
- lockdep_assume_in_rcu_reader(): assume RCU and call
  lockdep_assert_in_rcu_reader().
Assume RCU in the _bh() and _sched() variants because rcu_read_lock_bh()
and rcu_read_lock_sched() acquire RCU in addition to RCU_BH and
RCU_SCHED, and because __rcu_guarded pointers are guarded by RCU. Use
the lockdep_assume_ prefix for consistency with lockdep_assume_ctx_held()
and lockdep_assume_shared_ctx_held(): __assume_*() macros only inform
context analysis while lockdep_assume_*() macros also verify the
assumption at runtime.

Mention lockdep_assume_in_rcu_read_lock() in
Documentation/dev-tools/context-analysis.rst. Convert the RCU assertion
tests in lib/test_context-analysis.c to the lockdep_assume_in_rcu_*()
macros and verify in the _bh() and _sched() tests that RCU is assumed
too.

Prevent lockdep_assert_in_rcu_*() from silencing Clang context analysis
warnings in functions that lack a __must_hold_shared(RCU) annotation,
such that the compiler reports missing function annotations and checks
callers for RCU read lock ownership at compile time. This makes the RCU
assertions consistent with lockdep_assert_held*().

Verify at runtime with lockdep that the RCU read lock is held whenever
one of the lockdep_assume_in_rcu_*() macros instructs Clang context
analysis to assume RCU read lock ownership without compile-time proof.

Change lockdep_assert_rcu_helper() from __always_inline back into inline.
Commit 451c3a035609 ("rcu: Mark lockdep_assert_rcu_helper()
__always_inline") made this function __always_inline because if it was
not inlined, its callers passed the address of the RCU, RCU_BH or
RCU_SCHED context token to it. Since these tokens are not defined
anywhere, that resulted in link failures. Now that the ctx argument has
been removed from lockdep_assert_rcu_helper(), the context tokens are no
longer referenced and hence __always_inline is no longer necessary.

Cc: Marco Elver <elver@xxxxxxxxxx>
Cc: Peter Zijlstra <peterz@xxxxxxxxxxxxx>
Cc: Paul E. McKenney <paulmck@xxxxxxxxxx>
Signed-off-by: Bart Van Assche <bvanassche@xxxxxxx>
---
 Documentation/dev-tools/context-analysis.rst | 26 ++++---
 include/linux/rcupdate.h                     | 82 +++++++++++++++++---
 lib/test_context-analysis.c                  | 10 ++-
 3 files changed, 92 insertions(+), 26 deletions(-)

diff --git a/Documentation/dev-tools/context-analysis.rst b/Documentation/dev-tools/context-analysis.rst
index 88b4ed6a182f..bc94e5bb56a5 100644
--- a/Documentation/dev-tools/context-analysis.rst
+++ b/Documentation/dev-tools/context-analysis.rst
@@ -111,18 +111,20 @@ checking): either with a ``context_unsafe(var = init)`` expression, or by
 marking small initialization functions with the ``__context_unsafe(init)``
 attribute.
 
-`lockdep_assume_ctx_held()` and `lockdep_assume_shared_ctx_held()` inform the
-compiler's context analysis that the associated synchronization primitive is
-held and verify this at runtime via Lockdep. This avoids false positives in
-complex control-flow scenarios while still checking lock ownership with Lockdep
-where static analysis is limited. For example, this is useful when a function
-doesn't *always* require a lock, making `__must_hold()` inappropriate.
-
-Lockdep assertions, such as `lockdep_assert_held()`, only perform a runtime
-check and do not inform the compiler's context analysis that a lock is held.
-For context locks that are not tracked by Lockdep, e.g. bit spinlocks, use
-`__assume_ctx_lock()` or `__assume_shared_ctx_lock()` instead. These only
-inform the compiler's context analysis and do not perform a runtime check.
+`lockdep_assume_ctx_held()`, `lockdep_assume_shared_ctx_held()` and
+`lockdep_assume_in_rcu_read_lock()` inform the compiler's context analysis that
+the associated synchronization primitive is held and verify this at runtime via
+Lockdep. This avoids false positives in complex control-flow scenarios while
+still checking lock ownership with Lockdep where static analysis is limited.
+For example, this is useful when a function doesn't *always* require a lock,
+making `__must_hold()` inappropriate.
+
+Lockdep assertions, such as `lockdep_assert_held()` and
+`lockdep_assert_in_rcu_read_lock()`, only perform a runtime check and do not
+inform the compiler's context analysis that a lock is held. For context locks
+that are not tracked by Lockdep, e.g. bit spinlocks, use `__assume_ctx_lock()`
+or `__assume_shared_ctx_lock()` instead. These only inform the compiler's
+context analysis and do not perform a runtime check.
 
 Keywords
 ~~~~~~~~
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 3f74ae6d6e1f..3effadda45a3 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -403,8 +403,7 @@ static inline void rcu_preempt_sleep_check(void) { }
 
 // See RCU_LOCKDEP_WARN() for an explanation of the double call to
 // debug_lockdep_rcu_enabled().
-static __always_inline bool lockdep_assert_rcu_helper(bool c, const struct __ctx_lock_RCU *ctx)
-	__assumes_shared_ctx_lock(RCU) __assumes_shared_ctx_lock(ctx)
+static inline bool lockdep_assert_rcu_helper(bool c)
 {
 	return debug_lockdep_rcu_enabled() &&
 	       (c || !rcu_is_watching() || !rcu_lockdep_current_cpu_online()) &&
@@ -417,7 +416,7 @@ static __always_inline bool lockdep_assert_rcu_helper(bool c, const struct __ctx
  * Splats if lockdep is enabled and there is no rcu_read_lock() in effect.
  */
 #define lockdep_assert_in_rcu_read_lock() \
-	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map), RCU))
+	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map)))
 
 /**
  * lockdep_assert_in_rcu_read_lock_bh - WARN if not protected by rcu_read_lock_bh()
@@ -427,7 +426,7 @@ static __always_inline bool lockdep_assert_rcu_helper(bool c, const struct __ctx
  * actual rcu_read_lock_bh() is required.
  */
 #define lockdep_assert_in_rcu_read_lock_bh() \
-	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_bh_lock_map), RCU_BH))
+	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_bh_lock_map)))
 
 /**
  * lockdep_assert_in_rcu_read_lock_sched - WARN if not protected by rcu_read_lock_sched()
@@ -437,7 +436,7 @@ static __always_inline bool lockdep_assert_rcu_helper(bool c, const struct __ctx
  * instead an actual rcu_read_lock_sched() is required.
  */
 #define lockdep_assert_in_rcu_read_lock_sched() \
-	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_sched_lock_map), RCU_SCHED))
+	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_sched_lock_map)))
 
 /**
  * lockdep_assert_in_rcu_reader - WARN if not within some type of RCU reader
@@ -455,20 +454,83 @@ static __always_inline bool lockdep_assert_rcu_helper(bool c, const struct __ctx
 	WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map) &&			\
 					       !lock_is_held(&rcu_bh_lock_map) &&		\
 					       !lock_is_held(&rcu_sched_lock_map) &&		\
-					       preemptible(), RCU))
+					       preemptible()))
 
 #else /* #ifdef CONFIG_PROVE_RCU */
 
 #define RCU_LOCKDEP_WARN(c, s) do { } while (0 && (c))
 #define rcu_sleep_check() do { } while (0)
 
-#define lockdep_assert_in_rcu_read_lock() __assume_shared_ctx_lock(RCU)
-#define lockdep_assert_in_rcu_read_lock_bh() __assume_shared_ctx_lock(RCU_BH)
-#define lockdep_assert_in_rcu_read_lock_sched() __assume_shared_ctx_lock(RCU_SCHED)
-#define lockdep_assert_in_rcu_reader() __assume_shared_ctx_lock(RCU)
+#define lockdep_assert_in_rcu_read_lock() do { } while (0)
+#define lockdep_assert_in_rcu_read_lock_bh() do { } while (0)
+#define lockdep_assert_in_rcu_read_lock_sched() do { } while (0)
+#define lockdep_assert_in_rcu_reader() do { } while (0)
 
 #endif /* #else #ifdef CONFIG_PROVE_RCU */
 
+/**
+ * lockdep_assume_in_rcu_read_lock() - assume and verify RCU protection
+ *
+ * Instruct the compiler's context analysis to assume that the RCU read lock
+ * is held from this point on and verify at runtime with
+ * lockdep_assert_in_rcu_read_lock() that rcu_read_lock() is in effect. Use
+ * this macro in functions that are called with rcu_read_lock() held but for
+ * which a __must_hold_shared(RCU) annotation is not appropriate.
+ */
+#define lockdep_assume_in_rcu_read_lock()		\
+	do {						\
+		__assume_shared_ctx_lock(RCU);		\
+		lockdep_assert_in_rcu_read_lock();	\
+	} while (0)
+
+/**
+ * lockdep_assume_in_rcu_read_lock_bh() - assume and verify RCU-bh protection
+ *
+ * Instruct the compiler's context analysis to assume that both RCU and
+ * RCU_BH are held from this point on and verify at runtime with
+ * lockdep_assert_in_rcu_read_lock_bh() that rcu_read_lock_bh() is in
+ * effect. RCU is assumed too because rcu_read_lock_bh() acquires both RCU
+ * and RCU_BH, and because __rcu_guarded pointers are guarded by RCU.
+ */
+#define lockdep_assume_in_rcu_read_lock_bh()		\
+	do {						\
+		__assume_shared_ctx_lock(RCU);		\
+		__assume_shared_ctx_lock(RCU_BH);	\
+		lockdep_assert_in_rcu_read_lock_bh();	\
+	} while (0)
+
+/**
+ * lockdep_assume_in_rcu_read_lock_sched() - assume and verify RCU-sched
+ *
+ * Instruct the compiler's context analysis to assume that both RCU and
+ * RCU_SCHED are held from this point on and verify at runtime with
+ * lockdep_assert_in_rcu_read_lock_sched() that rcu_read_lock_sched() is in
+ * effect. RCU is assumed too because rcu_read_lock_sched() acquires both
+ * RCU and RCU_SCHED, and because __rcu_guarded pointers are guarded by RCU.
+ */
+#define lockdep_assume_in_rcu_read_lock_sched()			\
+	do {							\
+		__assume_shared_ctx_lock(RCU);			\
+		__assume_shared_ctx_lock(RCU_SCHED);		\
+		lockdep_assert_in_rcu_read_lock_sched();	\
+	} while (0)
+
+/**
+ * lockdep_assume_in_rcu_reader() - assume and verify any RCU reader
+ *
+ * Instruct the compiler's context analysis to assume that the RCU read lock
+ * is held from this point on and verify at runtime with
+ * lockdep_assert_in_rcu_reader() that some type of RCU reader is in effect.
+ * Unlike lockdep_assume_in_rcu_read_lock(), this macro also accepts regions of
+ * code in which preemption, softirqs or interrupts are disabled, since such
+ * regions also qualify as RCU readers.
+ */
+#define lockdep_assume_in_rcu_reader()			\
+	do {						\
+		__assume_shared_ctx_lock(RCU);		\
+		lockdep_assert_in_rcu_reader();		\
+	} while (0)
+
 /*
  * Helper functions for rcu_dereference_check(), rcu_dereference_protected()
  * and rcu_assign_pointer().  Some of these could be folded into their
diff --git a/lib/test_context-analysis.c b/lib/test_context-analysis.c
index 048b839bae92..2b5e8612c14e 100644
--- a/lib/test_context-analysis.c
+++ b/lib/test_context-analysis.c
@@ -466,25 +466,27 @@ static void __used test_rcu_lock_reentrant(void)
 
 static void __used test_rcu_assert(void)
 {
-	lockdep_assert_in_rcu_read_lock();
+	lockdep_assume_in_rcu_read_lock();
 	wants_rcu_held();
 }
 
 static void __used test_rcu_assert_bh(void)
 {
-	lockdep_assert_in_rcu_read_lock_bh();
+	lockdep_assume_in_rcu_read_lock_bh();
+	wants_rcu_held();
 	wants_rcu_held_bh();
 }
 
 static void __used test_rcu_assert_sched(void)
 {
-	lockdep_assert_in_rcu_read_lock_sched();
+	lockdep_assume_in_rcu_read_lock_sched();
+	wants_rcu_held();
 	wants_rcu_held_sched();
 }
 
 static void __used test_rcu_assert_reader(void)
 {
-	lockdep_assert_in_rcu_reader();
+	lockdep_assume_in_rcu_reader();
 	wants_rcu_held();
 }