Re: [PATCH] mm/mmu_notifier: Remove non_block_start/end() from notifier invocation

[David Woodhouse]

On Wed, 2026-08-12 at 15:05 +0100, David Woodhouse wrote:
> I'll rephrase that for my own understanding:
> 
> *If* we go all the way to building a whole SRCU flavour for this *and*
> implementing a spin-only variant of srcu_synchronize() which is
> tailored to the atomic-reader use case, *then* we don't need to remove
> the non_block_{start,end} guards around the MMU notifiers, which are
> basically never being called anyway and don't actually seem to protect
> against any real bugs.
> 
> Yes?

FWIW it looks something like this. I'll throw it into my torture and
latency tests, and we can see what Paul thinks of it. I'm still utterly
unconvinced it's needed, but I concede it has its good points.

From f912e839bf55759a1969e38d9e7897a30967fdb1 Mon Sep 17 00:00:00 2001
From: David Woodhouse <dwmw@xxxxxxxxxxxx>
Date: Wed, 12 Aug 2026 15:29:12 +0100
Subject: [PATCH 1/2] srcu: Add an ATOMIC reader flavor and a spinning
 synchronize_srcu_atomic()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Some dedicated srcu_struct users have read-side critical sections
which are short, never sleep, and never block on anything which may
itself depend on memory allocation — because they were, until
recently, spinlock or rwlock critical sections. For such a domain the
update side can safely wait for readers by spinning, from contexts
where sleeping is undesirable or the grace-period machinery's
latency (workqueue scheduling, jiffy-paced retries) dominates the
actual reader drain time.

But that is only safe if every reader keeps the promise. Make the
promise explicit and machine-checkable:

 - srcu_read_lock_atomic() / srcu_read_unlock_atomic() enter the
   usual (smp_mb-based) read-side critical section with preemption
   disabled and (except in hardirq, where it is redundant)
   non_block_start() armed, recording SRCU_READ_FLAVOR_ATOMIC in the
   per-CPU reader flavor. Disabling preemption enforces the
   no-sleeping promise on every configuration and bounds the section,
   so it is always running on some CPU; non_block_start() extends the
   enforcement to even potentially-sleeping calls on paths which
   happen not to block. The existing reader-flavor consistency checks
   complain about any mixing with other flavors.

 - synchronize_srcu_atomic() waits for all pre-existing readers by
   repeating the try_synchronize_srcu() both-epoch counter proof with
   cpu_relax() until it succeeds: no sleeping, no index flip, no
   grace-period sequence update, and therefore no interaction with
   concurrent call_srcu(), synchronize_srcu() or srcu_barrier(). It
   always provides the full grace-period guarantee: if the domain
   turns out to have had readers of any other flavor — a caller bug,
   since such a reader may be asleep and spinning on it would be
   unbounded — it complains and falls back to a real (sleeping) grace
   period internally, that being the only correct wait for a
   possibly-sleeping reader. The flavor mask is rechecked on every
   iteration so a first non-atomic reader appearing mid-spin takes
   the same path.

On Tiny SRCU (!SMP), spinning cannot be sane — an observed mid-section
reader can only make progress if we yield — but it is also never
needed: an atomic-flavor reader cannot be observed mid-section from
process context on the sole CPU, so the try_synchronize_srcu() proof
either succeeds immediately or the fallback is required anyway.

The immediate motivation is the proposed conversion of KVM's
gfn_to_pfn_cache to SRCU¹, whose mmu_notifier invalidation path drains
readers before the primary MMU zaps a page. With the readers declared
atomic, that drain becomes spin-only: no sleeping at all in the
notifier, bounded by the longest reader section, satisfying even the
strictest reading of the OOM-reaper non-blocking requirement without
needing to touch the non_block_start() annotation².

¹ https://lore.kernel.org/all/20260811132237.102400-1-dwmw2@xxxxxxxxxxxxx/
² https://lore.kernel.org/all/20260812134934.GC662699@xxxxxxxx/

Signed-off-by: David Woodhouse <dwmw@xxxxxxxxxxxx>
Assisted-by: Claude:claude-mythos-5
---
 include/linux/srcu.h  | 59 ++++++++++++++++++++++++++++++++++++++-
 kernel/rcu/srcutiny.c | 21 ++++++++++++++
 kernel/rcu/srcutree.c | 65 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 144 insertions(+), 1 deletion(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 0d7543c7becc..c81dfd0896ad 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -71,8 +71,10 @@ int init_srcu_struct_fast_updown(struct srcu_struct *ssp);
 //					0x4		// SRCU-lite is no longer with us.
 #define SRCU_READ_FLAVOR_FAST		0x4		// srcu_read_lock_fast(), also NMI-safe.
 #define SRCU_READ_FLAVOR_FAST_UPDOWN	0x8		// srcu_read_lock_fast_updown().
+#define SRCU_READ_FLAVOR_ATOMIC		0x10		// srcu_read_lock_atomic().
 #define SRCU_READ_FLAVOR_ALL		(SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \
-					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
+					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN | \
+					 SRCU_READ_FLAVOR_ATOMIC)
 						// All of the above.
 #define SRCU_READ_FLAVOR_SLOWGP		(SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
 						// Flavors requiring synchronize_rcu()
@@ -92,6 +94,7 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
 void cleanup_srcu_struct(struct srcu_struct *ssp);
 void synchronize_srcu(struct srcu_struct *ssp);
 bool try_synchronize_srcu(struct srcu_struct *ssp);
+void synchronize_srcu_atomic(struct srcu_struct *ssp);
 
 #define SRCU_GET_STATE_COMPLETED 0x1
 
@@ -296,6 +299,43 @@ static inline int srcu_read_lock(struct srcu_struct *ssp)
 	return retval;
 }
 
+/**
+ * srcu_read_lock_atomic - register a new reader promising an atomic section
+ * @ssp: srcu_struct in which to register the new reader.
+ *
+ * As srcu_read_lock(), but the caller promises that the read-side
+ * critical section never sleeps and never blocks on anything which
+ * may itself depend on memory allocation to make progress. Preemption
+ * is disabled for the duration, which both enforces that promise (any
+ * sleepable call in the section will splat on every configuration)
+ * and bounds the section so that the update side may spin rather
+ * than sleep when waiting for readers: see synchronize_srcu_atomic().
+ *
+ * The lock and matching srcu_read_unlock_atomic() must be invoked on
+ * the same CPU, from the same context; passing the return value to
+ * another task is not permitted for this flavor.
+ */
+static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
+	__acquires_shared(ssp)
+{
+	int retval;
+
+	preempt_disable();
+	/*
+	 * Arm might_sleep() to catch even a *potentially* sleeping call
+	 * in the section, not just an actual schedule: the atomic-domain
+	 * promise must hold on every path, contended or not. In hardirq
+	 * the annotation would land on the interrupted task; it is also
+	 * redundant there, so skip it.
+	 */
+	if (!in_hardirq())
+		non_block_start();
+	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
+	retval = __srcu_read_lock(ssp);
+	srcu_lock_acquire(&ssp->dep_map);
+	return retval;
+}
+
 /**
  * srcu_read_lock_fast - register a new reader for an SRCU-protected structure.
  * @ssp: srcu_struct in which to register the new reader.
@@ -488,6 +528,23 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)
 	__srcu_read_unlock(ssp, idx);
 }
 
+/**
+ * srcu_read_unlock_atomic - unregister an atomic-section reader
+ * @ssp: srcu_struct from which to unregister the old reader.
+ * @idx: return value from corresponding srcu_read_lock_atomic().
+ */
+static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
+	__releases_shared(ssp)
+{
+	WARN_ON_ONCE(idx & ~0x1);
+	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
+	srcu_lock_release(&ssp->dep_map);
+	__srcu_read_unlock(ssp, idx);
+	if (!in_hardirq())
+		non_block_end();
+	preempt_enable();
+}
+
 /**
  * srcu_read_unlock_fast - unregister a old reader from an SRCU-protected structure.
  * @ssp: srcu_struct in which to unregister the old reader.
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
index f07159cae241..c04112d9c332 100644
--- a/kernel/rcu/srcutiny.c
+++ b/kernel/rcu/srcutiny.c
@@ -283,6 +283,27 @@ bool try_synchronize_srcu(struct srcu_struct *ssp)
 }
 EXPORT_SYMBOL_GPL(try_synchronize_srcu);
 
+/**
+ * synchronize_srcu_atomic - spinning grace period for atomic-reader domains
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * On !SMP this cannot spin: a reader observed mid-section is preempted
+ * or interrupted-out, and can only finish if we yield the CPU. But it
+ * is also never needed: an atomic-flavor reader (preemption disabled)
+ * cannot be observed mid-section from process context on the sole CPU.
+ * So a reader observed here has broken the atomic-domain promise, and
+ * the only correct wait for it is a real grace period.
+ */
+void synchronize_srcu_atomic(struct srcu_struct *ssp)
+{
+	if (try_synchronize_srcu(ssp))
+		return;
+	WARN_ONCE(1, "non-atomic readers on srcu_struct at %ps; falling back to sleeping grace period\n",
+		  ssp);
+	synchronize_srcu(ssp);
+}
+EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
+
 /*
  * get_state_synchronize_srcu - Provide an end-of-grace-period cookie
  */
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 76a78336feaa..5fa10c07cb3f 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1715,6 +1715,71 @@ bool try_synchronize_srcu(struct srcu_struct *ssp)
 }
 EXPORT_SYMBOL_GPL(try_synchronize_srcu);
 
+/**
+ * synchronize_srcu_atomic - spinning grace period for atomic-reader domains
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * Wait for all pre-existing readers of @ssp to complete, without
+ * sleeping and without involving the SRCU grace-period machinery —
+ * provided the domain keeps the promise that all of its readers use
+ * srcu_read_lock_atomic().
+ *
+ * Atomic-flavor readers run with preemption disabled, so every
+ * read-side critical section is bounded and running (not blocked) on
+ * some CPU, which is what makes spinning here sane: the wait is
+ * bounded by the longest such section. The proof of reader absence is
+ * the same both-epoch counter comparison as try_synchronize_srcu(),
+ * repeated until it succeeds; no index flip and no grace-period
+ * sequence update occur, so concurrent call_srcu(), synchronize_srcu()
+ * and srcu_barrier() are entirely unaffected.
+ *
+ * If the domain has ever had readers of any other flavor, the promise
+ * is broken — such a reader may be asleep, and spinning on it would be
+ * unbounded. That is a caller bug: complain, and fall back to a real
+ * (sleeping) grace period, which is the only correct wait for a
+ * possibly-sleeping reader. The reader-flavor mask is rechecked on
+ * every iteration so a first non-atomic reader appearing mid-spin
+ * takes the same path.
+ */
+void synchronize_srcu_atomic(struct srcu_struct *ssp)
+{
+	unsigned long unlocks0, unlocks1;
+	unsigned long rdm0, rdm1;
+
+	check_init_srcu_struct(ssp);
+
+	/*
+	 * Order the caller's prior stores before the counter reads;
+	 * pairs with the smp_mb() in __srcu_read_lock() as described
+	 * in try_synchronize_srcu().
+	 */
+	smp_mb();
+
+	for (;;) {
+		unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
+		unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
+
+		if (WARN_ONCE((rdm0 | rdm1) & ~SRCU_READ_FLAVOR_ATOMIC,
+			      "non-atomic readers on srcu_struct at %ps; falling back to sleeping grace period\n",
+			      ssp)) {
+			synchronize_srcu_expedited(ssp);
+			return;
+		}
+
+		smp_mb();
+
+		if (srcu_readers_lock_idx(ssp, 0, false, unlocks0) &&
+		    srcu_readers_lock_idx(ssp, 1, false, unlocks1))
+			break;
+
+		cpu_relax();
+	}
+
+	/* Order the caller's subsequent accesses after the proof. */
+	smp_mb();
+}
+EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
+
 /**
  * get_state_synchronize_srcu - Provide an end-of-grace-period cookie
  * @ssp: srcu_struct to provide cookie for.
-- 
2.43.0

Attachment: smime.p7s
Description: S/MIME cryptographic signature