Re: [PATCH] mm/mmu_notifier: Remove non_block_start/end() from notifier invocation

[David Woodhouse]

On Wed, 2026-08-12 at 15:34 +0100, David Woodhouse wrote:
> On Wed, 2026-08-12 at 15:05 +0100, David Woodhouse wrote:
> > I'll rephrase that for my own understanding:
> > 
> > *If* we go all the way to building a whole SRCU flavour for this *and*
> > implementing a spin-only variant of srcu_synchronize() which is
> > tailored to the atomic-reader use case, *then* we don't need to remove
> > the non_block_{start,end} guards around the MMU notifiers, which are
> > basically never being called anyway and don't actually seem to protect
> > against any real bugs.
> > 
> > Yes?
> 
> FWIW it looks something like this. I'll throw it into my torture and
> latency tests, and we can see what Paul thinks of it. I'm still utterly
> unconvinced it's needed, but I concede it has its good points.

This slightly refactored version is the one that's actually going into
my torture tests...

I'll defer to Paul here. If we think this whole srcu_read_lock_atomic
thing is really worth it for SRCU in the general case — perhaps we can
use it to convert existing rwlock_t users, some of whom have been also
blindsided by those becoming sleepable on RT — then I guess it makes
sense.

I can't honestly defend the added complexity *purely* for the KVM GPC
use case though; I just can't bring myself to lose sleep over the use
of the wait queue from the MMU notifier even in the OOM reaper path.
It's *latency* that matters.

From f09baac8dc596596018e7009c15a1cd73cdfd421 Mon Sep 17 00:00:00 2001
From: David Woodhouse <dwmw@xxxxxxxxxxxx>
Date: Wed, 12 Aug 2026 15:29:12 +0100
Subject: [PATCH 1/2] srcu: Add an ATOMIC reader flavor and a spinning
 synchronize_srcu_atomic()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Some dedicated srcu_struct users have read-side critical sections
which are short, never sleep, and never block on anything which may
itself depend on memory allocation — because they were, until
recently, spinlock or rwlock critical sections. For such a domain the
update side can safely wait for readers by spinning, from contexts
where sleeping is undesirable or the grace-period machinery's
latency (workqueue scheduling, jiffy-paced retries) dominates the
actual reader drain time.

But that is only safe if every reader keeps the promise. Make the
promise explicit and machine-checkable:

 - srcu_read_lock_atomic() / srcu_read_unlock_atomic() enter the
   usual (smp_mb-based) read-side critical section with preemption
   disabled and (except in hardirq, where it is redundant)
   non_block_start() armed, recording SRCU_READ_FLAVOR_ATOMIC in the
   per-CPU reader flavor. Disabling preemption enforces the
   no-sleeping promise on every configuration and bounds the section,
   so it is always running on some CPU; non_block_start() extends the
   enforcement to even potentially-sleeping calls on paths which
   happen not to block. The existing reader-flavor consistency checks
   complain about any mixing with other flavors.

 - synchronize_srcu_atomic() waits for all pre-existing readers by
   repeating the try_synchronize_srcu() both-epoch counter proof with
   cpu_relax() until it succeeds: no sleeping, no index flip, no
   grace-period sequence update, and therefore no interaction with
   concurrent call_srcu(), synchronize_srcu() or srcu_barrier(). It
   always provides the full grace-period guarantee: if the domain
   turns out to have had readers of any other flavor — a caller bug,
   since such a reader may be asleep and spinning on it would be
   unbounded — it complains and falls back to a real (sleeping) grace
   period internally, that being the only correct wait for a
   possibly-sleeping reader. The flavor mask is rechecked on every
   iteration so a first non-atomic reader appearing mid-spin takes
   the same path.

On Tiny SRCU (!SMP), spinning cannot be sane — an observed mid-section
reader can only make progress if we yield — but it is also never
needed: an atomic-flavor reader cannot be observed mid-section from
process context on the sole CPU, so the try_synchronize_srcu() proof
either succeeds immediately or the fallback is required anyway.

The immediate motivation is the proposed conversion of KVM's
gfn_to_pfn_cache to SRCU¹, whose mmu_notifier invalidation path drains
readers before the primary MMU zaps a page. With the readers declared
atomic, that drain becomes spin-only: no sleeping at all in the
notifier, bounded by the longest reader section, satisfying even the
strictest reading of the OOM-reaper non-blocking requirement without
needing to touch the non_block_start() annotation².

¹ https://lore.kernel.org/all/20260811132237.102400-1-dwmw2@xxxxxxxxxxxxx/
² https://lore.kernel.org/all/20260812134934.GC662699@xxxxxxxx/

Signed-off-by: David Woodhouse <dwmw@xxxxxxxxxxxx>
Assisted-by: Claude:claude-mythos-5
---
 include/linux/srcu.h  |  59 +++++++++++++++++-
 kernel/rcu/srcutiny.c |  21 +++++++
 kernel/rcu/srcutree.c | 135 ++++++++++++++++++++++++++++++------------
 3 files changed, 175 insertions(+), 40 deletions(-)

diff --git a/include/linux/srcu.h b/include/linux/srcu.h
index 0d7543c7becc..c81dfd0896ad 100644
--- a/include/linux/srcu.h
+++ b/include/linux/srcu.h
@@ -71,8 +71,10 @@ int init_srcu_struct_fast_updown(struct srcu_struct *ssp);
 //					0x4		// SRCU-lite is no longer with us.
 #define SRCU_READ_FLAVOR_FAST		0x4		// srcu_read_lock_fast(), also NMI-safe.
 #define SRCU_READ_FLAVOR_FAST_UPDOWN	0x8		// srcu_read_lock_fast_updown().
+#define SRCU_READ_FLAVOR_ATOMIC		0x10		// srcu_read_lock_atomic().
 #define SRCU_READ_FLAVOR_ALL		(SRCU_READ_FLAVOR_NORMAL | SRCU_READ_FLAVOR_NMI | \
-					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
+					 SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN | \
+					 SRCU_READ_FLAVOR_ATOMIC)
 						// All of the above.
 #define SRCU_READ_FLAVOR_SLOWGP		(SRCU_READ_FLAVOR_FAST | SRCU_READ_FLAVOR_FAST_UPDOWN)
 						// Flavors requiring synchronize_rcu()
@@ -92,6 +94,7 @@ void call_srcu(struct srcu_struct *ssp, struct rcu_head *head,
 void cleanup_srcu_struct(struct srcu_struct *ssp);
 void synchronize_srcu(struct srcu_struct *ssp);
 bool try_synchronize_srcu(struct srcu_struct *ssp);
+void synchronize_srcu_atomic(struct srcu_struct *ssp);
 
 #define SRCU_GET_STATE_COMPLETED 0x1
 
@@ -296,6 +299,43 @@ static inline int srcu_read_lock(struct srcu_struct *ssp)
 	return retval;
 }
 
+/**
+ * srcu_read_lock_atomic - register a new reader promising an atomic section
+ * @ssp: srcu_struct in which to register the new reader.
+ *
+ * As srcu_read_lock(), but the caller promises that the read-side
+ * critical section never sleeps and never blocks on anything which
+ * may itself depend on memory allocation to make progress. Preemption
+ * is disabled for the duration, which both enforces that promise (any
+ * sleepable call in the section will splat on every configuration)
+ * and bounds the section so that the update side may spin rather
+ * than sleep when waiting for readers: see synchronize_srcu_atomic().
+ *
+ * The lock and matching srcu_read_unlock_atomic() must be invoked on
+ * the same CPU, from the same context; passing the return value to
+ * another task is not permitted for this flavor.
+ */
+static inline int srcu_read_lock_atomic(struct srcu_struct *ssp)
+	__acquires_shared(ssp)
+{
+	int retval;
+
+	preempt_disable();
+	/*
+	 * Arm might_sleep() to catch even a *potentially* sleeping call
+	 * in the section, not just an actual schedule: the atomic-domain
+	 * promise must hold on every path, contended or not. In hardirq
+	 * the annotation would land on the interrupted task; it is also
+	 * redundant there, so skip it.
+	 */
+	if (!in_hardirq())
+		non_block_start();
+	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
+	retval = __srcu_read_lock(ssp);
+	srcu_lock_acquire(&ssp->dep_map);
+	return retval;
+}
+
 /**
  * srcu_read_lock_fast - register a new reader for an SRCU-protected structure.
  * @ssp: srcu_struct in which to register the new reader.
@@ -488,6 +528,23 @@ static inline void srcu_read_unlock(struct srcu_struct *ssp, int idx)
 	__srcu_read_unlock(ssp, idx);
 }
 
+/**
+ * srcu_read_unlock_atomic - unregister an atomic-section reader
+ * @ssp: srcu_struct from which to unregister the old reader.
+ * @idx: return value from corresponding srcu_read_lock_atomic().
+ */
+static inline void srcu_read_unlock_atomic(struct srcu_struct *ssp, int idx)
+	__releases_shared(ssp)
+{
+	WARN_ON_ONCE(idx & ~0x1);
+	srcu_check_read_flavor(ssp, SRCU_READ_FLAVOR_ATOMIC);
+	srcu_lock_release(&ssp->dep_map);
+	__srcu_read_unlock(ssp, idx);
+	if (!in_hardirq())
+		non_block_end();
+	preempt_enable();
+}
+
 /**
  * srcu_read_unlock_fast - unregister a old reader from an SRCU-protected structure.
  * @ssp: srcu_struct in which to unregister the old reader.
diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c
index f07159cae241..c04112d9c332 100644
--- a/kernel/rcu/srcutiny.c
+++ b/kernel/rcu/srcutiny.c
@@ -283,6 +283,27 @@ bool try_synchronize_srcu(struct srcu_struct *ssp)
 }
 EXPORT_SYMBOL_GPL(try_synchronize_srcu);
 
+/**
+ * synchronize_srcu_atomic - spinning grace period for atomic-reader domains
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * On !SMP this cannot spin: a reader observed mid-section is preempted
+ * or interrupted-out, and can only finish if we yield the CPU. But it
+ * is also never needed: an atomic-flavor reader (preemption disabled)
+ * cannot be observed mid-section from process context on the sole CPU.
+ * So a reader observed here has broken the atomic-domain promise, and
+ * the only correct wait for it is a real grace period.
+ */
+void synchronize_srcu_atomic(struct srcu_struct *ssp)
+{
+	if (try_synchronize_srcu(ssp))
+		return;
+	WARN_ONCE(1, "non-atomic readers on srcu_struct at %ps; falling back to sleeping grace period\n",
+		  ssp);
+	synchronize_srcu(ssp);
+}
+EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
+
 /*
  * get_state_synchronize_srcu - Provide an end-of-grace-period cookie
  */
diff --git a/kernel/rcu/srcutree.c b/kernel/rcu/srcutree.c
index 76a78336feaa..3cafb1c2160b 100644
--- a/kernel/rcu/srcutree.c
+++ b/kernel/rcu/srcutree.c
@@ -1645,6 +1645,47 @@ void synchronize_srcu(struct srcu_struct *ssp)
 }
 EXPORT_SYMBOL_GPL(synchronize_srcu);
 
+/*
+ * Attempt to prove that @ssp has no readers in either epoch. Returns:
+ *   1  - proof succeeded: no pre-existing readers, sums stable.
+ *   0  - readers (or counter movement) observed; try again later.
+ *   -1 - the domain has reader flavors incompatible with @allowed
+ *        (those which elide the read-side smp_mb() that the proof
+ *        depends on, or, for the spinning caller, those which may
+ *        sleep); the caller needs a real grace period.
+ *
+ * The caller must provide the leading smp_mb() ordering its prior
+ * stores before this proof (pairing with the smp_mb() in
+ * __srcu_read_lock()), and is responsible for ordering its subsequent
+ * accesses after a successful proof.
+ */
+static int srcu_readers_provably_absent(struct srcu_struct *ssp, unsigned long allowed)
+{
+	unsigned long unlocks0, unlocks1;
+	unsigned long rdm0, rdm1;
+
+	unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
+	unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
+
+	if ((rdm0 | rdm1) & ~allowed)
+		return -1;
+
+	/*
+	 * As in srcu_readers_active_idx_check(): ensure that a lock is
+	 * always counted if the corresponding unlock is counted, so that
+	 * a reader racing with these sums can only inflate the lock sum
+	 * and force the (safe) retry or fallback. Summing both epochs
+	 * means no index flip is needed: a stable equality proves there
+	 * was a moment at which no readers existed at all.
+	 */
+	smp_mb();
+
+	if (srcu_readers_lock_idx(ssp, 0, false, unlocks0) &&
+	    srcu_readers_lock_idx(ssp, 1, false, unlocks1))
+		return 1;
+	return 0;
+}
+
 /**
  * try_synchronize_srcu - inline grace period for a reader-free srcu_struct
  * @ssp: srcu_struct with which to synchronize.
@@ -1656,64 +1697,80 @@ EXPORT_SYMBOL_GPL(synchronize_srcu);
  * failure the caller must fall back to synchronize_srcu() or
  * synchronize_srcu_expedited().
  *
- * This serves dedicated srcu_struct structures whose read-side critical
- * sections are short, atomic, and usually absent — where even an
- * expedited grace period costs two trips through the workqueue and an
- * unconditional sleep of the caller, three orders of magnitude more
- * than the check below.
- *
- * Only readers of the srcu_read_lock() and srcu_read_lock_nmisafe()
- * flavors are compatible with this proof; if the _fast() flavors have
- * ever been used on @ssp, this function always returns false.
+ * Only readers of the flavors which include a read-side smp_mb() are
+ * compatible with this proof; if the _fast() flavors have ever been
+ * used on @ssp, this function always returns false.
  */
 bool try_synchronize_srcu(struct srcu_struct *ssp)
 {
-	unsigned long unlocks0, unlocks1;
-	unsigned long rdm0, rdm1;
-
 	check_init_srcu_struct(ssp);
 
 	/*
-	 * Order the caller's prior stores before the counter reads below.
-	 * Pairs (store-buffering pattern) with the smp_mb() in
-	 * __srcu_read_lock(): any reader whose lock increment is not
-	 * observed by the sums below is guaranteed to observe, within its
+	 * Order the caller's prior stores before the counter reads in
+	 * the proof. Pairs (store-buffering pattern) with the smp_mb()
+	 * in __srcu_read_lock(): any reader whose lock increment is not
+	 * observed by the sums is guaranteed to observe, within its
 	 * critical section, every store the caller made before calling
 	 * this function.
 	 */
 	smp_mb();
 
-	unlocks0 = srcu_readers_unlock_idx(ssp, 0, &rdm0);
-	unlocks1 = srcu_readers_unlock_idx(ssp, 1, &rdm1);
-
-	/*
-	 * Reader flavors which elide the read-side smp_mb() that the
-	 * pairing above depends on cannot be proven absent this way;
-	 * they need a real grace period.
-	 */
-	if ((rdm0 | rdm1) & SRCU_READ_FLAVOR_SLOWGP)
+	if (srcu_readers_provably_absent(ssp, ~SRCU_READ_FLAVOR_SLOWGP) != 1)
 		return false;
 
-	/*
-	 * As in srcu_readers_active_idx_check(): ensure that a lock is
-	 * always counted if the corresponding unlock is counted, so that
-	 * a reader racing with these sums can only inflate the lock sum
-	 * and force the (safe) fallback. Summing both epochs means no
-	 * index flip is needed: a stable equality proves there was a
-	 * moment in this function at which no readers existed at all.
-	 */
+	/* Order the caller's subsequent accesses after the proof. */
 	smp_mb();
+	return true;
+}
+EXPORT_SYMBOL_GPL(try_synchronize_srcu);
 
-	if (!srcu_readers_lock_idx(ssp, 0, false, unlocks0))
-		return false;
-	if (!srcu_readers_lock_idx(ssp, 1, false, unlocks1))
-		return false;
+/**
+ * synchronize_srcu_atomic - spinning grace period for atomic-reader domains
+ * @ssp: srcu_struct with which to synchronize.
+ *
+ * Wait for all pre-existing readers of @ssp to complete, without
+ * sleeping and without involving the SRCU grace-period machinery —
+ * provided the domain keeps the promise that all of its readers use
+ * srcu_read_lock_atomic().
+ *
+ * Atomic-flavor readers run with preemption disabled, so every
+ * read-side critical section is bounded and running (not blocked) on
+ * some CPU, which is what makes spinning here sane: the wait is
+ * bounded by the longest such section. No index flip and no
+ * grace-period sequence update occur, so concurrent call_srcu(),
+ * synchronize_srcu() and srcu_barrier() are entirely unaffected.
+ *
+ * If the domain has ever had readers of any other flavor, the promise
+ * is broken — such a reader may be asleep, and spinning on it would be
+ * unbounded. That is a caller bug: complain, and fall back to a real
+ * (sleeping) grace period, which is the only correct wait for a
+ * possibly-sleeping reader. The reader-flavor mask is rechecked on
+ * every iteration so a first non-atomic reader appearing mid-spin
+ * takes the same path.
+ */
+void synchronize_srcu_atomic(struct srcu_struct *ssp)
+{
+	int ret;
+
+	check_init_srcu_struct(ssp);
+
+	/* As in try_synchronize_srcu(). */
+	smp_mb();
+
+	while ((ret = srcu_readers_provably_absent(ssp, SRCU_READ_FLAVOR_ATOMIC)) == 0)
+		cpu_relax();
+
+	if (WARN_ONCE(ret < 0,
+		      "non-atomic readers on srcu_struct at %ps; falling back to sleeping grace period\n",
+		      ssp)) {
+		synchronize_srcu_expedited(ssp);
+		return;
+	}
 
 	/* Order the caller's subsequent accesses after the proof. */
 	smp_mb();
-	return true;
 }
-EXPORT_SYMBOL_GPL(try_synchronize_srcu);
+EXPORT_SYMBOL_GPL(synchronize_srcu_atomic);
 
 /**
  * get_state_synchronize_srcu - Provide an end-of-grace-period cookie
-- 
2.43.0

Attachment: smime.p7s
Description: S/MIME cryptographic signature