[PATCH tip/core/rcu 5/9] doc: Update requirements based on recent changes

From: Paul E. McKenney
Date: Wed Apr 12 2017 - 12:41:18 EST

Next message: Paul E. McKenney: "[PATCH tip/core/rcu 9/9] doc: Update control-dependencies section of memory-barriers.txt"
Previous message: Joe Perches: "Re: [PATCH V14 02/10] ras: acpi/apei: cper: generic error data entry v3 per ACPI 6.1"
In reply to: Paul E. McKenney: "[PATCH tip/core/rcu 6/9] doc: Update rcu_assign_pointer() definition in whatisRCU.txt"
Next in thread: Paul E. McKenney: "[PATCH tip/core/rcu 9/9] doc: Update control-dependencies section of memory-barriers.txt"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

These changes include lighter-weight expedited grace periods, the fact
that expedited grace periods and rcu_barrier() no longer block CPU
hotplug, some HTML font fixups, noting that rcu_barrier() need not wait
for a grace period (even if callbacks are posted), the fact that SRCU
read-side critical sections can be used from offline CPUs, and the fact
that SRCU now maintains per-CPU callback lists.

Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>
---
.../RCU/Design/Requirements/Requirements.html | 120 ++++++++++++++++-----
1 file changed, 94 insertions(+), 26 deletions(-)

diff --git a/Documentation/RCU/Design/Requirements/Requirements.html b/Documentation/RCU/Design/Requirements/Requirements.html
index 999b3ed3444e..f60adf112663 100644
--- a/Documentation/RCU/Design/Requirements/Requirements.html
+++ b/Documentation/RCU/Design/Requirements/Requirements.html
@@ -659,8 +659,9 @@ systems with more than one CPU:
In other words, a given instance of <tt>synchronize_rcu()</tt>
can avoid waiting on a given RCU read-side critical section only
if it can prove that <tt>synchronize_rcu()</tt> started first.
+ 

- 
+ 
A related question is “When <tt>rcu_read_lock()</tt>
doesn't generate any code, why does it matter how it relates
to a grace period?”
@@ -675,8 +676,9 @@ systems with more than one CPU:
within the critical section, in which case none of the accesses
within the critical section may observe the effects of any
access following the grace period.
+ 

- 
+ 
As of late 2016, mathematical models of RCU take this
viewpoint, for example, see slides 62 and 63
of the
@@ -1616,8 +1618,8 @@ CPUs should at least make reasonable forward progress.
In return for its shorter latencies, <tt>synchronize_rcu_expedited()</tt>
is permitted to impose modest degradation of real-time latency
on non-idle online CPUs.
-That said, it will likely be necessary to take further steps to reduce this
-degradation, hopefully to roughly that of a scheduling-clock interrupt.
+Here, “modest” means roughly the same latency
+degradation as a scheduling-clock interrupt.


There are a number of situations where even
@@ -1913,12 +1915,9 @@ This requirement is another factor driving batching of grace periods,
but it is also the driving force behind the checks for large numbers
of queued RCU callbacks in the <tt>call_rcu()</tt> code path.
Finally, high update rates should not delay RCU read-side critical
-sections, although some read-side delays can occur when using
+sections, although some small read-side delays can occur when using
<tt>synchronize_rcu_expedited()</tt>, courtesy of this function's use
-of <tt>try_stop_cpus()</tt>.
-(In the future, <tt>synchronize_rcu_expedited()</tt> will be
-converted to use lighter-weight inter-processor interrupts (IPIs),
-but this will still disturb readers, though to a much smaller degree.)
+of <tt>smp_call_function_single()</tt>.


Although all three of these corner cases were understood in the early
@@ -2192,7 +2191,7 @@ Unfortunately, <tt>synchronize_rcu()</tt> can't do this until all of
its kthreads are spawned, which doesn't happen until some time during
<tt>early_initcalls()</tt> time.
But this is no excuse: RCU is nevertheless required to correctly handle
-synchronous grace periods during this time period, which it currently does.
+synchronous grace periods during this time period.
Once all of its kthreads are up and running, RCU starts running
normally.

@@ -2206,8 +2205,10 @@ normally.
<tr><th align="left">Answer:</th></tr>
<tr><td bgcolor="#ffffff">
Very carefully!
+ 

- During the “dead zone” between the time that the
+ 
+ During the “dead zone” between the time that the
scheduler spawns the first task and the time that all of RCU's
kthreads have been spawned, all synchronous grace periods are
handled by the expedited grace-period mechanism.
@@ -2220,8 +2221,10 @@ normally.
using workqueues, as is required to avoid problems that would
otherwise occur when a user task received a POSIX signal while
driving an expedited grace period.
+ 

- And yes, this does mean that it is unhelpful to send POSIX
+ 
+ And yes, this does mean that it is unhelpful to send POSIX
signals to random tasks between the time that the scheduler
spawns its first kthread and the time that RCU's kthreads
have all been spawned.
@@ -2308,12 +2311,61 @@ situation, and Dipankar Sarma incorporated <tt>rcu_barrier()</tt> into RCU.
The need for <tt>rcu_barrier()</tt> for module unloading became
apparent later.

+
+Important note: The <tt>rcu_barrier()</tt> function is not,
+repeat, not, obligated to wait for a grace period.
+It is instead only required to wait for RCU callbacks that have
+already been posted.
+Therefore, if there are no RCU callbacks posted anywhere in the system,
+<tt>rcu_barrier()</tt> is within its rights to return immediately.
+Even if there are callbacks posted, <tt>rcu_barrier()</tt> does not
+necessarily need to wait for a grace period.
+
+<table>
+<tr><th> </th></tr>
+<tr><th align="left">Quick Quiz:</th></tr>
+<tr><td>
+ Wait a minute!
+ Each RCU callbacks must wait for a grace period to complete,
+ and <tt>rcu_barrier()</tt> must wait for each pre-existing
+ callback to be invoked.
+ Doesn't <tt>rcu_barrier()</tt> therefore need to wait for
+ a full grace period if there is even one callback posted anywhere
+ in the system?
+</td></tr>
+<tr><th align="left">Answer:</th></tr>
+<tr><td bgcolor="#ffffff">
+ Absolutely not!!!
+ 
+
+ 
+ Yes, each RCU callbacks must wait for a grace period to complete,
+ but it might well be partly (or even completely) finished waiting
+ by the time <tt>rcu_barrier()</tt> is invoked.
+ In that case, <tt>rcu_barrier()</tt> need only wait for the
+ remaining portion of the grace period to elapse.
+ So even if there are quite a few callbacks posted,
+ <tt>rcu_barrier()</tt> might well return quite quickly.
+ 
+
+ 
+ So if you need to wait for a grace period as well as for all
+ pre-existing callbacks, you will need to invoke both
+ <tt>synchronize_rcu()</tt> and <tt>rcu_barrier()</tt>.
+ If latency is a concern, you can always use workqueues
+ to invoke them concurrently.
+</td></tr>
+<tr><td> </td></tr>
+</table>
+
<h3><a name="Hotplug CPU">Hotplug CPU</a></h3>


The Linux kernel supports CPU hotplug, which means that CPUs
can come and go.
-It is of course illegal to use any RCU API member from an offline CPU.
+It is of course illegal to use any RCU API member from an offline CPU,
+with the exception of <a href="#Sleepable RCU">SRCU</a> read-side
+critical sections.
This requirement was present from day one in DYNIX/ptx, but
on the other hand, the Linux kernel's CPU-hotplug implementation
is “interesting.”
@@ -2323,19 +2375,18 @@ The Linux-kernel CPU-hotplug implementation has notifiers that
are used to allow the various kernel subsystems (including RCU)
to respond appropriately to a given CPU-hotplug operation.
Most RCU operations may be invoked from CPU-hotplug notifiers,
-including even normal synchronous grace-period operations
-such as <tt>synchronize_rcu()</tt>.
-However, expedited grace-period operations such as
-<tt>synchronize_rcu_expedited()</tt> are not supported,
-due to the fact that current implementations block CPU-hotplug
-operations, which could result in deadlock.
+including even synchronous grace-period operations such as
+<tt>synchronize_rcu()</tt> and <tt>synchronize_rcu_expedited()</tt>.


-In addition, all-callback-wait operations such as
+However, all-callback-wait operations such as
<tt>rcu_barrier()</tt> are also not supported, due to the
fact that there are phases of CPU-hotplug operations where
the outgoing CPU's callbacks will not be invoked until after
the CPU-hotplug operation ends, which could also result in deadlock.
+Furthermore, <tt>rcu_barrier()</tt> blocks CPU-hotplug operations
+during its execution, which results in another type of deadlock
+when invoked from a CPU-hotplug notifier.

<h3><a name="Scheduler and RCU">Scheduler and RCU</a></h3>

@@ -2877,6 +2928,27 @@ API, which, in combination with <tt>srcu_read_unlock()</tt>,
guarantees a full memory barrier.


+Also unlike other RCU flavors, SRCU's callbacks-wait function
+<tt>srcu_barrier()</tt> may be invoked from CPU-hotplug notifiers,
+though this is not necessarily a good idea.
+The reason that this is possible is that SRCU is insensitive
+to whether or not a CPU is online, which means that <tt>srcu_barrier()</tt>
+need not exclude CPU-hotplug operations.
+
+
+As of v4.12, SRCU's callbacks are maintained per-CPU, eliminating
+a locking bottleneck present in prior kernel versions.
+Although this will allow users to put much heavier stress on
+<tt>call_srcu()</tt>, it is important to note that SRCU does not
+yet take any special steps to deal with callback flooding.
+So if you are posting (say) 10,000 SRCU callbacks per second per CPU,
+you are probably totally OK, but if you intend to post (say) 1,000,000
+SRCU callbacks per second per CPU, please run some tests first.
+SRCU just might need a few adjustment to deal with that sort of load.
+Of course, your mileage may vary based on the speed of your CPUs and
+the size of your memory.
+
+
The
<a href="https://lwn.net/Articles/609973/#RCU Per-Flavor API Table">SRCU API</a>
includes
@@ -3034,8 +3106,8 @@ to do some redesign to avoid this scalability problem.


RCU disables CPU hotplug in a few places, perhaps most notably in the
-expedited grace-period and <tt>rcu_barrier()</tt> operations.
-If there is a strong reason to use expedited grace periods in CPU-hotplug
+<tt>rcu_barrier()</tt> operations.
+If there is a strong reason to use <tt>rcu_barrier()</tt> in CPU-hotplug
notifiers, it will be necessary to avoid disabling CPU hotplug.
This would introduce some complexity, so there had better be a very
good reason.
@@ -3109,9 +3181,5 @@ Andy Lutomirski for their help in rendering
this article human readable, and to Michelle Rankin for her support
of this effort.
Other contributions are acknowledged in the Linux kernel's git archive.
-The cartoon is copyright (c) 2013 by Melissa Broussard,
-and is provided
-under the terms of the Creative Commons Attribution-Share Alike 3.0
-United States license.

</body></html>
--
2.5.2

Next message: Paul E. McKenney: "[PATCH tip/core/rcu 9/9] doc: Update control-dependencies section of memory-barriers.txt"
Previous message: Joe Perches: "Re: [PATCH V14 02/10] ras: acpi/apei: cper: generic error data entry v3 per ACPI 6.1"
In reply to: Paul E. McKenney: "[PATCH tip/core/rcu 6/9] doc: Update rcu_assign_pointer() definition in whatisRCU.txt"
Next in thread: Paul E. McKenney: "[PATCH tip/core/rcu 9/9] doc: Update control-dependencies section of memory-barriers.txt"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]