Re: [PATCH tip/core/rcu 04/13] rcu: Make RCU_FANOUT_LEAF help text more explicit about skew_tick

[Paul E. McKenney]

On Thu, Apr 13, 2017 at 08:23:09PM +0200, Peter Zijlstra wrote:
> On Thu, Apr 13, 2017 at 11:19:26AM -0700, Paul E. McKenney wrote:
> 
> > First get me some system-level data showing that the current layout is
> > causing a real problem.  RCU's fastpath code doesn't come anywhere near
> > the rcu_node tree, so in the absence of such data, I of course remain
> > quite doubtful that there is a real need.  And painfully aware of the
> > required increase in complexity.
> > 
> > But if there is a real need demonstrated by real system-level data,
> > I will of course make the needed changes, as I have done many times in
> > the past in response to other requests.
> 
> I read what you wrote here:
> 
> > > > Increasing it reduces the number of rcu_node structures, and thus the
> > > > number of cache misses during grace-period initialization and cleanup.
> > > > This has proven necessary in the past on large machines having long
> > > > memory latencies.  And there are starting to be some pretty big machines
> > > > running in production, and even for typical commerical workloads.
> 
> to mean you had exactly that pain. Or am I now totally not understanding
> you?

I believe that you are missing the fact that RCU grace-period
initialization and cleanup walks through the rcu_node tree breadth
first, using rcu_for_each_node_breadth_first().  This macro (shown below)
implements this breadth-first walk using a simple sequential traversal of
the ->node[] array that provides the structures making up the rcu_node
tree.  As you can see, this scan is completely independent of how CPU
numbers might be mapped to rcu_data slots in the leaf rcu_node structures.

							Thanx, Paul

/*
 * Do a full breadth-first scan of the rcu_node structures for the
 * specified rcu_state structure.
 */
#define rcu_for_each_node_breadth_first(rsp, rnp) \
	for ((rnp) = &(rsp)->node[0]; \
	     (rnp) < &(rsp)->node[rcu_num_nodes]; (rnp)++)