Re: Energy-efficiency options within RCU

[Paul E. McKenney]

On Mon, Dec 14, 2020 at 01:12:48PM -0500, Joel Fernandes wrote:
> On Thu, Dec 10, 2020 at 10:37:37AM -0800, Paul E. McKenney wrote:
> > Hello, Joel,
> > 
> > In case you are -seriously- interested...  ;-)
> 
> I am always seriously interested :-). The issue becomes when life throws me a
> curveball. This was the year of curveballs :-)
> 
> Thank you for your reply and I have added it to my list to investigate how we
> are configuring nocb on our systems. I don't think anyone over here has given
> these RCU issues a serious look over here.

In your defense, I would guess that many of them don't have that much
effect.  But true, you never know until you try.

							Thanx, Paul

> thanks,
> 
>  - Joel
> 
> 
> 
> > 						Thanx, Paul
> > 
> > rcu_nocbs=
> > 
> > 	Adding a CPU to this list offloads RCU callback invocation from
> > 	that CPU's softirq handler to a kthread.  In big.LITTLE systems,
> > 	this kthread can be placed on a LITTLE CPU, which has been
> > 	demonstrated to save significant energy in benchmarks.
> > 	http://www.rdrop.com/users/paulmck/realtime/paper/AMPenergy.2013.04.19a.pdf
> > 
> > nohz_full=
> > 
> > 	Any CPU specified by this boot parameter is handled as if it was
> > 	specified by rcu_nocbs=.
> > 
> > rcutree.jiffies_till_first_fqs=
> > 
> > 	Increasing this will decrease wakeup frequency to the grace-period
> > 	kthread for the first FQS scan.  And increase grace-period
> > 	latency.
> > 
> > rcutree.jiffies_till_next_fqs=
> > 
> > 	Ditto, but for the second and subsequent FQS scans.
> > 
> > 	My guess is that neither of these makes much difference.  But if
> > 	they do, maybe some sort of backoff scheme for FQS scans?
> > 
> > rcutree.jiffies_till_sched_qs=
> > 
> > 	Increasing this will delay RCU's getting excited about CPUs and
> > 	tasks not responding with quiescent states.  This excitement
> > 	can cause extra overhead.
> > 
> > 	No idea whether adjusting this would help.  But if you increase
> > 	rcutree.jiffies_till_first_fqs or rcutree.jiffies_till_next_fqs,
> > 	you might need to increase this one accordingly.
> > 
> > rcutree.qovld=
> > 
> > 	Increasing this will increase the grace-period duration at which
> > 	RCU starts sending IPIs, thus perhaps reducing the total number
> > 	of IPIs that RCU sends.  The destination CPUs are unlikely to be
> > 	idle, so it is not clear to me that this would help much.  But
> > 	perhaps I am wrong about them being mostly non-idle, who knows?
> > 
> > rcupdate.rcu_cpu_stall_timeout=
> > 
> > 	If you get overly zealous about the earlier kernel boot parameters,
> > 	you might need to increase this one as well.  Or instead use the
> > 	rcupdate.rcu_cpu_stall_suppress= kernel boot parameter to suppress
> > 	RCU CPU stall warnings entirely.
> > 
> > rcutree.rcu_nocb_gp_stride=
> > 
> > 	Increasing this might reduce grace-period work somewhat.  I don't
> > 	see why a (say) 16-CPU system really needs to have more than one
> > 	rcuog kthread, so if this does help it might be worthwhile setting
> > 	a lower limit to this kernel parameter.
> > 
> > rcutree.rcu_idle_gp_delay=  (Only CONFIG_RCU_FAST_NO_HZ=y kernels.)
> > 
> > 	This defaults to four jiffies on the theory that grace periods
> > 	tend to last about that long.  If grace periods tend to take
> > 	longer, then it makes a lot of sense to increase this.	And maybe
> > 	battery-powered devices would rather have it be about 2x or 3x
> > 	the expected grace-period duration, who knows?
> > 
> > 	I would keep it to a power of two, but the code should work with
> > 	other numbers.  Except that I don't know that this has ever been
> > 	tested.  ;-)
> > 
> > srcutree.exp_holdoff=
> > 
> > 	Increasing this decreases the number of SRCU grace periods that
> > 	are treated as expedited.  But you have to have closely-spaced
> > 	SRCU grace periods for this to matter.	(These do happen at least
> > 	sometimes because I added this only because someone complained
> > 	about the performance regression from the earlier non-tree SRCU.)
> > 
> > rcupdate.rcu_task_ipi_delay=
> > 
> > 	This kernel parameter delays sending IPIs for RCU Tasks Trace,
> > 	which is used by sleepable BPF programs.  Increasing it can
> > 	reduce overhead, but can also increase the latency of removing
> > 	sleepable BPF programs.
> > 
> > rcupdate.rcu_task_stall_timeout=
> > 
> > 	If you slow down RCU Tasks Trace too much, you may need this.
> > 	But then again, the default 10-minute value should suffice.
> > 
> > CONFIG_RCU_FAST_NO_HZ=y
> > 
> > 	This only has effect on CPUs not specified by rcu_nocbs, and thus
> > 	might be useful on systems that offload RCU callbacks only on
> > 	some of the CPUs.  For example, a big.LITTLE system might offload
> > 	only the big CPUs.  This Kconfig option reduces the frequency of
> > 	timer interrupts (and thus of RCU-related softirq processing)
> > 	on idle CPUs.  This has been shown to save significant energy
> > 	in benchmarks:
> > 	http://www.rdrop.com/users/paulmck/realtime/paper/AMPenergy.2013.04.19a.pdf
> > 
> > CONFIG_RCU_STRICT_GRACE_PERIOD=y
> > 
> > 	This works hard (as in burns CPU) to sharply reduce grace-period
> > 	latency.  The effect is probably to greatly increase power
> > 	consumption, but there might well be workloads where the shorter
> > 	grace periods more than make up for the extra CPU time.  Or not.
> > 
> > CONFIG_HZ=
> > 
> > 	Reducing the scheduler-clock interrupt frequency has the opposite
> > 	effect, namely of increasing RCU grace-period latency, but while
> > 	also reducing RCU's CPU utilization.
> > 
> > CONFIG_TASKS_TRACE_RCU_READ_MB=y
> > 
> > 	Reduce the need to IPI RCU Tasks Trace holdout tasks, but at the
> > 	expense of an increase in to/from idle overhead.  This Kconfig
> > 	option also slows down the rate at which RCU Tasks Trace polls
> > 	for holdout tasks.  This polling rate cannot be separately
> > 	specified, but if changing the initial source-code values of
> > 	either rcu_tasks_trace.gp_sleep or rcu_tasks_trace.init_fract
> > 	proves useful, kernel boot parameters could be created.
> > 
> > 	That said, automatic initialization heuristics are more
> > 	convenient.  When they work, anyway.