Re: RCU vs NOHZ

[Paul E. McKenney]

On Thu, Sep 15, 2022 at 10:39:12AM +0200, Peter Zijlstra wrote:
> Hi,
> 
> After watching Joel's talk about RCU and idle ticks I was wondering
> about why RCU doesn't have NOHZ hooks -- that is regular NOHZ, not the
> NOHZ_FULL stuff.

It actually does, but they have recently moved into the context-tracking
code, courtesy of Frederic's recent patch series.

> These deep idle states are only feasible during NOHZ idle, and the NOHZ
> path is already relatively expensive (which is offset by then mostly
> staying idle for a long while).
> 
> Specifically my thinking was that when a CPU goes NOHZ it can splice
> it's callback list onto a global list (cmpxchg), and then the
> jiffy-updater CPU can look at and consume this global list (xchg).
> 
> Before you say... but globals suck (they do), NOHZ already has a fair
> amount of global state, and as said before, it's offset by the CPU then
> staying idle for a fair while. If there is heavy contention on the NOHZ
> data, the idle governor is doing a bad job by selecting deep idle states
> whilst we're not actually idle for long.
> 
> The above would remove the reason for RCU to inhibit NOHZ.
> 
> 
> Additionally; when the very last CPU goes idle (I think we know this
> somewhere, but I can't reaily remember where) we can insta-advance the
> QS machinery and run the callbacks before going (NOHZ) idle.
> 
> 
> Is there a reason this couldn't work? To me this seems like a much
> simpler solution than the whole rcu-cb thing.

To restate Joel's reply a bit...

Maybe.

Except that we need rcu_nocbs anyway for low latency and HPC applications.
Given that we have it, and given that it totally eliminates RCU-induced
idle ticks, how would it help to add cmpxchg-based global offloading?

							Thanx, Paul