Re: [PATCH RFC] rcu/nocb: Provide default all-CPUs mask for RCU_NOCB_CPU=y

[Paul E. McKenney]

On Mon, Apr 11, 2022 at 11:17:02AM -0400, Joel Fernandes wrote:
> Hi Paul,
> 
> On Fri, Apr 8, 2022 at 4:54 PM Paul E. McKenney <paulmck@xxxxxxxxxx> wrote:
> [...]
> > > > > > > > > And is it really all -that- hard to specify an additional boot parameter
> > > > > > > > > across ChromeOS devices?  Android seems to manage it.  ;-)
> > > > > > > >
> > > > > > > > That's not the hard part I think. The hard part is to make sure a
> > > > > > > > future Linux user who is not an RCU expert does not forget to turn it
> > > > > > > > on. ChromeOS is not the only OS that I've seen someone forget to do it
> > > > > > > > ;-D. AFAIR, there were Android devices too in the past where I saw
> > > > > > > > this forgotten. I don't think we should rely on the users doing the
> > > > > > > > right thing (as much as possible).
> > > > > > > >
> > > > > > > > The single kernel binary point makes sense but in this case, I think
> > > > > > > > the bigger question that I'd have is what is the default behavior and
> > > > > > > > what do *most* users of RCU want. So we can keep sane defaults for the
> > > > > > > > majority and reduce human errors related to configuration.
> > > > > > >
> > > > > > > If both the ChromeOS and Android guys need it, I could reinstate the
> > > > > > > old RCU_NOCB_CPU_ALL Kconfig option.  This was removed due to complaints
> > > > > > > about RCU Kconfig complexity, but I believe that Reviewed-by from ChromeOS
> > > > > > > and Android movers and shakers would overcome lingering objections.
> > > > > > >
> > > > > > > Would that help?
> > > > > >
> > > > > > Yes, I think I would love for such a change. I am planning to add a
> > > > > > test to ChromeOS to check whether config options were correctly set
> > > > > > up. So I can test for both the RCU_NOCB_CPU options.
> > > > >
> > > > > Very good!
> > > > >
> > > > > Do you love such a change enough to create the patch and to collect
> > > > > convincing Reviewed-by tags?
> > > >
> > > > Yes sure, just so I understand - basically I have to make the code in
> > > > my patch run when RCU_NOCB_CPU_ALL option is passed (and keep the
> > > > option default disabled), but otherwise default to the current
> > > > behavior, right?
> > >
> > > Sorry rephrasing, "make the code in my patch run when the new
> > > CONFIG_RCU_NOCB_CPU_ALL is enabled".
> >
> > Here is what I believe you are proposing:
> >
> >
> >                                 ---     rcu_nocbs       rcu_nocbs=???
> >
> > CONFIG_RCU_NOCB_CPU_ALL=n       [1]     [2]             [3]
> >
> > CONFIG_RCU_NOCB_CPU_ALL=y       [4]     [4]             [3]
> 
> It is always a pleasure to read your well thought out emails ;-)
> 
> >
> > [1]     No CPUs are offloaded at boot.  CPUs cannot be offloaded at
> >         runtime.
> >
> > [2]     No CPUs are offloaded at boot, but any CPU can be offloaded
> >         (and later de-offloaded) at runtime.
> >
> > [3]     The set of CPUs that are offloaded at boot are specified by the
> >         mask, represented above with "???".  The CPUs that are offloaded
> >         at boot can be de-offloaded and offloaded at runtime.  The CPUs
> >         not offloaded at boot cannot be offloaded at runtime.
> 
> Hmm, in other words you are saying that in current code, if only
> select CPUs are offloaded at boot - then only those can be toggled,
> but the others are deemed not offload-able? I am happy to leave that
> quirk/behavior alone as I don't care much right now (for our use
> cases) for runtime toggling.

That is intentional behavior.  To see why, suppose that only CPU 0 was
offloaded at boot.  Now try offloading some other CPU.  A large quantity
of previously read-only data becomes read-write.  Synchronization is
not pretty.

On the other hand, default-enabling (de-)offloading on all CPUs creates
lots of unneeded rcuo kthreads.

So we didn't get here by accident.  ;-)

If this becomes a problem, I would be thinking in terms of an additional
kernel-boot parameter that made all CPUs offloadable by default.  But if
you have a better idea, please do not keep it a secret!

> > [4]     All CPUs are offloaded at boot, and any CPU can be de-offloaded
> >         and offloaded at runtime.  This is the same behavior that
> >         you would currently get with CONFIG_RCU_NOCB_CPU_ALL=n and
> >         rcu_nocbs=0-N.
> 
> Yes, this is the behavior I intend. So then there would not be a need
> to pass a mask (and I suspect for a large number of users, it
> simplifies boot params).

Very good, and from what I can see, this should work for everyone.

> > I believe that Steve Rostedt's review would carry weight for ChromeOS,
> > however, I am suffering a senior moment on the right person for Android.
> 
> I think for Android, Kalesh Singh is in the kernel team and Tim Murray
> is the performance lead. They could appropriately represent their RCU
> needs.

Sounds good!  Please collect a Reviewed-by from one or both of them.

							Thanx, Paul