Re: [RFC PATCH 7/9] housekeeping: Use own boot option, independant from nohz

[Frederic Weisbecker]

On Sun, Aug 13, 2017 at 11:13:40AM -0400, Luiz Capitulino wrote:
> On Sat, 12 Aug 2017 16:10:06 +0200
> Frederic Weisbecker <fweisbec@xxxxxxxxx> wrote:
> > > Am I right that from now on nohz_full= users will also have
> > > to specify housekeeping= in order to get nohz_full working?
> > > If that's correct, then won't this patch break nohz_full for
> > > existing setups?  
> > 
> > nohz_full= will still work but will only imply tick stop. A few isolation
> > details that were enabled by nohz_full= won't be handled anymore such as:
> > unbound timers affinity, watchdog disablement, rcu threads affinity, sched idle
> > load balancing... Those are now handled by housekeeping=
> > 
> > So yes in a sense, this can break some setup that assume nohz_full= does more
> > than stopping the tick.
> 
> Yes, the problem is that this is how it has always worked. Also,
> the breakage will be very subtle and hard to debug.

[...]

> 
> > Perhaps I should remove the nohz_full= parameter altogether and let nohz_full controlled
> > by housekeeping= only. How much can kernel parameters be considered as kernel ABIs?
> 
> That's a very good question, I don't have an answer for that.

That said, "nohz_full=" never implied too much isolation features so far, and those have
often changed over time, as in RCU. I think unbound timer affinity is the most important
one.

Perhaps we can keep "nohz_full=1-15" as an alias for a future "cpu_isolation=nohz,1-15"
and at least imply unbound timer affinity with it.

> 
> > Also I'm wondering if "housekeeping=" is a clear name for users. "isolation=" or
> > "cpu_isolation=" would be better and more obvious. Housekeeping based naming would only be
> > internal implementation detail. And deactivating the tick through "cpu_isolation=" would
> > be clearer than if we did through "housekeeping=".
> 
> That's exactly my thinking while I was reviewing the series!
> 
> > Of course the problem is that we already have "isolcpus=". But re-implementing isolcpus
> > on top of housekeeping might be a good idea. I believe that the current implementation on
> > top of NULL domains isn't much beloved. A less controversial implementation might even
> > allow us to control it though cpusets.
> 
> You're completely right. Some people don't use isolcpus= because it
> disables load balancing and that may be a problem for setups where
> tasks are pinned to a set of CPUs where the number of tasks is greater
> than the number of CPUs. However, for the cases where you have a
> single task pinned to a CPU, having load balancing taking place adds
> an extra latency (I won't remember how much, but I guess it was more
> than 10us).

What is the source of the load balancing inducing such latency when a single
task is affine to a CPU? If this is idle load balancing, it is now affine to
housekeepers. If this is task wakeup then it's suprising because select_task_rq()
is optimized toward single CPU affinity.

Is there another source I'm overlooking?

> If there's a way to "disable" load balancing from user-space, say
> with cpusets, then I think we should keep the isolated CPUs attached
> to a domain as you suggest.

I'm not sure such a solution would be accepted. The most sensible way
to disable load balancing is still to tune the affinity of tasks. If there
is an off-case overhead with load balancing (ie: when no more than one
task is affine to that CPU) then we should solve that with a fast path.

> Another detail about isolcpus= is that it doesn't isolate the CPU
> from kernel threads. That is, unpinned kernel threads are allowed
> to run on CPUs not isolated with isolcpus=. We might consider changing
> that for a new isolation option.

You mean unpinned kernel threads are allowed to run on isolcpus, right?
That definetly can be solved.

> 
> I know that there are many arguments against isolcpus= and some people
> advice using cpusets. The problem with that advice is that isolcpus=
> goes a bit beyond isolating a CPU from user-space tasks. One additional
> thing is does for example, is pinning the kernel_init() thread to
> housekeeping CPUs. This is key, because that thread will create timers
> at early boot that will pin themselves to the CPU they run.

Right, but also unbound timers are affine to housekeepers, we needed that for
nohz_full.

> Finally, I'm wondering how all this will fit together with TASK_ISOLATION.
> One of the questions I ask myself is: can/should the things TASK_ISOLATION
> does be done by a kernel command-line parameter instead? Or should we
> try to come up with a list of global things to control (eg. the tick,
> kernel thread affinity, etc) and per-task controls?

So I've been thinking a lot about that lately. I told Chris that TASK_ISOLATION
shouldn't be a CPU feature but a task feature. Then I realized that it doesn't work
either, my bad :-)  In the end I think that the most part of it must be a CPU
property: nohz, task isolation, timers and workqueue affinity, etc... Then what's
left for the per task thing is to tell it when it is unexpectingly interrupted by noise.

Therefore I think most of the isolation features should be controlled by
command line and cpusets (through a new cpuset subsystem maybe) then TASK_ISOLATION
through prtcl() for the noise monitoring.

Thanks.