Re: sched: tweak select_idle_sibling to look for idle threads

[Yuyang Du]

On Wed, May 11, 2016 at 06:17:51AM +0200, Mike Galbraith wrote:
> > >  static inline unsigned long cfs_rq_runnable_load_avg(struct cfs_rq *cfs_rq)
> > >  {
> > > +> > 	> > if (sched_feat(LB_TIP_AVG_HIGH) && cfs_rq->load.weight > cfs_rq->runnable_load_avg*2)
> > > +> > 	> > 	> > return cfs_rq->runnable_load_avg + min_t(unsigned long, NICE_0_LOAD,
> > > +> > 	> > 	> > 	> > 	> > 	> > 	> > 	> >  cfs_rq->load.weight/2);
> > >  > > 	> > return cfs_rq->runnable_load_avg;
> > >  }
> >   
> > cfs_rq->runnable_load_avg is for sure no greater than (in this case much less
> > than, maybe 1/2 of) load.weight, whereas load_avg is not necessarily a rock
> > in gearbox that only impedes speed up, but also speed down.
> 
> Yeah, just like everything else, it'll cuts both ways (why you can't
> win the sched game).  If I can believe tbench, at tasks=cpus, reducing
> lag increased utilization and reduced latency a wee bit, as did the
> reserve thing once a booboo got fixed up.

Ok, so you have a secret IDLE_RESERVE? Good luck and show it, ;)

> Makes sense, robbing Peter
> to pay Paul should work out better for Paul.
> 
> NO_LB_TIP_AVG_HIGH
> Throughput 27132.9 MB/sec  96 clients  96 procs  max_latency=7.656 ms
> Throughput 28464.1 MB/sec  96 clients  96 procs  max_latency=9.905 ms
> Throughput 25369.8 MB/sec  96 clients  96 procs  max_latency=7.192 ms
> Throughput 25670.3 MB/sec  96 clients  96 procs  max_latency=5.874 ms
> Throughput 29309.3 MB/sec  96 clients  96 procs  max_latency=1.331 ms
> avg        27189   1.000                                     6.391   1.000
> 
> NO_LB_TIP_AVG_HIGH IDLE_RESERVE
> Throughput 24437.5 MB/sec  96 clients  96 procs  max_latency=1.837 ms
> Throughput 29464.7 MB/sec  96 clients  96 procs  max_latency=1.594 ms
> Throughput 28023.6 MB/sec  96 clients  96 procs  max_latency=1.494 ms
> Throughput 28299.0 MB/sec  96 clients  96 procs  max_latency=10.404 ms
> Throughput 29072.1 MB/sec  96 clients  96 procs  max_latency=5.575 ms
> avg        27859   1.024                                     4.180   0.654
> 
> LB_TIP_AVG_HIGH NO_IDLE_RESERVE
> Throughput 29068.1 MB/sec  96 clients  96 procs  max_latency=5.599 ms
> Throughput 26435.6 MB/sec  96 clients  96 procs  max_latency=3.703 ms
> Throughput 23930.0 MB/sec  96 clients  96 procs  max_latency=7.742 ms
> Throughput 29464.2 MB/sec  96 clients  96 procs  max_latency=1.549 ms
> Throughput 24250.9 MB/sec  96 clients  96 procs  max_latency=1.518 ms
> avg        26629   0.979                                     4.022   0.629
> 
> LB_TIP_AVG_HIGH IDLE_RESERVE
> Throughput 30340.1 MB/sec  96 clients  96 procs  max_latency=1.465 ms
> Throughput 29042.9 MB/sec  96 clients  96 procs  max_latency=4.515 ms
> Throughput 26718.7 MB/sec  96 clients  96 procs  max_latency=1.822 ms
> Throughput 28694.4 MB/sec  96 clients  96 procs  max_latency=1.503 ms
> Throughput 28918.2 MB/sec  96 clients  96 procs  max_latency=7.599 ms
> avg        28742   1.057                                     3.380   0.528
> 
> > But I really don't know the load references in select_task_rq() should be
> > what kind. So maybe the real issue is a mix of them, i.e., conflated balancing
> > and just wanting an idle cpu. ?
> 
> Depends on the goal.  For both, load lagging reality means the high
> frequency component is squelched, meaning less migration cost, but also
> higher latency due to stacking.  It's a tradeoff where Chris' latency
> is everything" benchmark, and _maybe_ the real world load it's based
> upon is on Peter's end of the rob Peter to pay Paul transaction.  The
> benchmark says it definitely is, the real world load may have already
> been fixed up by the select_idle_sibling() rewrite.
 
Obviously, load avgs are good at balancing in a larger scale in a timeframe,
so they should be used in comparing/balancing sd's not cpus. However, this
is not the case currently: avgs are mixed with idle cpu/core selection, so
I think better job can be done before and after select_idle_sibling().

For example, I don't know what the complex wake_affine() is really doing for
what. Am i missing something, you think?

Kudos to select_idle_sibling() rewrite, like Peter said, a second step and
an even third step scans are really helping, in addition to many cleanups
and refactors.