Re: [RFC 1/2] sched: reduce migration cost between faster caches for idle_balance

[Mike Galbraith]

On Fri, 2018-02-09 at 11:08 -0500, Steven Sistare wrote:
> >> @@ -8804,7 +8803,8 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf)
> >>  		if (!(sd->flags & SD_LOAD_BALANCE))
> >>  			continue;
> >>  
> >> -		if (this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost) {
> >> +		if (this_rq->avg_idle < curr_cost + sd->max_newidle_lb_cost +
> >> +		    sd->sched_migration_cost) {
> >>  			update_next_balance(sd, &next_balance);
> >>  			break;
> >>  		}
> > 
> > Ditto.
> 
> The old code did not migrate if the expected costs exceeded the expected idle
> time.  The new code just adds the sd-specific penalty (essentially loss of cache 
> footprint) to the costs.  The for_each_domain loop visit smallest to largest
> sd's, hence visiting smallest to largest migration costs (though the tunables do 
> not enforce an ordering), and bails at the first sd where the total cost is a lose.

Hrm..

You're now adding a hypothetical cost to the measured cost of running
the LB machinery, which implies that the measurement is insufficient,
but you still don't say why it is insufficient.  What happens if you
don't do that?  I ask, because when I removed the...

   this_rq->avg_idle < sysctl_sched_migration_cost

...bits to check removal effect for Peter, the original reason for it
being added did not re-materialize, making me wonder why you need to
make this cutoff more aggressive.

	-Mike