Re: [PATCH 2/2] sched: Actual changes after adding SCHED_SOFT_AFFINITY to make it work with the scheduler

[Rohit Jain]

On 09/21/2017 04:52 AM, Peter Zijlstra wrote:
> On Tue, Sep 19, 2017 at 03:37:12PM -0700, Rohit Jain wrote:
> > @@ -6019,7 +6040,8 @@ void __update_idle_core(struct rq *rq)
> >   static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int target)
> >   {
> >   	struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_idle_mask);
> > +	struct cpumask *pcpus = this_cpu_cpumask_var_ptr(select_idle_mask);
> This is broken... they're the exact _same_ variable.
>
> > +	int core;
> >   
> >   	if (!static_branch_likely(&sched_smt_present))
> >   		return -1;
> > @@ -6028,20 +6050,21 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int
> >   		return -1;
> >   
> >   	cpumask_and(cpus, sched_domain_span(sd), &p->cpus_allowed);
> > +	cpumask_and(pcpus, cpus, &p->cpus_preferred);
> > +	core = scan_cpu_mask_for_idle_cores(pcpus, target);
> >   
> > +	if (core >= 0)
> > +		return core;
> >   
> > +	if (cpumask_equal(cpus, pcpus))
> > +		goto out;
> Therefore this _must_ be true.

You are right, totally screwed it up here :(

> Also, you're touching one of the hottest paths in the whole scheduler

Yes, I am touching that code path but the number of CPUs scanned will
still remain the same (by intent at least). Because we will scan the
preferred first and then the allowed mask. I just added an 'ordering' in
searching. Please correct me if you meant something else entirely.

> You further failed to teach the actual load-balancer of this new mask,
> so it will still happily move tasks around.

We were thinking because on every wakeup in idle CPU search we 'force'
it to search the preferred CPUs first may be its OK to let it be stolen
in idle CPU load balancing because eventually it will come back to the
preferred CPUs? Again, may be my understanding is not right.