Re: [RFC PATCH 2/3] sched: change scheduler to give preference to soft affinity CPUs

From: Subhra Mazumdar
Date: Thu Jul 18 2019 - 22:57:46 EST



On 7/18/19 5:07 PM, Peter Zijlstra wrote:
On Wed, Jul 17, 2019 at 08:31:25AM +0530, Subhra Mazumdar wrote:
On 7/2/19 10:58 PM, Peter Zijlstra wrote:
On Wed, Jun 26, 2019 at 03:47:17PM -0700, subhra mazumdar wrote:
The soft affinity CPUs present in the cpumask cpus_preferred is used by the
scheduler in two levels of search. First is in determining wake affine
which choses the LLC domain and secondly while searching for idle CPUs in
LLC domain. In the first level it uses cpus_preferred to prune out the
search space. In the second level it first searches the cpus_preferred and
then cpus_allowed. Using affinity_unequal flag it breaks early to avoid
any overhead in the scheduler fast path when soft affinity is not used.
This only changes the wake up path of the scheduler, the idle balancing
is unchanged; together they achieve the "softness" of scheduling.
I really dislike this implementation.

I thought the idea was to remain work conserving (in so far as that
we're that anyway), so changing select_idle_sibling() doesn't make sense
to me. If there is idle, we use it.

Same for newidle; which you already retained.
The scheduler is already not work conserving in many ways. Soft affinity is
only for those who want to use it and has no side effects when not used.
Also the way scheduler is implemented in the first level of search it may
not be possible to do it in a work conserving way, I am open to ideas.
I really don't understand the premise of this soft affinity stuff then.

I understood it was to allow spreading if under-utilized, but group when
over-utilized, but you're arguing for the exact opposite, which doesn't
make sense.
You are right on the premise. The whole knob thing came into existence
because I couldn't make the first level of search work conserving. I am
concerned that trying to make that work conserving can introduce
significant latency in the code path when SA is used. I have made the
second level of search work conserving when we search the LLC domain.

Having said that, SA need not necessarily be binary i.e only spill over to
the allowed set if the preferred set is 100% utilized (work conserving).
The spill over can happen before that and SA can have a degree of softness.

The above two points made me go down the knob path for the first level of
search.