Re: [PATCH RFC 0/2] sched: Introduce idle SMT priority for asymmetric capacity systems

[Mete Durlu]

[..snip..]

> > > What you describe really fit into the prefered core that Shrikanth
> > > Hegde has developped
> >
> > I agree, but Shrikanth's solution can not bring some of the
> > behavior changes required. cpu_preferred_mask can help
> > direct task placement away from low capacity cores but idle
> > core first approach still places tasks to fully idle low
> > capacity cores. This patches proposal tries to move tasks
> > first to the idle SMT siblings.
> >
> > Consider the following;
> > On a system with 2 high capacity and 4 low capacity cpus.
> > steal governor of vCPU backoff mechanism calculates 2 low capacity
> > cpus should be taken off from the cpu_preferred_mask to decrease
> > experienced steal time. cpu_preferred_mask has 2 high capacity
> > and 2 low capacity cpus.
>
>
> In this scenario, do you still have reasonable steal time or no?

steal time should be sitting between low and high steal time
threshold values of the steal governor. (no adjustments)

> > Current behavior tires to place tasks to fully idle cores first
> > even if they are low capacity cores.
> >
> >        c0(high) c1(high)  c2(low)  c3(low)  ...
> > -------------------------------------------
> > SMT 0 task     task      task     task
> > SMT 1
> >
> >
> > With the proposal, the fully idle core preference is circumvented
> > and the most fitting SMT thread is selected.
> >
> >        c0(high) c1(high)  c2(low)  c3(low)  ...
> > -------------------------------------------
> > SMT 0 task     task
> > SMT 1 task     task
> >
> > This is beneficial because at the time of contention(system pressure)
> > low capacity cores could perform worse than SMT siblings of busy
> > high capacity cores, even if there is not enough steal time for
> > steal governor of Shrikanth's vCPU backoff to mark said low
> > capacity cores as non-preferred.
>
> I read this as tunable problem of low,high thresholds to adjust
> preferred_mask same as high capacity cores. No?

No, because cpu_preferred_mask would contain some low capacity
cpus as the experienced steal time is not high enough to exclude
all low capacity cpus from the mask.

And any new task could be placed on them like below as they should
not cause an increase in steal time. See below (for 6 tasks);
(non preferred cpus marked with X)

       c0(high) c1(high)  c2(low)  c3(low)  c4(low)  c5(low)
------------------------------------------------------------
SMT 0 task     task       task     task          X        X
SMT 1 task     task                              X        X

The soft affinity should come in as full utilization of high capacity
cores(and their SMT siblings) before placing tasks to low capacity
cores (for 4 tasks);

       c0(high) c1(high)  c2(low)  c3(low)  c4(low)  c5(low)
------------------------------------------------------------
SMT 0 task     task                              X        X
SMT 1 task     task                              X        X

> > Andrea came up with an idea to extend Shrikanth's mechanism further
> > by adding a soft affinity to a selected set of CPUs. Task placement
> > would first fully load these CPUs before spilling load onto rest
> > of the CPUs on cpu_preferred_mask. That would help achieve the
> > behavior I describe above if the soft affinity covers high
> > capacity CPUs.
> >
> > The discussion is still on going though. If Shrikanth thinks this is
> > a worthwhile concept then I'd be glad to work on the implementation
> > and provide the patches.
>
> What i don't understand is.
> - if there is no system pressure to use those low capacity cores, why
>    not use them if the task util fits its capacity. If the task utilization
>    doesn't, it won't land on them anyways today right?

Current task placement looks like this (5 tasks);

       c0(high) c1(high)  c2(low)  c3(low)  c4(low)  c5(low)
------------------------------------------------------------
SMT 0  task     task      task     task*         X        X
SMT 1  task     (**)                             X        X

Even with lowered capacities task don't migrate away from
fully idle cores, cpu selection prioritizes fully idle
cores before everything else if i am not mistaken.

The proposal with this patch series tries to circumvent exactly that
idle core prioritization and uses SMT threads first, like below;

       c0(high) c1(high)  c2(low)  c3(low)  c4(low)  c5(low)
------------------------------------------------------------
SMT 0  task     task      task     (**)          X        X
SMT 1  task     task*                            X        X

*the task that got placed to SMT1 of core1 instead of fully idle
 core3
**where the next task would likely land.

After high capacity CPUs are fully saturated, remaining low capacity
CPUs in cpu_preferred_mask can be utilized with idle core priority
once again as there are no distinction between them.
Utilizing high capacity cores(or entitled cores...) should lower
the footprint of the guest and allow less noise from guests.
Steal time(or contention) is expected first on low capacity cores
anyway.

Does this make sense?

Thank you!
-Mete