Re: [PATCH RFC 0/2] sched: Introduce idle SMT priority for asymmetric capacity systems
From: Mete Durlu
Date: Tue Oct 13 2026 - 07:09:42 EST
On 12/10/2026 16:49, Vincent Guittot wrote:
On Mon, 12 Oct 2026 at 13:04, Mete Durlu <meted@xxxxxxxxxxxxx> wrote:
On 09/10/2026 17:32, Vincent Guittot wrote:
On Thu, 8 Oct 2026 at 17:25, Mete Durlu <meted@xxxxxxxxxxxxx> wrote:
On 08/10/2026 16:58, Shrikanth Hegde wrote:
Hi Meter/Andrea.
On 10/8/26 5:40 PM, Andrea Righi wrote:
Hi Mete,
On Thu, Oct 08, 2026 at 10:30:46AM +0200, Mete Durlu wrote:
Summary
===========================================================================
On systems with asymmetric CPU capacities, the scheduler prefers
fully idle
cores over idle SMT siblings of busy cores. This works generally well
but
virtualized platforms where low capacity cores should be avoided are not
considered. Introduce a new config option and arch hook to prioritize
SMT utilization.
That's true only under physical CPU contention right? or is it always?
Right, because of that s390 treats all CPUs as equal and starts to
assign lower capacities to CPUs with low entitlement once a certain
steal time threshold is crossed (aka physical CPU contention).
Hi,
Sorry for causing a misunderstanding. I should have been more patient
and detailed while writing my reply.
I'm not sure to get what your topology is because "if s390 treats all
CPUs as equal" then you should never use select_idle_capacity()
because their capacity should be fixed and equal and
SD_ASYM_CPUCAPACITY_FULL never set.
On s390 the entitlement of the vCPUs are signified by an arch specific
value we call polarization. We can tell which vCPUs are assigned high
and low entitlement. This entitlement value tells us which vCPUs are
prone to experience steal time first. Without any contention all
vCPUs behave and perform *almost* the same(of course, we want
some priority for CPUs with high entitlement).
Prior to experiencing steal time(it's what we use to determine
contention) to get the most out of the system we assign all vCPUs
the same capacities, making the system symmetric like you say.
While there is no contention s390's vCPUs also perform better if
fully idle cores are selected for task placement.
If your hypervisor lowers the capacity once a certain threshold iss390's linux implementation has a mechanism called hiperdispatch,
crossed, we already account for the steal time in the scheduler with
irq and remove it from the capacity available for tasks but the
original capacity doesn't change.
that checks topology changes along with system steal time (and soon
the total free capacity of the pyhsical machine) periodically and
makes vCPU capacity adjustments accordingly.
After steal time starts ramping up and cross a certain threshold,
we start assigning lower capacities to vCPUs with low entitlement to
hint the scheduler that they should be avoided.
If all your vCPUs starts with the same capacity, you should not use
asym_capacity or enabled it . What you describe above is system
pressure (steal time or other) on your vCPU that change over time but
not a capacity change. The value returned by arch_scale_cpu_capacity
is not supposed to change at runtime or you need to rebuild the
sched_domain topology. As an example, max_allowed_capacity does assume
capacity fixed otherwise the misfit detection will sometimes fails.
Exactly for that reason on s390 we do topology updates after
capacity updates. Topology changes caused by the underlying hypervisor
or the cpu capacity changes can get reflected without a problem via
topology rebuilds. Without topology rebuild we wouldn't even get
the correct SD_ flags to trigger asym_capacity behavior like you say.
What you describe really fit into the prefered core that Shrikanth
Hegde has developped
I agree, but Shrikanth's solution can not bring some of the
behavior changes required. cpu_preferred_mask can help
direct task placement away from low capacity cores but idle
core first approach still places tasks to fully idle low
capacity cores. This patches proposal tries to move tasks
first to the idle SMT siblings.
Consider the following;
On a system with 2 high capacity and 4 low capacity cpus.
steal governor of vCPU backoff mechanism calculates 2 low capacity
cpus should be taken off from the cpu_preferred_mask to decrease
experienced steal time. cpu_preferred_mask has 2 high capacity
and 2 low capacity cpus.
Current behavior tires to place tasks to fully idle cores first
even if they are low capacity cores.
c0(high) c1(high) c2(low) c3(low) ...
-------------------------------------------
SMT 0 task task task task
SMT 1
With the proposal, the fully idle core preference is circumvented
and the most fitting SMT thread is selected.
c0(high) c1(high) c2(low) c3(low) ...
-------------------------------------------
SMT 0 task task
SMT 1 task task
This is beneficial because at the time of contention(system pressure)
low capacity cores could perform worse than SMT siblings of busy
high capacity cores, even if there is not enough steal time for
steal governor of Shrikanth's vCPU backoff to mark said low
capacity cores as non-preferred.
Andrea came up with an idea to extend Shrikanth's mechanism further
by adding a soft affinity to a selected set of CPUs. Task placement
would first fully load these CPUs before spilling load onto rest
of the CPUs on cpu_preferred_mask. That would help achieve the
behavior I describe above if the soft affinity covers high
capacity CPUs.
The discussion is still on going though. If Shrikanth thinks this is
a worthwhile concept then I'd be glad to work on the implementation
and provide the patches.