Re: [RFT][DONOTMERGE][PATCH 0/6] timers/migration: Prioritize lower capacity CPUs as migrators

[Christian Loehle]

On 7/3/26 13:53, Frederic Weisbecker wrote:
> Le Fri, Jul 03, 2026 at 11:13:13AM +0100, Christian Loehle a écrit :
>> On 6/25/26 17:41, Frederic Weisbecker wrote:
>>> Hi,
>>>
>>> It has been reported that the timer migration tree wasn't too helpful
>>> on asymetric systems, with timers migrating too often to big capacity
>>> CPUs, waking them more frequently from deep idle state.
>>>
>>> Splitting the tree into per CPU capacity hierarchies doesn't seem to
>>> improve the situation.
>>>
>>> So here is another proposal: gather CPUs belonging to the same capacity
>>> under the same groups and finally join those capacity groups under a
>>> common root. Exactly in the same way it is done for NUMA systems.
>>
>> This seems to work for my orion o6, the resulting hierarchy is:
>>
>>                            root lvl1
>>                       family=-1, want_low_migrator=1
>>           ________________|___________|___________|____________
>>          /                /           /           /            /
>>     mask 0x1         mask 0x2    mask 0x4    mask 0x8    mask 0x10
>>     grp279           grp866      grp905      grp984      grp1024
>>     CPUs 2,3,4,5     CPUs 8,9    CPUs 6,7    CPUs 10,11  CPUs 0,1
>>
>>>
>>> And then always pick up the lowest capacity non-idle CPU group as the
>>> migrator with the hope that handling timers will keep lower capacity
>>> CPUs busy enough to increase their chances to become global idle
>>> migrators.
>>>
>>> Please give it a try but make sure you have fewer than 9 capacities
>>> on your system. Handling more will require a bit more throughts. Let's
>>> see first if this is the right way to go.
>>
>> Honestly I think buckets should be fine here? One for each 128-capacity
>> step (or even 256), which would also make these awkward Grace capacities all
>> fall into one, which makes a lot of sense IMO?
> 
> It may be hard to find the right threshold. Also migrating from capX to capX-50
> can still make a visible difference. So I'd rather keep all capacities separate
> (for now) and create more levels above in the hierarchy to group capacities if
> necessary. It should only create one more level if nr_cap > 8 and two levels if
> nr_cap > 64, but reaching that latter doesn't sound reasonable on the hardware,
> right? (famous last words).

So I'm only aware of systems with <=3 different uArch CPUs, there's a few with
many more capacity classes, but these will be the same uArch with slightly different
available frequencies, like in my example there's:
Quad Big Cortex®-A720 Up to 2.6GHz
Quad Medium Cortex®-A720 Up to 2.4GHz
Quad Little Cortex®-A520 1.8GHz
i.e. A720 and A520. I think as long as the capacity classes don't mix uArchs we
should be fine...
We're far off from nr_cap > 64 FWIW but technically this can change even for existing
silicon e.g. if FW notices the different cores age differently it could start reporting
different frequencies from one boot to the next. That being said I agree nr_cap < 64 is
okay for now...
I would've grouped them just because we do fundamentally want bigger capacity classes,
right?

> 
>> Thanks for giving this a shot, here are the results, I tested this against
>> tmigr-off (which I've added ontop of series and capacity-unaware dd50a6e260ea "timers/migration: Revert per CPU capacity hierarchy"))
>> In short: tmigr-off-600, capacity-unaware-600, series-600
>>
>>
>> idle-600s
>> ----------
>>
>> Overall idle states:
>>
>> +------------------------+---------+---------+---------+--------------+
>> | Variant                | LPI-0   | LPI-1   | LPI-2   | LPI-1 + LPI-2|
>> +------------------------+---------+---------+---------+--------------+
>> | tmigr-off-600          | 3771.0s | 650.9s  | 2212.0s | 2862.9s      |
>> | capacity-unaware-600   | 2461.7s | 1245.3s | 2708.9s | 3954.3s      |
>> | series-600             | 2429.2s | 1260.7s | 2726.5s | 3987.2s      |
>> +------------------------+---------+---------+---------+--------------+
>>
>> Delta vs tmigr-off-600:
>>
>> +------------------------+----------+---------+---------+--------------+
>> | Variant                | LPI-0    | LPI-1   | LPI-2   | LPI-1 + LPI-2|
>> +------------------------+----------+---------+---------+--------------+
>> | capacity-unaware-600   | -1309.2s | 594.4s  | 497.0s  | 1091.4s      |
>> | series-600             | -1341.7s | 609.8s  | 514.5s  | 1124.3s      |
>> +------------------------+----------+---------+---------+--------------+
> 
> Ok here it looks good.

FWIW this should be the most important case.

> 
>>
>> LPI-2 by capacity group:
>>
>> +-------+----------------+------------------------+------------+
>> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
>> +-------+----------------+------------------------+------------+
>> | 279   | 805.3s         | 1007.3s                | 1003.5s    |
>> | 866   | 391.7s         | 483.5s                 | 477.3s     |
>> | 905   | 411.6s         | 466.1s                 | 501.8s     |
>> | 984   | 361.3s         | 468.3s                 | 458.9s     |
>> | 1024  | 242.2s         | 283.8s                 | 285.0s     |
>> +-------+----------------+------------------------+------------+
>>
>>
>> schbench-600s
>> -------------
>>
>> Overall idle states:
>>
>> +------------------------+--------+--------+--------+--------------+
>> | Variant                | LPI-0  | LPI-1  | LPI-2  | LPI-1 + LPI-2|
>> +------------------------+--------+--------+--------+--------------+
>> | tmigr-off-600          | 167.7s | 34.1s  | 22.3s  | 56.4s        |
>> | capacity-unaware-600   | 158.8s | 37.2s  | 28.1s  | 65.3s        |
>> | series-600             | 161.5s | 37.3s  | 26.3s  | 63.6s        |
>> +------------------------+--------+--------+--------+--------------+
> 
> But IIUC here it performs worse than before on LPI-2 ?
> 
>>
>> Delta vs tmigr-off-600:
>>
>> +------------------------+--------+-------+-------+--------------+
>> | Variant                | LPI-0  | LPI-1 | LPI-2 | LPI-1 + LPI-2|
>> +------------------------+--------+-------+-------+--------------+
>> | capacity-unaware-600   | -8.9s  | 3.1s  | 5.8s  | 8.9s         |
>> | series-600             | -6.2s  | 3.2s  | 4.0s  | 7.3s         |
>> +------------------------+--------+-------+-------+--------------+
> 
> And overall worse in fact?
> 
>>
>> LPI-2 by capacity group:
>>
>> +-------+----------------+------------------------+------------+
>> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
>> +-------+----------------+------------------------+------------+
>> | 279   | 8.9s           | 10.6s                  | 10.3s      |
>> | 866   | 3.3s           | 4.8s                   | 4.3s       |
>> | 905   | 3.8s           | 4.6s                   | 4.3s       |
>> | 984   | 3.3s           | 4.8s                   | 4.7s       |
>> | 1024  | 3.0s           | 3.2s                   | 2.7s       |
>> +-------+----------------+------------------------+------------+
>>
>>
>> tbench-4clients-600s
>> --------------------
>>
>> Overall idle states:
>>
>> +------------------------+---------+---------+--------+--------------+
>> | Variant                | LPI-0   | LPI-1   | LPI-2  | LPI-1 + LPI-2|
>> +------------------------+---------+---------+--------+--------------+
>> | tmigr-off-600          | 3125.7s | 1395.7s | 425.9s | 1821.6s      |
>> | capacity-unaware-600   | 3009.4s | 1437.7s | 456.3s | 1894.0s      |
>> | series-600             | 2998.3s | 1436.3s | 453.6s | 1889.9s      |
>> +------------------------+---------+---------+--------+--------------+
> 
> Same here?
> 
>>
>> Delta vs tmigr-off-600:
>>
>> +------------------------+---------+--------+--------+--------------+
>> | Variant                | LPI-0   | LPI-1  | LPI-2  | LPI-1 + LPI-2|
>> +------------------------+---------+--------+--------+--------------+
>> | capacity-unaware-600   | -116.2s | 41.9s  | 30.5s  | 72.4s        |
>> | series-600             | -127.3s | 40.6s  | 27.7s  | 68.3s        |
>> +------------------------+---------+--------+--------+--------------+
>>
>> LPI-2 by capacity group:
>>
>> +-------+----------------+------------------------+------------+
>> | Group | tmigr-off-600  | capacity-unaware-600   | series-600 |
>> +-------+----------------+------------------------+------------+
>> | 279   | 162.9s         | 179.3s                 | 180.5s     |
>> | 866   | 68.6s          | 73.4s                  | 71.9s      |
>> | 905   | 67.7s          | 69.5s                  | 71.8s      |
>> | 984   | 72.9s          | 75.5s                  | 71.5s      |
>> | 1024  | 53.7s          | 58.6s                  | 57.9s      |
>> +-------+----------------+------------------------+------------+
>>
>> Seems the series does what it claims
> 
> Does it really?

In the sense that it seems to restore the benefit of tmigr in the first place,
the rest would be interesting to hear from Sehee. I tried a little but found
nothing to produce something that would explain such a big power regression.
(Because the 'bad' 'little->big' migrations would have to absolutely dominate
over the 'good' little->big (or good little->little and big->big) migrations).

Your interpretation on schbench and tbench capacity-unaware-600 vs series-600
is correct though, I'll go look some more into why that is...
It might just be the fragmentation of the bigs... (i.e. a migr grp984->grp1024
now disallowed but most likely it would still be 'good' and the fact we have
many more big cores than little cores).