Trade-off between load_balance frequency and CPU utilization under high load

[liukai (Y)]

In our performance experiments, by gradually increasing the CPU load, we
observed that under high load, the CPU utilization (node CPU) in kernel 6.6
is higher than in 4.19, reaching up to 4% higher. By capturing flame graph
data, we found that the total execution time of load_balance in kernel 6.6
is 18% longer than in 4.19.

Benchmark: specjbb // 6.6
index   target QPS   actual QPS           RT      pod CPU     node CPU
    1        60000        60004         0.73         5.06        14.97
    2       120000       120074         0.79        10.22        29.67
    3       180000       180866         0.87        16.00        45.91
    4       240000       240091         0.92        21.69        62.94

Benchmark: specjbb // 4.19
index   target QPS   actual QPS           RT      pod CPU     node CPU
    1        60000        60004         0.72         4.86        14.81
    2       120000       120074         0.79         9.69        29.52
    3       180000       180870         0.83        14.57        42.72
    4       240000       240074         0.90        19.55        58.59

we found that in kernel 6.6, the execution of load_balance is less costly.
Even under high load, the condition this_rq->avg_idle <
sd->max_newidle_lb_cost is still easily satisfied. As a result, compared to
kernel 4.19, load_balance is executed more frequently in 6.6, leading to
higher CPU utilization.

if (!READ_ONCE(this_rq->rd->overload) ||
    (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) {

    if (sd)
        update_next_balance(sd, &next_balance);
    rcu_read_unlock();

    goto out;
}

We identified that the changes introduced by this patch
(https://lore.kernel.org/all/20211021095219.GG3891@xxxxxxx/).

--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10895,8 +10895,7 @@ static int newidle_balance(struct rq *this_rq, struct rq_flags *rf)
     rcu_read_lock();
     sd = rcu_dereference_check_sched_domain(this_rq->sd);

-     if (this_rq->avg_idle < sysctl_sched_migration_cost ||
-         !READ_ONCE(this_rq->rd->overload) ||
+     if (!READ_ONCE(this_rq->rd->overload) ||
         (sd && this_rq->avg_idle < sd->max_newidle_lb_cost)) {

             if (sd)

this_rq->avg_idle < sysctl_sched_migration_cost can reduce the frequency of
load_balance under high load, is there any way to dynamically adjust the
execution of load_balance in high-load scenarios, in order to strike a
balance between maintaining good CPU utilization and avoiding unnecessary
load_balance executions?