[RFC] Perfomance varies according to sysctl_sched_migration_cost

[Yicong Yang]

Hi all,

I noticed that some benchmark performance varies after tunning the sysctl_sched_migration_cost
through /sys/kernel/debug/sched/migration_cost_ns on arm64. The default value is 500000, and
I tried 10000, 100000, 1000000. Below are some results from mmtests, based on 5.14-release.

On Kunpeng920 (128cores, 4numa, 2socket):

pgbench (config-db-pgbench-timed-ro-medium)
                     mig-cost-500000        mig-cost-100000         mig-cost-10000       mig-cost-1000000
Hmean     1       9558.99 (   0.00%)     9735.31 *   1.84%*     9410.84 *  -1.55%*     9602.47 *   0.45%*
Hmean     8      17615.90 (   0.00%)    17439.78 *  -1.00%*    18056.44 *   2.50%*    19222.18 *   9.12%*
Hmean     12     25228.38 (   0.00%)    25592.69 *   1.44%*    26739.06 *   5.99%*    27575.48 *   9.30%*
Hmean     24     46623.27 (   0.00%)    48853.30 *   4.78%*    47386.02 *   1.64%*    48542.94 *   4.12%*
Hmean     32     60578.78 (   0.00%)    62116.81 *   2.54%*    59961.36 *  -1.02%*    58681.07 *  -3.13%*
Hmean     48     68159.12 (   0.00%)    67867.90 (  -0.43%)    65631.79 *  -3.71%*    66487.16 *  -2.45%*
Hmean     80     66894.87 (   0.00%)    73440.92 *   9.79%*    68751.63 *   2.78%*    67326.70 (   0.65%)
Hmean     112    68582.27 (   0.00%)    65339.90 *  -4.73%*    68454.99 (  -0.19%)    67211.66 *  -2.00%*
Hmean     144    76290.98 (   0.00%)    70455.65 *  -7.65%*    64851.23 * -14.99%*    64940.61 * -14.88%*
Hmean     172    63245.68 (   0.00%)    68790.24 *   8.77%*    66246.46 *   4.74%*    69536.96 *   9.95%*
Hmean     204    61793.47 (   0.00%)    63711.62 *   3.10%*    66055.64 *   6.90%*    58023.20 *  -6.10%*
Hmean     236    61486.75 (   0.00%)    68404.44 *  11.25%*    70499.70 *  14.66%*    58285.67 *  -5.21%*
Hmean     256    57476.13 (   0.00%)    65645.83 *  14.21%*    69437.05 *  20.81%*    60518.05 *   5.29%*

tbench (config-network-tbench)
                     mig-cost-500000        mig-cost-100000         mig-cost-10000       mig-cost-1000000
Hmean     1        333.12 (   0.00%)      332.93 (  -0.06%)      335.34 *   0.67%*      334.36 *   0.37%*
Hmean     2        665.88 (   0.00%)      667.19 *   0.20%*      666.47 *   0.09%*      667.02 *   0.17%*
Hmean     4       1324.10 (   0.00%)     1312.23 *  -0.90%*     1313.07 *  -0.83%*     1315.13 *  -0.68%*
Hmean     8       2618.85 (   0.00%)     2602.00 *  -0.64%*     2577.49 *  -1.58%*     2600.48 *  -0.70%*
Hmean     16      5100.74 (   0.00%)     5068.80 *  -0.63%*     5041.34 *  -1.16%*     5069.78 *  -0.61%*
Hmean     32      8157.22 (   0.00%)     8163.50 (   0.08%)     7936.25 *  -2.71%*     8329.18 *   2.11%*
Hmean     64      4824.56 (   0.00%)     4890.81 *   1.37%*     5319.97 *  10.27%*     4830.68 *   0.13%*
Hmean     128     4635.17 (   0.00%)     6810.90 *  46.94%*     5304.36 *  14.44%*     4516.06 *  -2.57%*
Hmean     256     8816.62 (   0.00%)     8851.28 *   0.39%*     8448.76 *  -4.17%*     6840.12 * -22.42%*
Hmean     512     7825.56 (   0.00%)     8538.04 *   9.10%*     8002.77 *   2.26%*     7946.54 *   1.55%*

Also on Raspberrypi 4B:

pgbench (config-db-pgbench-timed-ro-medium)
                   mig-cost-500000        mig-cost-100000
Hmean     1     1651.41 (   0.00%)     3444.27 * 108.56%*
Hmean     4     4015.83 (   0.00%)     6883.21 *  71.40%*
Hmean     7     4161.45 (   0.00%)     6646.18 *  59.71%*
Hmean     8     4277.28 (   0.00%)     6764.60 *  58.15%*

For tbench on Raspberrypi 4B and both pgbench and tbench on x86, tuning sysctl_sched_migration_cost
doesn't have such huge difference and will have some degradations (max -8% on x86 for pgbench) in some cases.

The sysctl_sched_migration_cost will affects the frequency of load balance. It will affect
directly in task_hot() and newidle_balance() to decide whether we can do a migration or load
balance. And affects other parameters like rq->avg_idle, rq->max_idle_balance_cost and
sd->max_newidle_lb_cost to indirectly affect the load balance process. These parameters record
the load_balance() cost and will be limited up to sysctl_sched_migration_cost, so I measure
the average cost of load_balance() on Kunpeng920 with bcc tools(./funclantency load_balance -d 10):

system status   idle   50%load  100%load
avg cost      3160ns    4790ns    7563ns

The average cost of load balance seems quite smaller than the default sysctl_sched_migration_cost
which is 500000(500ms).

So I have some RFC questions:
1. how is the default 500000 (500ms) migration cost is measured or caculated?
   The value has never changed in the past decade. I dig into the git commits and find it was introduced
   in da84d9617672 ("sched: reintroduce cache-hot affinity"). But it didn't explain how did this value come.
2. The ABI now has been removed from sysctl and moved to debugfs. As tuning this can improve the performance
   of some workloads on some platforms, maybe it's better to make it a formal sysctl again with docs?

I'll be appreciated for any comments and replies!

Thanks,
Yicong