Re: [PATCH v4] sched/fair: Fix CPU bandwidth limit bypass during CPU hotplug

[Vincent Guittot]

On Thu, 12 Dec 2024 at 05:32, Vishal Chourasia <vishalc@xxxxxxxxxxxxx> wrote:
>
> CPU controller limits are not properly enforced during CPU hotplug
> operations, particularly during CPU offline. When a CPU goes offline,
> throttled processes are unintentionally being unthrottled across all CPUs
> in the system, allowing them to exceed their assigned quota limits.
>
> Consider below for an example,
>
> Assigning 6.25% bandwidth limit to a cgroup
> in a 8 CPU system, where, workload is running 8 threads for 20 seconds at
> 100% CPU utilization, expected (user+sys) time = 10 seconds.
>
> $ cat /sys/fs/cgroup/test/cpu.max
> 50000 100000
>
> $ ./ebizzy -t 8 -S 20        // non-hotplug case
> real 20.00 s
> user 10.81 s                 // intended behaviour
> sys   0.00 s
>
> $ ./ebizzy -t 8 -S 20        // hotplug case
> real 20.00 s
> user 14.43 s                 // Workload is able to run for 14 secs
> sys   0.00 s                 // when it should have only run for 10 secs
>
> During CPU hotplug, scheduler domains are rebuilt and cpu_attach_domain
> is called for every active CPU to update the root domain. That ends up
> calling rq_offline_fair which un-throttles any throttled hierarchies.
>
> Unthrottling should only occur for the CPU being hotplugged to allow its
> throttled processes to become runnable and get migrated to other CPUs.
>
> With current patch applied,
> $ ./ebizzy -t 8 -S 20        // hotplug case
> real 21.00 s
> user 10.16 s                 // intended behaviour
> sys   0.00 s
>
> This also has another symptom, when a CPU goes offline, and if the cfs_rq
> is not in throttled state and the runtime_remaining still had plenty
> remaining, it gets reset to 1 here, causing the runtime_remaining of
> cfs_rq to be quickly depleted.
>
> Note: hotplug operation (online, offline) was performed in while(1) loop
>
> Signed-off-by: Vishal Chourasia <vishalc@xxxxxxxxxxxxx>
> Suggested-by: Zhang Qiao <zhangqiao22@xxxxxxxxxx>
> Tested-by: Madadi Vineeth Reddy <vineethr@xxxxxxxxxxxxx>

With fixing the typo below
Acked-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>

>
> v3: https://lore.kernel.org/all/20241210102346.228663-2-vishalc@xxxxxxxxxxxxx
> v2: https://lore.kernel.org/all/20241207052730.1746380-2-vishalc@xxxxxxxxxxxxx
> v1: https://lore.kernel.org/all/20241126064812.809903-2-vishalc@xxxxxxxxxxxxx
>
> ---
>  kernel/sched/fair.c | 20 +++++++++++++-------
>  1 file changed, 13 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index aa0238ee4857..72746e75700c 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -6679,6 +6679,10 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
>
>         lockdep_assert_rq_held(rq);
>
> +       // Do not unthrottle for an active CPU

typo: please use /* my comment */

> +       if (cpumask_test_cpu(cpu_of(rq), cpu_active_mask))
> +               return;
> +
>         /*
>          * The rq clock has already been updated in the
>          * set_rq_offline(), so we should skip updating
> @@ -6693,19 +6697,21 @@ static void __maybe_unused unthrottle_offline_cfs_rqs(struct rq *rq)
>                 if (!cfs_rq->runtime_enabled)
>                         continue;
>
> -               /*
> -                * clock_task is not advancing so we just need to make sure
> -                * there's some valid quota amount
> -                */
> -               cfs_rq->runtime_remaining = 1;
>                 /*
>                  * Offline rq is schedulable till CPU is completely disabled
>                  * in take_cpu_down(), so we prevent new cfs throttling here.
>                  */
>                 cfs_rq->runtime_enabled = 0;
>
> -               if (cfs_rq_throttled(cfs_rq))
> -                       unthrottle_cfs_rq(cfs_rq);
> +               if (!cfs_rq_throttled(cfs_rq))
> +                       continue;
> +
> +               /*
> +                * clock_task is not advancing so we just need to make sure
> +                * there's some valid quota amount
> +                */
> +               cfs_rq->runtime_remaining = 1;
> +               unthrottle_cfs_rq(cfs_rq);
>         }
>         rcu_read_unlock();
>
>
> base-commit: 231825b2e1ff6ba799c5eaf396d3ab2354e37c6b
> --
> 2.47.0
>