Re: [Patch v9 1/8] sched/pelt: Add support to track thermal pressure

[Amit Kucheria]

On Wed, Jan 29, 2020 at 4:06 AM Thara Gopinath
<thara.gopinath@xxxxxxxxxx> wrote:
>
> Extrapolating on the existing framework to track rt/dl utilization using
> pelt signals, add a similar mechanism to track thermal pressure. The
> difference here from rt/dl utilization tracking is that, instead of
> tracking time spent by a cpu running a rt/dl task through util_avg, the
> average thermal pressure is tracked through load_avg. This is because
> thermal pressure signal is weighted "delta" capacity and is not
> binary(util_avg is binary). "delta capacity" here means delta between the
> actual capacity of a cpu and the decreased capacity a cpu due to a thermal
> event.
>
> In order to track average thermal pressure, a new sched_avg variable
> avg_thermal is introduced. Function update_thermal_load_avg can be called
> to do the periodic bookkeeping (accumulate, decay and average) of the
> thermal pressure.
>
> Signed-off-by: Thara Gopinath <thara.gopinath@xxxxxxxxxx>
> Reviewed-by: Vincent Guittot <vincent.guittot@xxxxxxxxxx>
> ---
>
> v6->v7:
>         - Added CONFIG_HAVE_SCHED_THERMAL_PRESSURE to stub out
>           update_thermal_load_avg in unsupported architectures as per
>           review comments from Peter, Dietmar and Quentin.
>         - Updated comment for update_thermal_load_avg as per review
>           comments from Peter and Dietmar.
> v7->v8:
>         - Fixed typo in defining update_thermal_load_avg which was
>           causing build errors (reported by kbuild test report)
> v8->v9:
>         - Defined thermal_load_avg to read rq->avg_thermal.load_avg and
>           avoid cacheline miss in unsupported cases as per Peter's
>           suggestion.
>
>  include/trace/events/sched.h |  4 ++++
>  init/Kconfig                 |  4 ++++
>  kernel/sched/pelt.c          | 31 +++++++++++++++++++++++++++++++
>  kernel/sched/pelt.h          | 31 +++++++++++++++++++++++++++++++
>  kernel/sched/sched.h         |  3 +++
>  5 files changed, 73 insertions(+)
>
> diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
> index 420e80e..a8fb667 100644
> --- a/include/trace/events/sched.h
> +++ b/include/trace/events/sched.h
> @@ -613,6 +613,10 @@ DECLARE_TRACE(pelt_dl_tp,
>         TP_PROTO(struct rq *rq),
>         TP_ARGS(rq));
>
> +DECLARE_TRACE(pelt_thermal_tp,
> +       TP_PROTO(struct rq *rq),
> +       TP_ARGS(rq));
> +
>  DECLARE_TRACE(pelt_irq_tp,
>         TP_PROTO(struct rq *rq),
>         TP_ARGS(rq));
> diff --git a/init/Kconfig b/init/Kconfig
> index bd9f1fd..055c3bf 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -463,6 +463,10 @@ config HAVE_SCHED_AVG_IRQ
>         depends on IRQ_TIME_ACCOUNTING || PARAVIRT_TIME_ACCOUNTING
>         depends on SMP
>
> +config HAVE_SCHED_THERMAL_PRESSURE
> +       bool "Enable periodic averaging of thermal pressure"
> +       depends on SMP
> +
>  config BSD_PROCESS_ACCT
>         bool "BSD Process Accounting"
>         depends on MULTIUSER
> diff --git a/kernel/sched/pelt.c b/kernel/sched/pelt.c
> index bd006b7..5d1fbf0 100644
> --- a/kernel/sched/pelt.c
> +++ b/kernel/sched/pelt.c
> @@ -367,6 +367,37 @@ int update_dl_rq_load_avg(u64 now, struct rq *rq, int running)
>         return 0;
>  }
>
> +#ifdef CONFIG_HAVE_SCHED_THERMAL_PRESSURE
> +/*
> + * thermal:
> + *
> + *   load_sum = \Sum se->avg.load_sum but se->avg.load_sum is not tracked
> + *
> + *   util_avg and runnable_load_avg are not supported and meaningless.
> + *
> + * Unlike rt/dl utilization tracking that track time spent by a cpu
> + * running a rt/dl task through util_avg, the average thermal pressure is
> + * tracked through load_avg. This is because thermal pressure signal is
> + * weighted "delta" capacity and is not binary(util_avg is binary). "delta

May I suggest a slight rewording here and in the commit description,

   This is because the thermal pressure signal is weighted "delta"
capacity unlike util_avg which is binary.

It would also help, if you expanded on what you mean by binary in the
commit description and how the delta capacity is weighted.

> + * capacity" here means delta between the actual capacity of a cpu and the
> + * decreased capacity a cpu due to a thermal event.

Use consistent wording throughout the series - either capped or
decreased capacity.

> + */

This could be shortened to:

Delta capacity of cpu = Actual capacity - Capped capacity due to thermal event

> +int update_thermal_load_avg(u64 now, struct rq *rq, u64 capacity)
> +{
> +       if (___update_load_sum(now, &rq->avg_thermal,
> +                              capacity,
> +                              capacity,
> +                              capacity)) {
> +               ___update_load_avg(&rq->avg_thermal, 1, 1);
> +               trace_pelt_thermal_tp(rq);
> +               return 1;
> +       }
> +
> +       return 0;
> +}
> +#endif
> +
>  #ifdef CONFIG_HAVE_SCHED_AVG_IRQ
>  /*
>   * irq:
> diff --git a/kernel/sched/pelt.h b/kernel/sched/pelt.h
> index afff644..916979a 100644
> --- a/kernel/sched/pelt.h
> +++ b/kernel/sched/pelt.h
> @@ -7,6 +7,26 @@ int __update_load_avg_cfs_rq(u64 now, struct cfs_rq *cfs_rq);
>  int update_rt_rq_load_avg(u64 now, struct rq *rq, int running);
>  int update_dl_rq_load_avg(u64 now, struct rq *rq, int running);
>
> +#ifdef CONFIG_HAVE_SCHED_THERMAL_PRESSURE
> +int update_thermal_load_avg(u64 now, struct rq *rq, u64 capacity);
> +
> +static inline u64 thermal_load_avg(struct rq *rq)
> +{
> +       return READ_ONCE(rq->avg_thermal.load_avg);
> +}
> +#else
> +static inline int
> +update_thermal_load_avg(u64 now, struct rq *rq, u64 capacity)
> +{
> +       return 0;
> +}
> +
> +static inline u64 thermal_load_avg(struct rq *rq)
> +{
> +       return 0;
> +}
> +#endif
> +
>  #ifdef CONFIG_HAVE_SCHED_AVG_IRQ
>  int update_irq_load_avg(struct rq *rq, u64 running);
>  #else
> @@ -159,6 +179,17 @@ update_dl_rq_load_avg(u64 now, struct rq *rq, int running)
>  }
>
>  static inline int
> +update_thermal_load_avg(u64 now, struct rq *rq, u64 capacity)
> +{
> +       return 0;
> +}
> +
> +static inline u64 thermal_load_avg(struct rq *rq)
> +{
> +       return 0;
> +}
> +
> +static inline int
>  update_irq_load_avg(struct rq *rq, u64 running)
>  {
>         return 0;
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 1a88dc8..1f256cb 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -944,6 +944,9 @@ struct rq {
>  #ifdef CONFIG_HAVE_SCHED_AVG_IRQ
>         struct sched_avg        avg_irq;
>  #endif
> +#ifdef CONFIG_HAVE_SCHED_THERMAL_PRESSURE
> +       struct sched_avg        avg_thermal;
> +#endif
>         u64                     idle_stamp;
>         u64                     avg_idle;
>
> --
> 2.1.4
>